Thursday, October 13, 2016

Setting Up A Mail Server - Part 1: MySQL, Postfix and Dovecot - Incomplete

I've spent a few days going through the ISPMail server (debian Jessie with dovecot and postfix) tutorial and I've found the experience somewhat frustrating.

The use of the database, for example, seems to introduce redundancy when the whole point of a relational database is to remove that risk. Nothing in the tutorial seems to scale all that well i.e. how do you separate out functions but allow them to keep communicating between different machines?

More frustrating for me was the fact that it attempts to tackled everything NOW NOW NOW rather than addressing things in a logical order (I had to pull out my whiteboard to make sense of it).

Within code snippets, any text in red should/could be changed. Blue indicates that the same data is going to be used for a whole chunk.

I'm making an assumption about the structure of the sort of network this is sitting on. The MySQL, File Server, Postfix, Dovecot and web server machines can all be set up on different machines BUT exist on a trusted network i.e. they sit behind a firewall making communication between the machines relatively trusted. Most of this should still work if this isn't the case BUT avoid anything to do with lmtp - it's not made for an untrusted environment.

Warning: This tutorial is incomplete and untested. I will be testing it within the next couple of days and figuring out the missing bits. It needs a test at the end to make sure that it is all working as expected. The SMTP server doesn't appear to have been configured yet (the ISPMail tutorial does this after setting up roundcube) and security on that to make sure the mail server isn't used for relaying spam.


So here goes...

The Structure of an E-mail Server

Very basically you need to be able to send emails, get your emails, which are handled by a 'mta server' (Mail Transport Agent) and retrieve our emails using either IMAP or POP3.

Because editing text files sucks a great big one to handle email addresses, a database is a damn good idea. And of course, we're going to want to do a whole lot of things around reducing the spam coming in and stop our server from being used to send lots of spam. We also want to do this securely so we're going to need some domain verification certificates. Oh and we have to set up DNS to tell the Internet where mail should go to. Eventually we're probably going to want to set up a web front end for the mail so that you can check it anywhere you like and/or configure an email client (ick).

Yep. It's messy.

If you need more information, go and have a look at this page.

We're going to use Postfix as our MTA, Dovecot for our IMAP/POP3 server (though we're going to disable POP3. I'll explain later). For our database we're going to use MySQL (although the instructions likely don't change for MariaDB).

Setting up the Infrastructure

DNS

I'm going to trust that you have purchased a domain name. In your DNS records you'll have something like:
 A  example.org     8.8.8.8  

Which is your A record for the base domain. Add another A record for something like mx.example.org or mail.example.org I'm going to stick to mx (Mail eXchange). Then add an MX record. It should look something like this:

 A  example.org       8.8.8.8  
 A  mx.example.org    8.8.8.8  
 MX example.org       mx.example.org  

This is basically saying "use mail.example.org to serve mail for @example.org". If you don't know who to use for your DNS, I've found Cloudflare to be pretty good.

Certificates

This is something I was kind of annoyed about with most tutorials. They'd talk about using self signed certificates and there was little information on using the free certificates issued by Let's Encrypt. This is what I'm using.

Let's Encrypt state in their FAQ that the certificates they issue are for domain verification only and aren't suitable for email encryption.

My concern here is that there's an ambiguity. I care more about the encryption when transmitting the email than I do about encrypting the email itself. I hope to be able to trust the user to be able to worry about the encryption of the email if it is needed via PGP (Pretty Good Privacy) or some other means. In which case, I have no idea why I should be worried about my Certificate Authority being able to encrypt emails.

To get started, go to this site, and get instructions on how to download certbot. I'm not going to go into using this tool. You'll need to set up a web server (I recommend nginx myself. I was using Apache but found it horrendously slow) to get the certificates.

Sharing Files - add ssh-keygen and move to it's own tutorial

In my case, I want to keep my mail server well away from other functions such as my database and web server. Which means there's the potential of me needing to securely share some files, such as certificates, between machines (Let's Encrypt uses my webserver to verify my domain but I'm not sure if it needs it to renew. It's just easier for me be able to share the files).

While I have a firewall and my network seems to be reasonably secure, it's worthwhile having some security on the inside of your own network. In which case, I'm going to use SSH to share files and lock things down as much as humanly possible.

I'm going to assume you have ssh servers on all of your machines.

On the machine containing the files:

 apt-get install openssh-server  
Edit /etc/ssh/sshd_config:
Change the line starting:
 Subsystem sftp  
To:
 Subsystem sftp internal-sftp
If you need to debug sshfs later on, change this to:
 Subsystem sftp internal-sftp -l DEBUG1  
At the end of this file (It must be at the end), add something along the following:
 Match User cert  
   ChrootDirectory /etc/letsencrypt  
   ForceCommand internal-sftp  
   AllowTCPForwarding no  
   X11Forwarding no  
   PasswordAuthentication no  

WARNING: When using ChrootDirectory, that folder MUST be owned by root and not be group writable.
 
What this does is allows a user named "cert" to access our certificates only. The cert user can not log in or do anything else except access the /etc/letsencrypt folder.

Add the cert user (at BASH):
 useradd cert -p '!' -s /bin/false  

Give the cert user permissions to the files it actually needs:
 chgrp cert /etc/letsencrypt/live /etc/letsencrypt/archive  
 chmod 750 /etc/letsencrypt/live /etc/letsencrypt/archive  

On the machine that needs the files:

We're going to use a combination of sshfs and autofs just for robustness (read: we don't really want to have to worry too much about the order in which machines have to be booted in).

Install the needed software:
 apt-get install sshfs autofs  
Make a folder for autofs to control:
 mkdir /mnt/sshfs  
Edit /etc/auto.master. And the following line:
 /mnt/sshfs /etc/auto.sshfs uid=1000,gid=1000,--timeout=30,--ghost  
If only one user will be using the files from here, it's worthwhile setting the uid and gid to that user. I can get that information with:
 id dovecot  
Save and exit. Make the file /etc/auto.sshfs and put in the following:
 certs -fstype=fuse,ro,nodev,nonempty,noatime,max_read=65536 :sshfs\#cert@webserver1\:/  
I could change the 'ro' to 'rw' for readwrite access. I could also add "allow_other" which would give everyone on the system access to that mount.

The Database - MySQL

We want to use a database to store information about 3 things:
  • Domain(s) - we're setting things up to allow for scalability which means we should be able to easily add domains should we need to.
  • Mailboxes.
  • Aliases - virtual addresses that lead to mailboxes.
On the machine you're setting MySQL up on (this can be the same machine as anything else. It will create little branches throughout this tutorial), install the needed software:

 apt-get install mysql-server mysql-client 

We're going to rely on the command line to configure our database. The reasoning for this is that phpMyAdmin doesn't really abstract things away to become any more or less user friendly though does install a piece of web accessible software on your server that seems unnecessary.

If you don't know your mysql root password, you may need to reset it:

 sudo service mysql stop  
 sudo mysqld --skip-grant-tables &  
 mysql -u root mysql  
 UPDATE user SET Password=PASSWORD('YOURNEWPASSWORD') WHERE User='root';  
 exit;  
 sudo service mysql restart  

Log into mysql:
 mysql -u root -p  

Create and configure the user that Postfix and Dovecot will use to access the database:
 CREATE USER 'mailuser'@'127.0.0.1' IDENTIFIED BY 'DBPassword';  
 GRANT SELECT,INSERT,UPDATE,DELETE ON mailserver.* TO 'mailuser'@'127.0.0.1';  

Both Postfix and Dovecot need access to the database. If you're running MySQL on a different machine from postfix and dovecot, you need to change '127.0.0.1' to the IP address of the machine running Postfix or Dovecot. If Postfix and Dovecot are running on different machines from each other, you need to create separate accounts for each of those users. They can be the same user name. It's just the host portion that needs to be different.

To test that you're able to log in from the computer you're going to need to access the database from, install mysql-client and attempt to login via:

 mysql --host database.example.org -u mailuser -p  

Back to the database.... Make your database:
 CREATE DATABASE mailserver;  
 USE mailserver;

Our first table is going to contain information about the domain names we're providing email for. A domain name can be a maximum size of around 255 characters long.
 CREATE TABLE domains (  
   id    INT(11)       NOT NULL AUTO_INCREMENT,  
   name  VARCHAR(255)  NOT NULL,  
   PRIMARY KEY ( id )  
 );  

 INSERT INTO domains( id, name ) VALUES( 1, 'example.org' ); # test data

The next table is all about our mailboxes:
 CREATE TABLE mailboxes(  
   id         INT             NOT NULL AUTO_INCREMENT,  
   domain_id  INT             NOT NULL,  
   name       VARCHAR(65)     NOT NULL,  
   password   VARCHAR(128)    NOT NULL,  
   PRIMARY KEY( id ),  
   UNIQUE( name, domain_id ),  
   FOREIGN KEY(domain_id) REFERENCES domains(id)  
     ON DELETE CASCADE  
 );  

 INSERT INTO mailboxes( domain_id, name, password )
 VALUES( 1, 'test', 'b109f3bbbc244eb82441917ed06d618b9008dd09b3befd1b5e07394c706a8bb980b1d7785e5976ec049b46df5f1326af5a2ea6d103fd07c95385ffab0cacbc86' );

For password we're going to use SHA512 which should be available on most Linux systems. SHA512 hashes passwords into 128 hexidecimal characters i.e. the length doesn't need to be variable.

We don't need to store the email address because that's a concatenation between the mailbox name and domain name and given that the domain name is already stored in the domains table, and we've got a link to the domain using a foreign key, it can be considered a calculated field.

And finally, aliases...
 CREATE TABLE aliases(  
   id          INT           NOT NULL AUTO_INCREMENT,  
   domain_id   INT           NOT NULL,  
   source      VARCHAR(65)   NOT NULL,  
   destination VARCHAR(320)  NOT NULL,  
   PRIMARY KEY( id ),  
   FOREIGN KEY( domain_id ) REFERENCES domains( id ),  
   UNIQUE( domain_id, source, destination )
 );  

 INSERT INTO aliases( domain_id, source, destination )
 VALUES( 1, 'alias', 'otheraddress@testdomain.com' );

You'll notice that source and destination have completely different lengths. The source can be calculated by the source and domain name whereas the destination can potentially be sent to an entirely different domain.

Postfix

 apt-get install postfix postfix-mysql  

We're going to put our configuration in its own folder just because it's a touch cleaner. Then we need to create 3 files which tell Postfix how to access our database.

 mkdir /etc/postfix/config  

Make a file called /etc/postfix/config/domains.cf and populate it with the following:
 user = mailuser  
 password = DBPassword  
 hosts = 127.0.0.1  
 dbname = mailserver  
 query = SELECT 1 FROM domains WHERE name='%s'  

You can test the query in mysql (substituting %s for a domain name). If the domain exists, it returns 1 (true). Otherwise it returns an empty set (false).

Enable the configuration in Postfix:
 postconf virtual_mailbox_domains=mysql:/etc/postfix/config/domains.cf  

And finally, test this configuration setting:
 postmap -q example.org mysql:/etc/postfix/config/domains.cf  

Make a file called /etc/postfix/config/mailboxes.cf and populate it with the following:
 user = mailuser  
 password = DBPassword  
 hosts = 127.0.0.1  
 dbname = mailserver  
 query = SELECT 1 FROM mailboxes JOIN domains ON mailboxes.domain_id=domains.id WHERE CONCAT_WS('@',mailboxes.name,domains.name )='%s'  

What's happening here is that we're joining the domains table so that we can get the domain name to form the email address. The reason for doing this is that it leads to less errors. i.e. if I'm delivering emails for 'example.org' but accidentally put in 'example.com', using this configuration theres only one place I could have made that mistake - in the domains table and fixing it for one fixes it for all. Whereas if I store the entire email address in a single field in the mailboxes table, I can make that mistake in a bunch of different places.

Enable the configuration in Postfix:
 postconf virtual_mailbox_maps=mysql:/etc/postfix/config/mailboxes.cf  

And test it...
 postmap -q test@example.org mysql:/etc/postfix/config/mailboxes.cf  

And finally, create a file called /etc/postfix/config/aliases.cf and populate it with the following:
 user = mailuser  
 password = DBPassword  
 hosts = 127.0.0.1  
 dbname = mailserver  
 query = SELECT aliases.destination FROM aliases JOIN domains ON aliases.domain_id = domains.id WHERE CONCAT_WS('@',aliases.source,domains.name)='%s'

This query string is much like the last.

Enable the configuration:
 postconf virtual_alias_maps=mysql:/etc/postfix/config/aliases.cf  

And test it...
 postmap -q alias@example.org mysql:/etc/postfix/config/aliases.cf   

Set your permissions on the files:
 chgrp postfix /etc/postfix/config/*  
 chmod u=rw,g=r,o= /etc/postfix/config/*  

Enabling outgoing mail
1. Get postfix to use dovecot for authentication
At the command line run:
 postconf smtpd_sasl_type=dovecot  
 postconf smtpd_sasl_path=private/auth  
 postconf smtpd_sasl_auth_enable=yes  

2. Enable encryption
At the command line run:
 postconf smtpd_tls_security_level=may  
 postconf smtpd_tls_auth_only=yes  
 postconf smtpd_tls_cert_file=/etc/ssl/certs/mailserver.pem  
 postconf smtpd_tls_key_file=/etc/ssl/private/mailserver.pem  


If Postfix and Dovecot are going to run on the same machine:
Run:
 postconf virtual_transport=lmtp:unix:private/dovecot-lmtp  

If Postfix and Dovecot are going to run on different machines:
Run:
 postconf virtual_transport = lmtp:inet:dovecot.example.org  
Use the address of the dovecot machine here. The port specified here is one that's reserved for private mail use.

Dovecot

 apt-get install dovecot-mysql dovecot-imapd dovecot-managesieved dovecot-lmtpd  

Dovecot handles how we get our emails and is also going to be responsible for storing emails. The line above is missing POP3 support. If you need it, then do this:
 apt-get install dovecot-pop3d   

POP3 just isn't great with Spam Assassin. POP3 can't grab folders. Instead it just grabs emails from the inbox. Very simple. The problem with this is that you then can't put emails that have been marked as spam  into their own folder. In a POP3 system, the user never gets these emails. In an imap world, email folders are stored on the server.

First things first: Set up a user (and group) for the dovecot service to run under:
 groupadd -g 5000 vmail  
 useradd -g vmail -u 5000 vmail -d /var/vmail -m  

Create a place to store emails in /var/vmail. For me, I'm going to be using the sshfs stuff because my file storage is on a different machine from where my mail server is being run. I'm not going to step you through how to do this as there's (hopefully) enough detail in the sshfs section to do this along with using 'mount -o bind' to get the folder accessible from the right place.

Set your permissions:
 chown -R vmail.vmail /var/vmail  

Dovecot, by default on Debian, stores its configuration files in:
/etc/dovecot/conf.d/
The files are processed in order so files starting with 99 are processed after files starting with 01. Most of the files are commented out and mostly contain examples so most of what we're doing here is appending to the relevant files.

10-auth.conf

Make sure that this line is uncommented.
 auth_mechanisms = plain  

If you're using Outlook Express on Windows XP or Windows Vista (though there's really no good reason you should be), that line needs to be:
 auth_mechanisms = plain login  

Plain may look dangerous though by default dovecot does not accept passwords sent via plain text i.e. TLS encrypted passwords only.

Uncomment (remove the '#') the following line.
 !include auth-sql.conf.ext  

Comment out the all of the '!include auth-' lines. They're well out of scope for this hastily thrown together tutorial.

auth-sql.conf.ext

Comment out the userdb section entirely and append the following to the bottom:
 userdb {  
  driver = static  
  args = uid=vmail gid=vmail home=/var/vmail/%d/%n  
 }  

This tells the dovecot daemon to run as the user 'vmail', group 'vmail' and where to place our emails (/var/vmail/[domain_name]/[mailbox_name]).

10-mail.conf

Change the line that reads:
 mail_location = mbox:~/mail:INBOX=/var/mail/%u  

to:
 mail_location = maildir:/var/vmail/%d/%n/Maildir  

10-master.conf

This file deals with what services are available. We need to concern ourselves with 2 things here:
  • Allowing postfix to communicate with dovecot for authentication.
  • Allowing postfix to send emails to dovecot (lmtp)

If postfix and dovecot are running on the same machine: then look for the line that starts with:
 #unix_listener /var/spool/postfix/private/auth {  

Uncomment that line and change the whole stanza to look like the following:
 unix_listener /var/spool/postfix/private/auth {  
  mode = 0660  
  user = postfix  
  group = postfix  
 }  

Change the stanza that looks like:
 unix_listener lmtp {  
  #mode = 0666  
 }  
To:
 service lmtp {  
  unix_listener /var/spool/postfix/private/dovecot-lmtp {  
   group = postfix  
   mode = 0600  
   user = postfix  
  }  
 }  



If postfix and dovecot are running on different machines: 

Dovecot

Look for the stanza that starts with:
 service lmtp {  

and make it look like the following:
 inet_listener lmtp {  
  address = 192.168.0.24 127.0.0.1 ::1  
  port = 24  
 }  
The address is the IP address of the NIC to listen on. You may want to use a firewall on this machine to limit access further i.e. look at the address of traffic coming in on port 24 and limit access to that port to very specific (the machine running postfix) machines.

Look for the block that starts with:
 service auth {  

Within that block, add the following stanza:
 inet_listener {  
   port = 12345  
 }  

Choose a random port between 1024 and 65535.

Postfix

On the command line enter:
 smtpd_sasl_path = inet:dovecot.example.org:12345  
 smtpd_sasl_type = dovecot  

Where it reads "dovecot.example.org", change it to the address (ip or host name) of the machine running dovecot. Use the port number chosen above where it reads 12345.

10-ssl.conf

If you've got a certificate for your mail server domain name (mx.example.org) from let's encrypt, the certificates are probably somewhere along the times of /etc/letsencrypt/live/mx.example.org. The 2 files we care about are cert.pem and privkey.pem.

Look for the line that starts with:
 ssl =   
And make sure it says:
 ssl = yes  

Look for the lines that say:
 #ssl_cert = </etc/dovecot/dovecot.pem  
 #ssl_key = </etc/dovecot/private/dovecot.pem  

Uncomment them and change them to read:
 ssl_cert = </etc/letsencrypt/live/mx.example.org/cert.pem  
 ssl_key = </etc/letsencrypt/live/mx.example.org/privkey.pem  

15-mailboxes.conf

We need to configure a couple of folders to exist by default. Look for the section that says:
 mailbox Drafts {  
  special_use = \Drafts  
 }  
 mailbox Junk {  
  special_use = \Junk  
 }  
 mailbox Trash {  
  special_use = \Trash  
 }  

For each of those stanzas, add auto = subscribe. It should now look like this:
 mailbox Drafts {  
  auto = subscribe
  special_use = \Drafts  
 }  
 mailbox Junk {  
  auto = subscribe
  special_use = \Junk  
 }  
 mailbox Trash {  
  auto = subscribe
  special_use = \Trash  
 }  

This makes it so that your users can't remove these folders. They're all special use folders and users generally expect them to exist anyway.

/etc/dovecot/dovecot-sql.conf.ext

And finally, we need to tell dovecot how to talk to our database. To the bottom of dovecot-sql.conf.ext, add the following:
 driver = mysql  
 connect = host=127.0.0.1 dbname=mailserver user=mailuser password=DBPassword
 default_pass_scheme = SHA512-CRYPT  
 password_query = SELECT mailboxes.name AS username, domains.name AS domain, CONCAT_WS('@', mailboxes.name, domains.name) , mailboxes.password FROM mailboxes JOIN domains ON mailboxes.domain_id = domains.id WHERE mailboxes.name='%n' AND domains.name='%d'  

Change the host to the ip address of the server running the database server and the password to the password of your database for the mailuser user.

Make sure the permissions for dovecot-sql.conf.ext don't allow for users to get your mysql authentication details:

 chown root:root /etc/dovecot/dovecot-sql.conf.ext  
 chmod go= /etc/dovecot/dovecot-sql.conf.ext  

/etc/dovecot/20-lmtp.conf

We want to enable the sieve plugin. This allows us to apply rules to email on the server (filtering and the like). Change the line that reads:
 #mail_plugins = $mail_plugins  
To:
 mail_plugins = $mail_plugins sieve  

Running the new configuration:

On the command line run:
 service dovecot restart  

Monday, September 12, 2016

The Interconnectivity of All Things and Why We Should be Worried About Rape Culture

New Zealand has a serious problem with rape culture. It's in our politics. It's in our sports. One could even go as far as to say that it's our national identity.

Just last year our prime minister, John Key, was found to have harassed a waitress by pulling her ponytail. He did this over a series of months but after the news broke, it was referred to jovially as "Ponytailgate". It was relegated to the realms of playfulness. This, is rape culture. We are saying that the use of power over a woman is just "boys being boys".

Rape culture is a culture that enables, and sometimes even encourages, rape.

This year, we have a rugby team called "The Chiefs" who have been found to have acted in an abhorrent manner. For one of their celebrations, they hired a stripper. When the stripper spoke out about her treatment, which included being touched between the legs "very forcefully", had gravel and alcohol thrown at her and had her having to kick one of the players to make him stop (because "no" apparently wasn't enough), things went insane.

The sponsors stood with the team. The corporate services executive of Gallagher Group, Margaret Comer, who is also a trustee for the Waikato Women's Refuge, blatantly blamed the stripper. She appears to have suffered no consequences. Neither from Gallagher Group nor Women's Refuge. i.e. the sponsors unashamedly enabled the behavior.

The internal investigation, conducted by NZ Rugby, the governing body of rugby in NZ (and owner of the Chiefs), said "allegations from witnesses could not be substantiated" and so players were issued a caution but no further actions were taken. It wasn't the players who were actually there that were issued a caution. The whole team was.

Does everyone remember the "Roast Busters" a few years back? It was a case that saw a bunch of teenage boys raping, often in groups, and posting photos of said rape on the Internet. When the story broke, the police had said they were aware of the group and had them under investigation. They also stated that no one had complained about the group. It was later revealed that there had been multiple complaints made though the police had, as in this case, decided to take no further action.

Our minister of women, on this matter, as on the matter involving our very own PM harassing a woman over several months, had absolutely no comment on any of this.

The stripper lost 2 jobs.

This is where I think things got really weird. A campaign was started by the Human Rights Commission titled "Love Rugby. Respect Women". The campaign was supported by a whole lot of women's rights advocates and it eventually lead to The Chiefs accepting help in attempting to improve their culture toward women.

Where do we start to untangle this mess? Hint: It is all a culture that enables rape.

But it doesn't stop there. NZ Rugby are a commercial entity. Every time someone buys a ticket to a game, NZ Rugby benefits.

Every time we refer to rugby as being a major part of our national identity or "like a religion" (as was on the "Love Rugby. Respect Women" campaign), we have tied our national identity to NZ Rugby.

We have put rugby above the rights of 50% of our population.

And this is why I find "Love Rugby. Respect Women."  weird. It perpetuates the problem by focusing on a game first. Our very own Human Rights Commission didn't say Respect People and then worry about a game. It said Love a Game. Worry about the humans and their rights second.

And yet, none of this is all that surprising. Last year a prominent rugby player won NZer of the year. His ability to play a game was deemed a greater contribution to NZ than a woman, Louise Nicholas, who the Prime Minister acknowledged as having "....done more for sexual violence and sexual abuse than any other New Zealander." at the award ceremony. Outrage on social media was met with "but he does other things too" or "he deserves it!"

We made, and continue to make, a statement about our values; it seems our values are that we would rather preserve our rape culture than acknowledge our own part in supporting it.

Thursday, March 31, 2016

Why the Microsoft Linux Convergence Could Be Bad News

The web was a buzz with news about MS telling everyone that Bash is going to be a thing in Windows. Not a VM, but rather, a part of Windows.

The way I see it, it's not cause for celebration, but rather, cautious optimism.

Remember when MS Office included support for OpenOffice formats? Now MS Office could open OpenOffice formats! Except that they could only with certain provisions. They didn't follow the specifications of the format and instead broke it. Intentionally. So either OpenOffice had to include support for MS Office's broken implementation of it, or try and control people's perception of it.

MS are completely capable of looking to be being supportive and playing ball but also blindsiding the other players to their advantage.

My main concern though is those people who tell you that they can administer both Windows and Linux. The operating systems are like talking a different language. When I think of solving problems, I think in terms of Linux and I then usually end up having to translate back into Windows if I'm working on Windows. Ditto for Mac OSX. Linux first, translate. More often that not it works well for Mac OSX. Windows I often find myself swearing.

So what happens to people when they appear to be using a Linux type shell? Do they understand that there's a difference between thinking in Linux and thinking in Windows? There's a real issue with EVERYONE doing things badly.

A Windows person then goes and tells everyone that he can do Linux! but is limited by what they can do in Windows (what's a link and what's the difference between a symbolic link and a hard link?)?

A Linux person thinks something is happening in Windows because of commands they use in Linux and then discovers that either the OS doesn't support what they're trying to do at all or does it in a completely different way.

It's not that hard to imagine. It's more likely that Bash on Windows will be like Bash on BSD and Bash on Solaris - they're the same thing but they're not the same thing i.e. there are differences on each of the OSes and those differences make the way you solve problems different.

Whether that means Linux people suddenly think they're experts in Windows and Windows people suddenly think they're experts in Linux remains to be seen.

Thursday, February 18, 2016

New Zealander of the Year and Other Issues of Sexism and Privilege

This morning had me waking up on the wrong side of the bed. I rolled over, opened up my laptop to see what was going on on social media and immediately erupted in a fury of "What the fuck is going on?!?".

Richie McCaw was named New Zealander of the Year.

If you're not sure who Richie McCaw is, it's not entirely surprising. Richie McCaw is a rugby player. He also does some charitable works, but if you Google his name, you'll see only references to rugby.

Not sure what rubgy is? That's not entirely surprising either. Rugby is a game whose importance is over inflated in only very few countries. New Zealand would be the largest country whereby Rugby is THE primary major sport. So it's little wonder then that New Zealand have also always excelled at the sport. What is a mystery though is why the "All Blacks" have failed to win so many World Cups.

Anyway. So Richie McCaw is it. He's what we should all aspire to. That's one hell of a value statement that is amazingly dismissive of just about EVERYTHING else. The two other finalists, an environmentalist and a victim's rights advocate... Yep, Richie McCaw's ability to play a game is somehow more important.

The Victim's Rights Advocate - Louise Nicholas - primarily works within the field of rape victims. So we're saying that Richie McCaw's ability to play a game and some charitable works is somehow of more value than someone who works hard, not at being good at a game, but rather, giving people a voice within a part of society that we are still, excuse the language, shit at. i.e. rape culture.

This reeks of white male privilege to me. My saying so ended up with one guy unfriending me on Facebook, and my arguing with another about whether it was "just" privilege or white male privilege.

I maintain that it can be called white male privilege (I tend to replace the word "male" with "penis" - mainly to be offensive and because it points out the ridiculousness of arguments of gender) because it would take someone of that much privilege to make such a value statement. i.e. that someone who "works" full time at a game and does a bit of charity on the side is of more value than people who dedicate their lives to helping people.

As well as Louise Nicholas as a potential winner, there's Helen Kelly. Helen Kelly is currently dying of cancer (there's a whole other story in there about medical marijuana) and dedicated her life to improving the working conditions of others. In my personal life I know some extraordinary people. People who have dedicated their lives to education, whether facilitating it or pushing it. And in all seriousness, I would've liked a winner who's positive contributions to society were from a place of intelligence. The war being waged on intelligence is terrible!.

My mood didn't improve as the next thing that I read is that TVNZ (a local TV broadcaster) are starting another channel! Aimed at males, with sports and shows like Two and a Half Men. The channel is being called "Duke". As far as I'm concerned, they might as well have just called it "White Penis".

Equity and intelligence took a hit today as value statements were made dismissing those things. We should be striving to make things better. Instead, it feels like we're moving toward a future of casual misogamy as if Mad Men is an inspiration rather than a look into a flawed past.

Monday, January 18, 2016

Define Entertainment

When I first started this blog, it was because I needed somewhere to vent. The media is the worst!

I've changed my view. It's not the media that's the worst. It's the people who excuse the media for their idiotic stunts.

In 2010, Paul Henry, a morning show host resigned after allegations of racism (there had been various other problems with him). Wikipedia has this to say on the matter:
Henry's resignation polarised the New Zealand public, with supporters claiming he was a victim of political correctness, and critics accusing him of pandering to the lowest common denominator.
Remembering that "political correctness" roughly translates to "opportunity to not be a dick". Paul Henry then went to Australia, offended some people there (by stating that Asylum Seekers should 'starve to death') before coming back to NZ. He then ended up back in NZ TV.

At the time there was some outrage. BUT there was also plenty of fans happy to have him back on TV. Apparently "he's just funny" counteracts any social responsibility he has not to push bigoted views to the NZ public in his rather privileged position.

In 2009, the long running Australian show "Hey, Hey, It's Saturday" did a whole black face skit that the guest, Harry Connick Junior, was horrendously offended by. The rationale for the skit? "It's not meant to be racist". That was 2009 people. Are Australians really that clueless about racism?

Fast forward to today. Radio stations, notably "The Edge" here in New Zealand, continually publish "articles" that seem designed to condescend or normalize idiocy. Such articles include things like "Does that celeb who did an interview last night look sick to you?" probably looking to ride the coat tails of the recent celebrity deaths (Lemmy, David Bowie and Alan Rickman).

And in saying all of that, comedy has, for a very long time, been an avenue to creating conversations about awkward things. So sometimes it needs to offend.

The question is, what is entertainment? I don't think a reinforcement of society's worst traits should be excused with a "light entertainment" tag. It's not okay to be racist and/or sexist and then say "just joking" so why is it excused in the media?

There's also an odd sense of a normalizing of stupidity. I guess this has been going on for a long time and is often referred to as "pandering to the lowest common denominator"; although I think this tends to miss something too. It can be hoped that bigotry is not a common denominator. Stupidity also fits this category. Humour in the form of watching people getting hurt (Funniest Home Video type shows), for example, puts my teeth on edge. It's not entertaining. It's gruelling. I've seen other people cringe when they've watched it. Basically, the "common dominator" reasoning needs to be taken out back and shot in the head.

Which then makes it a reflection on ourselves i.e. we consume this so this is what is provided OR this is what the media thinks of us i.e. Hahah! Black face! in which case, we should be demanding better.

Thursday, December 17, 2015

Notes About IPTables

This is going to be a completely nerdy post. You've been warned.

I recently changed my Internet connection at home to fibre! Which is brilliant and fantastic and ... I'm cheap. So instead of going out and buying a router, I decided instead to use a pi to do it. I landed on my Banana Pi with a USB Ethernet adapter only to find that it was SLOW. Damn USB and Ethernet.

So I then brought a BPi-R1 to deal to the problem. I've still saved some money (I think a off the shelve jobbie was around the $200 mark whereas the BPi-R1 cost me around $150. I didn't realise the New Zealand dollar had dropped against the US dollar). Whoops. Thus, this post. I gain some value back by learning something:

What this is all leading to though is the fact that I finally had to learn a little something about IPTables. If you've never heard of it, this post might not be for you. It's the frontend to the firewall (netfilter) in Linux.

It was a bit of a struggle with very few explanations being around. So here's my attempt to get rid of some of that pain:

Anyone who tells you that it's easy is lying. It's not easy. The information out there is often inconsistent. For example, the terms "table" and "chain" seems to get confused and used interchangeably. So let's get this straight. A table is not a chain and a chain is not a table.

The tables I needed to concern myself with were the filter table and the nat table.

The next really confusing bit. LOADS of pages talk about using the command:

 iptables -L

This doesn't list out all of your rules (and the output is pretty much useless anyway). So here's the thing... When you use the iptables command, and you don't specify a table, you're using the filter table. So 'iptables -L' ONLY lists out the filter table.

Instead, you're much better off using:

 iptables-save  

Or, if you want it to be a little less... blerg (the comments and information you don't need), use:

 iptables-save | sed 's/\[[0-9]*:[0-9]*\]// ; /^#/d'  

So for all of this talk about tables and chains... what does it really mean? Well... the table names don't really seem to mean a great deal. They give you a sense of the purpose of the chains BUT the way traffic goes though the various set of chains isn't implied by the table name.


In this diagram, PREROUTING is part of the nat table. INPUT, FORWARD and OUTPUT are part of the filter table and POSTROUTING is part of the nat table. Confused?

Let's go through that diagram. When a packet comes in, a routing decision is made. Is that packet intended for the machine the firewall is on? If it is, it goes to the INPUT chain. Otherwise, it goes to the FORWARD chain. When a packet is going from the firewall machine out, it goes through the OUTPUT chain.

We need to make some decisions. When we're talking firewalls, we're talking about protection. In my case, I chose to only protect my network from the Internet. I don't really care what traffic is going out.

Which means I've already already got my first 3 rules! (They're not technically rules - they're policies. But let's not let that get in the way of our sense of accomplishment)

 iptables --table filter --policy INPUT drop  
 iptables --table filter --policy FORWARD drop  
 iptables --table filter --policy OUTPUT accept  

By default, drop packets going to the INPUT chain (if not matched by a rule), drop packets being forwarded (ditto), and 'accept' (allow to pass through) anything going outwards.

For the rest of this document, I'm going to be using:
  • lanDev as my LAN device in the rules. This will normally look something like eth0 or eth1 etc. On the Banana Pi it's eth0.102 OR br0. Totally irrelevant. My set up is likely to be different from yours.
  • wanDev as my WAN (Wide Area Network i.e. Internet) device.
  • 1.1.1.1 as an example internal IP address. In reality, this will normally be something along the lines of 192.168.0.1, 172.16.1.1 or 10.1.1.1.
  • 2.2.2.2 will be our external IP. If you've got a dynamic IP, read right to the end of this post.
  • I've kept the long form of the commands to make it clear what we're doing i.e. all of the examples specify the table they're using rather than relying on the default.
This hopefully makes it a little clearer.

Next we need to allow the computer to talk to itself (it does this more often than you'd think on the loopback - lo - device).

 iptables --table filter --append INPUT --in-interface lo --jump ACCEPT  

And because I don't want to limit the server from the internal network, I also do the same thing with my internal network:

 iptables --table filter --append INPUT --in-interface lanDev --jump ACCEPT  

If I wanted to harden this up a bit, I could:
  • Lock down the IP addresses that can talk to the server:

    iptables --table filter --append INPUT --source 1.1.1.0/24 --in-interface lanDev --jump ACCEPT 

    This allows anything with an IP address between 1.1.1.0 and 1.1.1.255 access to the router.

    If I wanted to lock down those addresses a little more, I could use the "iprange" extension (I tend to think of it as a module given that you use '-m' on the command line to use an extension):
     iptables --table filter --append INPUT --match iprange 1.1.1.100-255 --in-interface lanDev -j ACCEPT   
    
  • Or ONLY give access to particular services, i.e. ssh:
    iptables --table filter --append INPUT --source 1.1.1.0/24 -in-interface lanDev --protocol tcp --dport 22 --jump ACCEPT 
And finally, we want the Internet to be able send information back to the router BUT only when we've initiated a connection of some kind (when visiting a web page, I want the web page to be able to load):

  iptables --table --append INPUT --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT   


Here we use the conntrack extension to make a match on state. The router can tell which packets are part of an established or related connection.

Let's set up NAT (Internet Sharing). Everyone clear on what NAT is and does?

The needed rules are:

 iptables --table nat --append POSTROUTING --out-interface wanDev --jump MASQUERADE  
 iptables --table filter --append FORWARD --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT  

The first rule does all the real magic. When a device on your local network sends out a packet, the IP address is changed to the external ip address. When a packet comes in, the router then changes the IP address back to that of the local device.

Remember how we dropped everything by default in the forward chain? The second line there just does what we did with the INPUT chain i.e. if it's part of an established or related connection, let it through.

The other thing you'll probably need to check is that IP forwarding is enabled in the kernel. Edit /etc/sysctl.conf and uncomment or add the following line:

 net.ipv4.ip_forward=1  

This will take effect when you reboot. To enable it immediately, run the following:

 sysctl net.ipv4.ip_forward=1  

Now we have a router that's relatively locked down and is doing some routing!

You may have read previous posts where I talk about CG-NAT. The evilness of it meant that I couldn't use my system the way that I normally do. Which is to say that if I'm on holiday or working somewhere away from home, I normally remote into my system. This saves me maintaining multiple development environments.

I spent another $50 on getting a static IP. Basically, I always have the same address when connecting to the Internet and it gets me past the CG-NAT stuff. Neat! (Except that it probably makes it a bit easier to track me).

What if I want to run a service on my router? I would need to open a port:

 iptables --table filter --append INPUT --protocol tcp --dport 80 --jump ACCEPT   

To enable remote access to my system, I needed port forwarding. Say I want to connect to a web server externally from the Internet. To do this, I need the following rules:

 iptables -t nat -A PREROUTING -i wanDev -p tcp --dport 80 -j DNAT --to 1.1.1.1:80  
 iptables -t filter -A FORWARD -p tcp -d 1.1.1.1 --dport 80 -j ACCEPT  

This tells the PREROUTING chain to send traffic on port 80 coming in from the wanDev device to port 80 on 1.1.1.1 and the FORWARD chain to accept traffic coming in on port 80 destined for 1.1.1.1.

I could (but it seems pointless) only accept NEW connections and rely on the earlier "ESTABLISHED, RELATED" rules to continue the connection. To do this, I would replace:
 -A FORWARD -p tcp -d 1.1.1.1 --dport 80 -j ACCEPT  
with:
 -A FORWARD -p tcp -d 1.1.1.1 -m conntrack --cstate NEW --dport 80 -j ACCEPT  

The other thing to note is that the external port doesn't have to match the internal one. Imagine I want to listen to ssh connections on port 12345 but I want to still run ssh connections on 22 internally. In which case, I would use the following rules:

  iptables -t nat -A PREROUTING -i wanDev -p tcp --dport 12345 -j DNAT --to 1.1.1.1:22   
  iptables -t filter -A FORWARD -p tcp -d 1.1.1.1 --dport 22 -j ACCEPT   

Not content with that, I went a step further. The problem with all of this is that the forward porting doesn't work from inside my own LAN. That means that there are differences to how I access things externally.

Imagine you're using a Dynamic DNS service like noip.com. That way you don't need to memorize your IP address (it's an absolute godsend if you have a dynamic IP). From inside my own LAN, I'd be able to use that address to access the various services (the machine I ssh into isn't the same machine hosting the web page for example).

To do this, I need the following rules:
 iptables -t nat -A PREROUTING -p tcp -s 1.1.1.0/24 -i eth0.102 -d 2.2.2.2 --dport 80 -j DNAT --to-destination 1.1.1.1:80  
 iptables -t nat -A POSTROUTING -p tcp -s 1.1.1.0/24 -o eth0.102 -d 1.1.1.1 --dport 80 -j SNAT --to 2.2.2.2:80  
 iptables -t nat -A PREROUTING -p tcp -s 1.1.1.0/24 -i eth0.102 -d 2.2.2.2 --dport 12345 -j DNAT --to-destination 1.1.1.1:22  
 iptables -t nat -A POSTROUTING -p tcp -s 1.1.1.0/24 -o eth0.102 -d 1.1.1.1 --dport 22 -j SNAT --to 2.2.2.2:12345  

Uh oh! We've actually needed the external ip address! Which, if you have a dynamic IP,  you don't always know. I'll address that in a second.

We need to save our rules and make sure they are loaded. These instructions are for a debian based system though I think Arch Linux uses the same method. I'm pretty sure this won't work on a Redhat based system where they don't use the /etc/network/interfaces file for network configuration.
Save the rules:
 iptables-save > /etc/iptables.conf  

To tell the network configuration to restore those rules before bringing up the network, edit /etc/network/interfaces and add the line:
 pre-up iptables-restore < /etc/iptables.rules  


If you have a dynamic IP and you're using port forwarding from inside your own network:

This is where things get tricky. The problem is we won't know what the IP address will be until AFTER the interface is up. To get around this (I'm not sure if this is the recommended way, but it's a solution) we'll need to create a script.

By now you should have a file in /etc called iptables.conf. We need to remove the stuff that's reliant on having an external address to it's own file. We should be able to do this by running:
  echo '*nat' > /etc/iptables-extIP.rules ; grep '2.2.2.2' /etc/iptables.rules >> /etc/iptables-extIP.rules && sed '/2\.\2\.2\.2/d /etc/iptables.rules -i  ; echo 'COMMIT' >> iptables-extIP.rules

This:
  • Copies out all of the lines relevant to the external IP address to a file called /etc/iptables_extIP.rules and makes that file suitable for iptables-restore.
  • Removes those entries from /etc/iptables.rules.
Now we need to change that external IP address to something we can easily search for:
  sed 's/2\.2\.2\.2/EXT_IP/' /etc/iptables-extIP.rules -i

And finally, we need to run create a script to fill in these bits after the WAN device is up. Create a script in /etc/network/if-up.d/extIPRules with the following contents:

 #!/bin/bash  
 wan_device="wanDev" #replace with the name of your external device i.e. ppp0  
 iface=$1  
 if[ "$iface" == "$wan_device" ] ; then  
    ext_ip_addr=$(ip addr show $wan_device | grep inet | egrep '([0-9]{1,3}\.){3}[0-9]{1,3}' -o | head -n 1)  
    sed "s/EXT_IP/$ext_ip/g" | iptables-restore -n  
 fi  
(This is untested. If you do try this and it works, please let me know in the comments.)

Make it executable:

 chmod +x /etc/network/if-up.d/extIPRules  

And it should "Just Work™". If it doesn't, let me know in the comments and I'll try and update this post with any corrections that are needed.

Monday, November 30, 2015

Let's Talk About Education!

Education. The portfolio that sees MP's in hot water more often than not. The election promise that gets people salivating. The "thing" that we're all passionate about!

Lately I've been finding myself highly sceptical about IT training in general. Particularly for kids. On the electronics side of it, we get:

"Here's a robot kit! Build a robot"

And computers? Let's learn pointers everyone! It's a.... Wait.... what? What problem do they solve?!?

Here's the problem. They're prescriptive. If we think education hasn't changed since the 60's (how often do education presentations start off with a photo of a classroom from the 50's and a photo of a classroom now?), go into, ironically, one of the fastest moving fields and see how it's taught.

It's appalling.

The problem with kits? They're prescriptive. You attach the wheel here, you plug this bit in here and away you go! You've just made yourself a robot? How does it work? you ask. Well... shut up and play with your robot! Look! You can program it!

The net effect is that you get quite particular character types striving in the field. The real valuable people are those who can hear/see it prescriptively, apply it exploratory (apply it to a problem) and think creatively (taking what they know and apply it in novel ways). BUT the ones who succeed only need hear the words "industry standard" or "best practise" and that is the solution.

There's another problem that comes with this: Gender inequality. Yep. That old chestnut. The way that IT is taught encourages a gender inequality. That's the way things have always been done.

Going back to my programming example above, it could be taught in the following way:
Say we need to store similar information (we'll call these "blocks" of information). Like a database. Only we don't want to keep going back to the database. We're not sure how much information we're going to store. It depends on the data. Dealing with arrays is hard work. The stupid things want to be a size and stay that size in which case we have to create an all new array with added information.
How would you handle this problem?

And then bring it back to a solution. Okay, so some really clever folk came up with this idea of "pointers". At the end of the blocks of similar information, what if you could "point" to the next block of similar information? That way, if you need to add another block, you just get the pointer on the end of the last block to point to it.
Easy. Instead of the time spent having to reexplain pointers because there's no real understanding of why you'd use them, a deeper understanding is created because it's applied to a real problem. It allows time for a few novel ideas. It describes "linked lists" which is awesome for moving into Python. It encourages discussion.

If you don't know what a pointer is, don't worry. They're not used terribly much; most modern programming languages have gotten rid of them all together.

And electronics? What if you could put a bunch of stuff on a table, ask the kids what they want to build (elaborate. If it's a robot, what do they want the robot to be able to do?), and facilitate it in terms of learning how to find the information to do what they want. This is generally what we (makers) do as adults anyway. I've been regaled with pictures of food a friend of mine made in his homemade sous vide. AND IT'S AWESOME!!!

Confident learners, not ones who can recite what has been said to them, but rather, those who are confident to not only learn but also apply their knowledge, creates confident people. When my monitor blew up a few days ago, while my first reaction was a curse at the heavens, my second was to pick up a screwdriver to have a look. A few forums later, I identified the capacitor that was likely blown (I had missed the leak on my visual inspection), and ordered the parts needed (a fuse and a capacitor - well less than $10 worth of parts to get my 10 year old, still worth around $400 monitor working). While I was thinking about it that day, I asked myself "What do people normally do in this situation?". The privilege of empowerment struck me. Most people would throw the monitor out and go out looking for a new one. I want other people to have the empowerment to be able to solve the problem.

In other words... if the education isn't empowering... is it really education?