This article is the second in a series (see part 1 here).  Please see HA Network for the first part in setting up the network topology to be highly available.

It’s all good having a redundant network design, but putting web servers and the like on our hypervisors doesn’t make them redundant.  In the event where there’s a failure on one of our servers, all virtual machines on that server will die.  Looking at our previous network design, we can see a failure on a web server or database server would cause service outage.

In this example, I’m going to talk about a few pretty cool pieces of software that you can use to make a highly available (HA) service stack for hosting Apache web sites from a MySQL data store.  This post isn’t going to be about all the in’s and out’s of how to accomplish, but hopefully it will answer some questions you might have when setting out to accomplish this task as well as point you to the appropriate articles to help you achieve what you’re after.

Nginx load balancing proxy

Lets look first at an example of a previous example.  Lets take a simple network where we have a web server with a client.  The A record for www.example.com points to our web server at 192.168.0.10 (of course we’d use NAT on the firewalls in my example to expose it to the internet).

From my previous post, we can tell that the network topology to reach our web server on 192.168.0.10 is highly redundant, if the server itself, or the physical machine it’s hosted on in the case of virtual machines does die, we can no longer serve web pages.

To solve this problem, we’re going to create multiple web servers. The A record for www.example.com is now no longer going to be pointing to the web server itself, but a proxy server (I recommend Nginx.) Now you can see that if either web server dies, our proxy server is able to start handing out requests through the other server. This method is also recommended because it mitigates load from a single server. If your web site got more popular, we can just start scaling out and adding more web servers into the mix to handle the load. Now, I know what you’re thinking, and yes, we have just moved the single point of failure from the web server to the proxy server, but please read on to find out how to protect that host from failure.

Heartbeat, keep your servers beating

In the previous example, we had a server that was a single point of failure.  If the proxy server in this case died, then our web site would go down.  To plan against the failure of this machine, you can set up a tool such as Heartbeat.  An example works like the following, your proxy server above is running the Nginx daemon handling your client’s requests, but it does so using a virtual network adaptor where it’s IP address 192.168.0.10 sits.  You put a second server in here in the mix with the same Nginx daemon and the same configuration, but it’s not running or serving anything, this is known as our slave server.  The slave server is sending heartbeat messages to the master server.  In the event that the master server stops responding to the slave’s heartbeat messages, it assumes the master is down and will create a virtual adaptor assuming the working IP address (192.168.0.10 in our example) and brings up the master services.

I use heartbeat extensively through my HA configurations. Not only for allowing services to be taken up by a slave in the event of a master failure, but also where I have clusters (for example, a MySQL cluster). When the slave assumes the master’s IP address, it’s handy to write a little script/service here to stop any replication and assume the master’s role.

ProTip: Managing configuration across your servers when you start creating multiple instances for redundancy can start to get out of hand very quickly. Sometimes you can change a configuration on one server and forget to do it on another. I suggest using a tool like Puppet to manage the configuration on your servers for you

ProTip: From experience, I’ve had situations in failover testing where the failure of a core switch will cause heartbeat to stop receiving heartbeats and fail the servers over, even though the physical and virtual machines themselves are fine. If you’re running the (non-rapid) Spanning Tree Protocol (STP) on your network, I suggest making the timeout for Heartbeat about 45 seconds. This should be enough time to allow for STP convergence before it assumes it’s partner is dead.

NFS File Store

In the example above, we’ve got multiple web servers. As I’ve said, this setup could be used to host a HA WordPress site. The problem is that when content that sits on the file system (such as an image or theme uploaded, wordpress upgrade, etc) takes place, it will no longer be in sync with the other web servers. For us, hosting the wordpress installation from an NFS mount point worked fine, which begins a thought on how to make this NFS server highly available.

Just like the previous example, we’re going to use Heartbeat to make sure we’ve got a master and slave, and that when the master fails, the slave will start hosting the NFS services. There’s only one more added piece of complexity here. If the master fails, the slave has none of the data that was stored on the master. To get around this, I’m using a tool called DRBD. DRBD allows a block device to be created and synced across multiple hosts. When you write to this device, data is replicated on the slave too. That way when the master dies and the slave takes it’s roll, it will have all the data that existed previously on the master. A good tutorial to set this up can be found HERE

ProTip: Once again from experience, when everything is not working as expected, having the slave be promoted is a horrible thing. If the data has not been replicating for some reason over the past few months and the slave gets promoted with data that’s a few months old, it can be a horrible, horrible thing. I suggest running a tool like Nagios on your stack to monitor EVERYTHING you can think of. A good way to check if your DRBD servers are in sync is to look for the term UpToDate/UpToDate in /proc/drbd.

MySQL Cluster

There are two ways of running your MySQL server in high redundancy. One simple method using tools you’ve already used is to have the MySQL data store running over DRBD and have Heartbeat keeping it in track. This is pretty simple to set up and I’m running it on one of my sites without a problem.

The problem with running MySQL in this sort of setup is scale. Putting your web servers behind a load balancing proxy is a good first step in allowing for you to slot more servers in your solution and starting to scale out. Once the bottleneck moves to your MySQL server, running a single active/passive pair over DRBD won’t scale out, only up (more expensive, faster hardware).

The second time I had to set this up, I chose to run my servers in a MySQL cluster. This means that whenever a transaction is committed on the master, it is sent to the slave to commit as well. The advantage with this is that it allows you to spread out SELECT queries among your slaves instead of running everything on your one server.

Word for the wise

Now we have a highly available network stack in place with highly available system services, we can sleep better knowing that if a system outage were to occur, we have our insurance policy in place knowing that in about 50 seconds, a network issue can converge and services can be automatically moved over to hot standby machines. This does not however give you an excuse to not have a backup strategy in place.

If you’re doing backups as an afterthought, I can recommend setting up an OpenIndiana machine running ZFS on it. Setting it up like I have off site with rotating snapshots. At our work we do nightly backups and for us it’s as simple as doing a database dump, then rsync’ing everything over to this remote ZFS machine to get snapshotted, feeling safe I can access old data from 6 weeks ago (my nightly snapshot period.)

What’s to come?

If you’ve got this far, Thanks for reading and I hope this post pointed you in the right direction to help build highly available services. I think more and more in today’s world people are expecting IT infrastructure to be always up and available. If you don’t have standby servers available, sooner or later, you’re bound to have unhappy customers. I’ve always wanted to build a disaster recovery site, with a highly active database so that in the event of an entire site failing over (think natural disaster, people errors and the like) a site can fail over to another physical disaster recovery (DR) location.

In one of my last little tasks at work, I was asked to eliminate single points of failure in the software and hardware stack without spending a fortune on hardware or software licenses. During the process of ensuring high availability (HA), I realized that many small companies might have similar need, but with more pressing tasks and limited man hours, without a post that talks about all the issues and solutions in one place, many companies and organisations tend to leave single points of failure living with the chance that they’re not going to fail any time soon.

I’ve wanted to write this blog post for a while. After you’ve finished reading this blog post, you should have the knowledge to be able to eliminate the single point of failures in hosting a WordPress website. While I’ve chosen WordPress to be the demonstration of this post, the concepts will work with any apache/mysql LAMP stack software. During the course of this tutorial, I’ll run you through it in two parts. The first part is talking about setting up the physical hosts and topology (using Juniper EX2200 switches, SRX100 border firewall’s and ESXi (free) hypervisors for the software stack.) The second part is talking about setting up the software stack to deliver our LAMP stack in a highly redundant fashion.  However what I won’t be doing is providing complete configuration examples.  Instead, please consider this post as an overview to help enlighten you and link you to more specific information to help you set this up in your own environment.

This blog post is split up in two parts, this first post in the series talks about setting up the network infrastructure, the second talks about setting up the software stack.

Part 1 (physical topology)
- Network overview
- Setting up dual Switches
- Setting up ESXi network cards.
- Setting up SRX border firewall’s.

Part 2 (software stack)
- Nginx load balancing proxy
- Web Servers.
- NFS File Server
- MySQL cluster.

Part 1, Physical Topology

If you’re reading this, I’m guessing your current network topology looks something like

mine used to.  You have a single internet connection, single router/firewall, single switch and a bunch of hosts hanging from that switch.  In the event of a system failure, your system administrator (me in this case) will have to hop in a cab and rush to the server room to fix the problem.

The goal with this tutorial is to attempt to help your administrators sleep at night.  We will eliminate every single point of failure such that in the event of a system outage/failure, the system can self recover with at most a minute of unscheduled down time.

Physical Switches

We’ll be replacing the single switch (in my case, unmanaged old gigabit switch) with a pair of managed switches.  Because our bandwidth requirements in this site wasn’t terribly demanding (simple database server, few web, mail servers and the like), a single gigabit Ethernet link to all hosts was all that we required.  If you’re in the same boat I was, I can suggest a pair of Juniper EX2200’s.  If however, you’re going to be pumping some more bandwidth intensive applications through your network (thus require more then gigabit Ethernet connections to the hosts) or have the need for more then a single VLAN and intra-VLAN routing is required (that’s all outside the scope of this tutorial.)  I can strongly recommend you start looking at the EX4200 model switches (set up in a virtual chassis), which can do all your highly available layer 3 IP routing and support multi gigabit Ethernet to your hosts by spanning Ethernet channel across both physical switches.

Active/Passive Switch Design?

Ok, so this heading is a lie, but let me explain.  In my switch design with the EX2200’s, I’m using aggregate Ethernet 802.3ad Etherchannel between my two switches. I’ve opted to use 4 physical ports in my 24 port switches (ge-0/0/20 to ge-0/0/23) to give me a 4Gbit/s backbone between them. Obviously with gigabit Ethernet right through the network this isn’t much bandwidth, so the idea is to keep as little data traveling that link as possible (network broadcasts only hopefully!)

First, the following configuration configures the aggregate Ethernet link between the switches:

chassis {
    aggregated-devices {
        ethernet {
            device-count 1;
        }
    }
}

[interfaces]
    ge-0/0/20 {
        ether-options {
            802.3ad ae0;
        }
    }
    ge-0/0/21 {
        ether-options {
            802.3ad ae0;
        }
    }
    ge-0/0/22 {
        ether-options {
            802.3ad ae0;
        }
    }
    ge-0/0/23 {
        ether-options {
            802.3ad ae0;
        }
    }
    ae0 {
        aggregated-ether-options {
            lacp {
                passive;
            }
        }
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members all;
                }
            }
        }
    }

The only thing you have to watch out for, is that on at least one switch, you set the lacp mode to be active.

Setting up ESXi Network

In my environment, I’m using the free version of ESXi 4.1. I understand that you may wish to be connecting linux hosts directly. If you’re directly connecting linux hosts, I recommend you look at creating a Bonding Adaptor.  Thought without the more expensive EX4200’s, we’re best sticking with an active-backup setup and should stick with mode=1.

For the ESXi configuration, it’s pretty simple. We connect two NIC’s to the switches. A primary NIC to the primary switch, and a secondary NIC to the secondary switch.

The steps on configuring the vSwitch (virtual switch) are pretty simple. Virtual machines on these ESXi hosts won’t have to do anything special once this setup has been done to take advantage on the physical machines, they’ll just take advantage of our HA network topology.

First, ensure that we have two NIC’s setup on our vSwitch.

Next, look at the following configuration properties I’ve made to the NIC teaming information.  This basic configuration will ensure that when the link on vmnic0 is available (our active switch), it’ll use it.  When the link becomes unavailable, it will fail over to vmnic1.

Setting up border firewall’s

The last step we’ve got in our highly available infrastructure here is our border firewall’s. Please bare with me on this section, it is the most complicated and there are a few technologies introduced. If there is something I don’t explain completely, please feel free to leave a comment below and I’ll try and explain it better.  I expect you’ll have to jump back and forth between Wikipedia and this article to fully understand what it is we’re doing. Explaining the concepts behind BGP and autonomous systems is beyond the scope of this article.

In my role, I replaced an active/backup GNU based firewall solution (backup being run down to the data center as fast as possible and swap the cables over) with two Juniper SRX100’s configured in an active/passive configuration under a chassis cluster (see SRX100 high availability deployment guide at http://kb.juniper.net/InfoCenter/index?page=content&id=KB15669). We had get a second internet connection installed, then make sure that any one link (or firewall itself) can die and still have the system self recover. There are two major tasks that has to happen to make this gateway highly available. Firstly, our internal network hosts will be using one of these firewall’s as their default gateway. If the link to the primary switch, or the primary firewall itself should die, we still want the network hosts to be able to reach their gateway. We will achieve this with “redundant Ethernet” on the cluster. If you’ve come from the cisco networking world before, picture something like this to be like VRRP for the inside hosts. If the main link fails, the MAC and IP address will float over to the other physical port.

Let me give a bit more detail to our new example network topology here in this example so you can gain a better idea of how these settings work.

For the internal gateway address to fail over to the other switch should your link off Firewall1 die, You’ll want to make the following configuration

chassis {
    cluster {
        reth-count 2;
        redundancy-group 0 {
            node 0 priority 100;
            node 1 priority 1;
        }
        redundancy-group 1 {
            node 0 priority 254;
            node 1 priority 1;
            preempt;
            interface-monitor {
                fe-0/0/1 weight 255;
                fe-1/0/1 weight 255;
            }
        }
    }
}
interfaces {
    fe-0/0/1 {
            fastether-options {
                redundant-parent reth1;
            }
    }
    fe-1/0/1 {
            fastether-options {
                redundant-parent reth1;
            }
    }
}

You can see redundancy-group 1 is monitoring the local interfaces going back to the switches.

For the dual WAN links, I won’t go into much detail, but you’ll want to ask your ISP for a second internet connection. Some providers offer a cheap link that they only charge you for once you start flowing data over it (sometimes called a Shadow link). This is perfect as you can flow all your traffic through your primary internet connection, then on failure of it, you’ll move your traffic through the secondary. If you wanted complete redundancy, you could apply for a domain independent subnet (has to be a class C to advertise on world BGP tables) and your own ASN. This will let you use two different internet service providers.

In my case, I’m creating a redundant connection using the same ISP, so I’ve asked for a private ASN to be allocated (see http://en.wikipedia.org/wiki/Autonomous_System_(Internet) ).

For a small network such as ours (especially using the small base level SRX’s) you’ll want to ask your provider to advertise only the default route to you. In turn, you’ll advertise your network’s address space on both connections. On the event that a link dies, the BGP peer on the other end no longer receives updates from you and will no longer attempt to route to it.

The following configuration extract shows how we’d configure out SRX firewall’s to peer with our ISP’s routers. Things of note is that our primary link (the one on the left) has a lower metric-out then the shadow link, meaning a lower MED attribute is sent to our ISP and thus inbound traffic will, by preference use the main connection. The preference values under neighbor will determine the preference we will send traffic under that connection for outbound traffic.

routing-options {
    autonomous-system 64512
}

protocols {
    bgp {
        group ISP {
            metric-out 50;
            local-address 1.1.1.2;
            import ISP-in;
            export ISP-out;
            neighbor 1.1.1.1 {
                preference 170;
                peer-as 123;
            }
        }
        group ISP-shadow {
            metric-out 100;
            local-address 1.1.1.6;
            import ISP-in;
            export ISP-out;
            neighbor 1.1.1.5 {
                preference 180;
                peer-as 123;
            }
        }
    }
}
policy-options {
    policy-statement ISP-in {
        term default-in {
            from {
                route-filter 0.0.0.0/0 exact;
            }
            then accept;
        }
        term block {
            then reject;
        }
    }
    policy-statement ISP-out {
        term tnziPublic {
            from {
                protocol direct;
                route-filter 2.2.2.0/26 exact;
            }
            then accept;
        }
    }
}

A real quick run down, we all know what DNS does yeah? It translates domains like www.scottyob.com into IP addresses like 112.140.183.97. A DNS server has a job of translating these domain names to the IP addresses we can use.

Now, when it comes to Google DNS, if you believe in the propaganda http://code.google.com/speed/public-dns/ where Google DNS is said to

• Speed up your browsing experience
• Improve your Security.

What google doesn’t tell you is that it interferes with DNS servers that might try and give you a server that’s close to your home. I’ve been using Google DNS for months here at home, but only just have I decided against using it, and I’ll run through exactly what causes some performance issues.

I was trying to watch a program on iView on TPG’s internet connection. Now, TPG don’t have the best international links on peak times, so I first started getting frustrated at them for not letting me watch my iView program with buffer lags on my 8Mbit plan. I checked my Signal to Noise ratio and dropped packets, etc.. it was fine. I checked the bandwith going out on my ppp interface on the router to see if I was maxing out the net connection at home, but nope, that was fine too, so the problem must have been with TPG.

Looking at what was going on, I did a little traceroute to www.abc.net.au

traceroute: Warning: www.abc.net.au has multiple addresses; using 125.252.224.73
traceroute to a1632.g.akamai.net (125.252.224.73), 64 hops max, 52 byte packets
1 10.1.1.254 (10.1.1.254) 1.919 ms 1.260 ms 1.202 ms
2 * * *
3 202.7.173.17 (202.7.173.17) 27.056 ms 26.317 ms 26.693 ms
4 syd-sot-ken-crt1-ge-5-1-0.tpgi.com.au (202.7.162.173) 26.283 ms 26.884 ms 26.100 ms
5 ix-11-1-0-507.tcore2.tv2-tokyo.as6453.net (116.0.88.21) 153.325 ms 135.736 ms 126.568 ms
6 if-14-0-0-1720.core1.tv2-tokyo.as6453.net (209.58.61.121) 1481.461 ms
if-1-0-0-1715.core1.tv2-tokyo.as6453.net (209.58.61.125) 298.270 ms
if-14-0-0-1720.core1.tv2-tokyo.as6453.net (209.58.61.121) 282.141 ms
7 if-10-0-0-981.core3.hk2-hongkong.as6453.net (116.0.82.85) 208.232 ms
if-5-0-0.core3.hk2-hongkong.as6453.net (116.0.82.1) 199.170 ms
if-10-0-0-981.core3.hk2-hongkong.as6453.net (116.0.82.85) 206.027 ms
8 vlan31.icore1.hk2-hongkong.as6453.net (116.0.82.18) 219.937 ms 204.963 ms 196.273 ms
9 80.150.169.25 (80.150.169.25) 333.519 ms 306.138 ms 307.031 ms
10 80.156.224.6 (80.156.224.6) 358.402 ms 320.029 ms 349.133 ms
11 a125-252-224-73.deploy.akamaitechnologies.com (125.252.224.73) 332.188 ms 331.008 ms 314.833 ms

Looking at this traceroute, the first thing I thought was “Why on earth is ABC hosting it’s website in hongkong (or so the traffic has to go via hongkong.)? Immediately reject ABC would do this and I blame TPG’s stupid routing decisions.

Looking at the IP address further http://www.dnsstuff.com/tools/ipall/?tool_id=67&token=&toolhandler_redirect=0&ip=125.252.224.73 It looks like it’s hosted in Singapore, and some googling shows akamaitechnologies is the web host for ABC.. ok, so what is going on here?

then I did a little DIGging around on the ABC domain and check this out.

Using Google DNS (8.8.8.8, hosted in America)

; <<>> DiG 9.6.0-APPLE-P2 <<>> www.abc.net.au @8.8.8.8

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16084

;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:

;www.abc.net.au. IN A

;; ANSWER SECTION:

www.abc.net.au. 882 IN CNAME www.abc.net.au.edgesuite.net.

www.abc.net.au.edgesuite.net. 21581 IN CNAME a1632.g.akamai.net.

a1632.g.akamai.net. 2 IN A 63.150.131.41

a1632.g.akamai.net. 2 IN A 63.150.131.33

;; Query time: 161 msec

;; SERVER: 8.8.8.8#53(8.8.8.8)

;; WHEN: Sun Jul 3 00:37:00 2011

;; MSG SIZE rcvd: 135

And using TPG’s DNS servers (203.12.160.35)

macshell:~ scott$ dig www.abc.net.au @203.12.160.35

 

; <<>> DiG 9.6.0-APPLE-P2 <<>> www.abc.net.au @203.12.160.35

;; global options: +cmd

;; Got answer:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11673

;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0

 

;; QUESTION SECTION:

;www.abc.net.au. IN A

 

;; ANSWER SECTION:

www.abc.net.au. 523 IN CNAME www.abc.net.au.edgesuite.net.

www.abc.net.au.edgesuite.net. 12274 IN CNAME a1632.g.akamai.net.

a1632.g.akamai.net. 10 IN A 202.7.177.66

a1632.g.akamai.net. 10 IN A 202.7.177.83

 

;; Query time: 29 msec

;; SERVER: 203.12.160.35#53(203.12.160.35)

;; WHEN: Sun Jul  3 00:38:20 2011

;; MSG SIZE  rcvd: 135

 

So there we go, hosted in Sydney Australia.. So how does the traceroute compare?

macshell:~ scott$ traceroute www.abc.net.au

traceroute: Warning: www.abc.net.au has multiple addresses; using 202.7.177.83

traceroute to a1632.g.akamai.net (202.7.177.83), 64 hops max, 52 byte packets

1  10.1.1.254 (10.1.1.254)  1.711 ms  1.134 ms  1.100 ms

2  * * *

3  202.7.173.17 (202.7.173.17)  27.047 ms  26.072 ms  26.502 ms

4  syd-sot-ken-ak2-83.tpgi.com.au (202.7.177.83)  26.540 ms  26.835 ms  26.540 ms

Different nameservers can be set up to resolve to different addresses based on geographic positioning, in more of a first in best dressed kind of effort.  So if you’re using Google DNS servers (8.8.8.8), it so happens that because this is in America (check out their IP address, CA), then I started getting update servers and the like on DNS that were closest in latency terms to America then Sydney Australia.

Simply by changing from Google DNS servers to my ISP’s, I get healthier latency, and am no longer sending traffic over saturated overseas links.  So how does this compare with ABC iView?  Well, check it out

And when I’m using my own ISP’s DNS

Cifs Share

CIFS (Common Internet File System), the protocol windows users for all it’s ‘windows file sharing’ is the method I’ll allow for my desktops and roaming computers to access files on the file server.

Before we begin, Make sure we install the CIFS kernal modules

# pkg install SUNWsmbs # pkg install SUNWsmbskr

next we issue this command to make sure it auto starts

# svcadm enable -r smb/server

I’ve decided for every day use, I want a data store on the server, so..

# zfs create datastore/homes # zfs create datastore/homes/scott

now, set up compression on my home directory

# zfs set compression=on datastore/homes

Time to do some setup so we can log into this share, I’ll make the box join the workgroup ‘home

# smbadm join -w home

To start sharing with windows boxes, I need to change the pam.conf file to generate windows passwords too. Add the line below /etc/pam.conf

other password required pam_smb_passwd.so.1 nowarn

reset the password for my user scott, then I’ll be able to authenticate with him

# passwd scott

The next step is setting up guest access. You may remember we created a media share datastore/media. We want to share that with guests to the network (on the trusted subnet anyway). Before we go ahead and set that up, we want to map the windows Guest user to the unix user nobody.

# idmap add winname:Guest unixuser:nobody

then we’ll allow guest access to our media box

# zfs set sharesmb=name=media,guestok=true datastore/media

I also want to make my home directory only accessible by me, so I’m going to own the directory

chown scott /datastore/homes/scott chgrp staff /datastore/homes/scott chmod 700 /datastore/homes/scott

and share it

# zfs set sharesmb=name=scott datastore/homes/scott

So there we have it, a media folder anyone can access, and a ‘scott’ share that I’ll need to authenticate with (HOME\scott user)

NFS Shares

Now we’ve got CIFS set up for our clients, I want to set up NFS shares for other Linux boxes on the network (at this point, only my router) to be able to access. The idea is that my router will have all the home directories on the FileServer (so it’ll get the advantages of snapshots, etc) we well as not being limited to the dying 60GB hard disk for torrenting and such things.

As mentioned, I’m only interested in NFS shares with my router at this point, so we’ll make sure my routers IP address (10.12.1.254) is restricted in the shares.

The first thing we want to try is setting up the nfs mount on our homes directory.

# zfs set sharenfs=root=@10.12.1.254,rw=@10.12.1.254 datastore/homes

Now, on my debian router box I want to see if I can mount this (assuming the directory /home2/scott exists on the client)

sudo mount -t nfs -o nfsvers=3 10.12.1.1:/datastore/homes/scott /home2/scott

and Ta-Da! My home directory is mounted. What I want to do now is to set up auto-mounts. That is, when a directory is accessed for my users home directory, it’d mount it on the fly.

First, install the autofs package

apt-get install autofs

Add the following line into /etc/auto.master

/home2 /etc/auto.home -–timeout=60

and the following into the file /etc/auto.home

* -fstype=nfs,rw,nosuid,soft,vers=3 server:/datastore/homes/&

the * and /datastore/homes/& do their magic by automatically mounting the required directory when needed (as long as the /etc/init.d/autofs is started)

Now lets add a place for our downloads

# zfs create datastore/media/downloads # zfs set sharenfs=root=router,rw=router datastore/media/downloads

I chose to just mount this using the automounter under the home2 directory, I added this to the /etc/auto.home file

downloads -fstype=nfs,rw,nosuid,soft,vers=3 10.12.1.1:/datastore/media/downloads

Pretty neat, now when you head to a directory that’s not mounted yet (like /home2/scott/) in the linux client, it will auto mount the required NFS volume and presto, we’ve got ourselves network storage.

Other posts to come in the series:
1. Selecting the hardware
2. Installing the Operating System
3. Setting up File systems & Snapshots
4. Allowing access through NFS & SAMBA
5. Setting up encrypted off-site backups
6. Configuring Windows & Linux clients to dump backup info to the FileServer
7. My router setup, configuring IP tables & torrents on a low-powered server.

In part 1 of this blog post, I showed you how I created a script that would, when run, rotate your snapshots on a ZFS filesystem. For this to be usable, we need to create a mechanism for having it be automatically ran. We’ll do this with a cronjob.

On Unix systems, the cron daemon is used to execute scheduled commands. Picture it much like windows task scheduler for all you windows kiddies.

I saved the backup script we wrote yesterday to /FileStore/backups.sh. The first thing I want to get running is my hourly backups. To do this, we’ll start editing our cron file

sudo crontab -e

The crontab utility is a program used to edit the tables that drive the cron daemon.

On my OSX box, information about how to set up the cron file can be found in ‘man 5 crontab’

Basically, cron examines cron entries once every minute. the fields that we’ve got to play with are (in this order)

field allowed values
—– ————–
minute 0-59
hour 0-23
day of month 1-31
month 1-12 (or names, see below)
day of week 0-7 (0 or 7 is Sun, or use names)

Having said that, it’s time to think about when we want to create our snapshots. Because I intend machines to do their backups to the server on the hour, I’ll be creating my snapshots at half past the hour. My cron file now looks like this (for hourly snapshots with 24 rotations, daily snapshots with 7 rotations, Weekly snapshots with 4 rotations.

#**Snapshots for the Filesystem**
30 * * * * /bin/bash /FileSystem/backups.sh hourly 24
30 6 * * * /bin/bash /FileSystem/backups.sh daily 7
30 6 * * sun /bin/bash /FileSystem/backups.sh weekly 4

Now that that concludes our section on setting up my rotating snapshots.

Other posts to come in the series:
1. Selecting the hardware
2. Installing the Operating System
3. Setting up File systems & Snapshots
4. Allowing access through NFS & SAMBA
5. Setting up encrypted off-site backups
6. Configuring Windows & Linux clients to dump backup info to the FileServer
7. My router setup, configuring IP tables & torrents on a low-powered server.

Setting up the FileSystems is a trivial task.  First, you can see that when we’ve created a storage pool ‘datastore’ it created a filesystem for us (also called datastore) that can act as a container for child file systems.  I’m going to go ahead and create a place to store my media and downloads now

zfs create datastore/media

for now I’ll also want a place to store my backups.  It’s worth noting that while my media filesystem will contain compressed MP3’s and the like, it’s kind of a waste of CPU power to compress it, but my backups will be a lot of PHP pages and what not, so lets go ahead and enable compression on this one

# zfs create datastore/backups
# zfs set compression=on datastore/backups

As appalled as I am of my mums backup habits, one of the requirements of this server was to provide a medium for backing up her data without her having to do anything, so lets set up backup locations for my laptop (MacShell) and a place for mum (mum) assigning both of these 20GB quota’s (ok, MacShell gets 120GB)

# zfs create datastore/backups/MacShell
# zfs create datastore/backups/mum
# zfs set quota=120G datastore/backups/MacShell
# zfs set quota=20GB datastore/backups/mum
# zfs get datastore/backups/mum

Now, the idea is that a cron job will run rsyncing over the files every hour, on the hour.  For many reasons (in case I get a virus and need to revert back, in case somebody hacks in and does bad stuff, etc, etc) I’m going to choose to create Snapshots so I can roll back to a previous version.

The convention I want is hourly.HOUR, daily.DAY, weekly.WEEK for up to 7 days and 4 weeks.  This also means that once I delete a file, I won’t recover the space that it took (once a snapshot of the file has been created) in my data pool until the end of the 4 week period.  for instance, hourly.0 will be the last hours snapshot, hourly.1 will be the 2nd last hours snapshot, etc.

the following bash script will take care of the desired snapshots.  It’s based on a concept I took from this rolling snapshots made easy post but I like my scripted way of doing rotating snapshots much better.

#!/bin/bash

#print out usage
if [ $# -ne 2 ]
then
echo “Usage: snapshot.sh [snapName] [max]”
echo “  eg. snapshot.sh hour 24″
fi

#if the max snapshot already exists, just delete it
if [ `zfs list -t snapshot | grep datastore@$1.$2 | wc -l` -eq 1 ]
then
zfs destroy -r datastore@$1.$2
fi

#
for ((i=$2-1; i >= 0; i–)); do
if [ `zfs list -t snapshot | grep datastore@$1.$i | wc -l` -eq 1 ]
then
#this snapshot exists, so we want to move it up one
zfs rename -r datastore@$1.$i @$1.$[$i+1]
fi
done

zfs snapshot -r datastore@$1.0

so with this snapshot beauty in place, lets say I had an existing file and structure in place

root@thumper:/datastore/backups/MacShell# pwd
/datastore/backups/MacShell
root@thumper:/datastore/backups/MacShell# tree
.
|– hello_world.txt
`– someDir
`– someFile.dat

1 directory, 2 files

BUT, I had a snapshot in place

# /snapshot.sh hourly 24

then I did something silly like delete my entire backup directory (Oh No!!)

# rm -R *
# ls -l
total 0

never fear! check the snapshots

root@thumper:/datastore/backups/
MacShell/.zfs/snapshot/hourly.0# tree
.
|– hello_world.txt
`– someDir
`– someFile.dat

they’ll eventually roll off my snapshot cycle and be removed in 4 weeks with my plan, but hey, pretty good at this point

See Part 2 for a post on how to set up cron jobs to automatically call this script

Other posts to come in the series:
1. Selecting the hardware
2. Installing the Operating System
3. Setting up File systems & Snapshots
4. Allowing access through NFS & SAMBA
5. Setting up encrypted off-site backups
6. Configuring Windows & Linux clients to dump backup info to the FileServer
7. My router setup, configuring IP tables & torrents on a low-powered server.

Installing OS and setup Network

For the following set of posts, I have chosen to use VirtualBox to run through how to use ZFS in building your FileSystem. The first step is downloading and installing OpenIndiana. Get the latest build from http://openindiana.org/download/ (at the time of writing, I’m using oi-dev-148-x86) and install it onto your system. In VirtualBox I chose to install onto a 64 bit Solaris setup.

Make a hard disk when you’re setting up your VM image (I called mine OS_SSD1 because the HD is eventually going to be installed onto a solid state drive.)

Now, the VM is booted, we can start having some fun (SSH into it and Away we go)

Our first step is to setup the address for the box. I must admit, I’m pretty new to Solaris but from what I’ve found, we’ll run these commands to disable Auto Configuration via DHCP then enable our static config.

svcadm disable physical:nwam
svcadm enable physical:default

My fileserver is currently not on a domain (I’ll change this later), so I’ve added the line into my /etc/hosts file

10.12.1.1 thumper.local thumper

Put your hostname that you want to use (‘thumper’ for me) in /etc/nodename

Your default gateway (routers) address should go in /etc/defaultrouter

The last thing to do is tell the host about what subnet it’s on, For me, I added (for my server subnet)

10.12.1.0 255.255.255.0

The next step is set up your nameserver for DNS (/etc/resolv.conf) mine looks like this

nameserver 10.12.1.254

Copy /etc/nsswitch.dns to /etc/nsswitch.conf – so dns is used.

Now so the adaptor comes up on boot, find out the status of your network I/O by running

dladm show-link

(for me was e1000g0). Once you have this, you need to ‘plumbe” to set up the software in the kernel to set this interface on the TCP/IP stack.

ifconfig e100g0 plumb
echo 10.12.1.1/24 > /etc/hostname.e100g0

Last but not least for your network stack,

svcadm restart milestone/network

From this point on, that concludes the base setup with networking that I’m using for the FileServer. If this happened to be a critical server for you, perhaps you’d consider setting up redundant bootable drives with ZFS. It looks pretty cool.

Setting up the storage pool

Now we can get into the fun part. The first thing I want to do with this server is be able to store my data on it, so lets set up our storage pool. In my VM I’m using to test this I’ve added the disks I want to include in my pool (1GB disks for this test)

Once we’ve booted out VM (first thing I always do is ‘sudo bash’ because I’m evil in these silly little test environments).

I want to find out the device ID’s for these hard disks I just added

root@thumper:~# iostat -En | egrep Size\|Soft
c1t0d0 Soft Errors: 0 Hard Errors: 6 Transport Errors: 0
Size: 0.00GB <0 bytes>
c1t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Size: 17.18GB <17179869184 bytes>
c1t2d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Size: 1.07GB <1073741824 bytes>
c1t3d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Size: 1.07GB <1073741824 bytes>
c1t4d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Size: 1.07GB <1073741824 bytes>
c1t5d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Size: 1.07GB <1073741824 bytes>

As we can see, c1t0d0 is my CD-ROM drive, c1t1d0 is my hard disk, so I’ll want to create a Raidz-2 (two redundant drive) storage pool with the drives c1t2d0 c1t3d0 c1t4d0 c1t5d0

root@thumper:~# zpool create datastore raidz2 c1t2d0 c1t3d0 c1t4d0 c1t5d0
root@thumper:~# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
datastore 3.94G 256K 3.94G 0% 1.00x ONLINE –
rpool 15.9G 4.17G 11.7G 26% 1.00x ONLINE –
root@thumper:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
datastore 128K 1.93G 44.8K /datastore
rpool 4.52G 11.1G 45K /rpool
rpool/ROOT 3.61G 11.1G 31K legacy
rpool/ROOT/openindiana 3.61G 11.1G 3.58G /
rpool/dump 383M 11.1G 383M –
rpool/export 1002K 11.1G 32K /export
rpool/export/home 970K 11.1G 32K /export/home
rpool/export/home/scott 938K 11.1G 938K /export/home/scott
rpool/swap 544M 11.5G 187M –

We can see now we’ve created a pool of storage across our data (that’s zfs raided) that gives us double parity so we can loose two drives and still be running and we’ve got 2GB of usable space here (I’m using 1GB hard disks, in my real production box these will be 1TB disks).

Other posts to come in the series:
1. Selecting the hardware
2. Installing the Operating System
3. Setting up File systems & Snapshots
4. Allowing access through NFS & SAMBA
5. Setting up encrypted off-site backups
6. Configuring Windows & Linux clients to dump backup info to the FileServer
7. My router setup, configuring IP tables & torrents on a low-powered server.

This will be my first blog post into a guide of setting up a fileserver using Solaris (well, OpenIndiana) and ZFS to create a fileserver that has a main purpose of being a reliable server (the focus on this build is more about reliability then throughput).

My build will probably be different then most. One of the deciding factors in me choosing my hardware would have to be physical space and energy requirements. With this server, I’ve got this relatively unused space sitting behind my 27” monitor that is reserved for the box and as the primary purpose of this box is to backup and store my data in an effort to become paperless, the power requirements are going to be trying aimed at being energy efficient.

I have found a nice tower that I’d like to use that is attractive for a few reasons. It will fit the space behind my monitor nicely but also comes with a very efficient power supply.

For the CPU, at the time of writing (March 2011), the new Sandy Bridge processors are looking like they give a big bang for your buck in terms of power usage. The only problem now is that I have to find a mini-itx motherboard that is compatible with the sandy bridge architecture (has to be mini-itx because of the form factor of the case) and has enough SATA ports (or the ability to expand to meet my requirements) and is compatible with OpenIndiana. So far, the best I’ve found is the the DH67CF. Unfortunately for a fileserver that’s going to be hosting important information, this build won’t support ECC memory, which is pretty important as you can see here (probably a better article required to link there) but hopefully not regrettably, I’ll chose to risk it.

I’ve chosen to go with RAIDZ-2 to give two redundant hard disks in my data pool (with a total of 4 hard drives.) The reason being that if they are coming from the same batch then as hard disks stand, it’s likely that two will fail at more or less a similar time. It’s also worth mentioning for the critical data on my fileserver I’m going to be implementing remote off-site backups so while a dead pool will be frustrating, the likelihood of recovering some data should not be compromised.

I’ve decided that 2TB of storage should be sufficient for my requirements. For my storage array I’ve chosen to go with 4 1TB drives [[ToDo: Choose hard disks and why]] set up in Raidz-2.

Other things that I might require to put in are I/O expansion cards for more SATA drives.

Other posts to come in the series:
1. Selecting the hardware
2. Installing the Operating System
3. Setting up Snapshots
4. Allowing access through NFS & SAMBA
5. Setting up encrypted off-site backups
6. Configuring Windows & Linux clients to dump backup info to the FileServer
7. My router setup, configuring IP tables & torrents on a low-powered server.

The tools I’ll be using for this is my old favourite wget and a new tool, “at”.

The at daemon is required to be running first, so on debian or ubuntu

/etc/init.d/atd start

Then downloading your file at an off-peak time (4am for me) is as simple as

echo “wget –c http://ubuntu.virginmedia.com/releases//maverick/ubuntu-10.10-desktop-i386.iso” | at 04:00

Simple hey

My scanner at home has a document feeder on it. The problem is that it doesn’t do duplex, only a set of sides. So far, I can scan one side of the document, flip the paper of, then scan the back pages. This will result two PDF’s with two sets of pages
Set A: 1,3,5,7
SET B: 8,6,4,2

To merge the two, I could open them both up in Adobe PDF, Preview and start clicking and dragging my time away, but that’s pointless. I’d like to introduce you to Automator in OSX.

Using a pdftk binary for OSX and the automator script above, all I need to do now is select my two pdf documents (Set A & B), right hand click, then select ‘Duplex Merge PDF’s’. After that, I’ll have a nice merged.pdf file on my desktop that’s the resulting page.

(the shell script uses pdftk to make a /tmp/2.pdf file that’s a set 2,4,6,8. Copies the Set A to /tmp/1.pdf and then pipes it through some pdf tools built into OSX to merge the sets into 1,2,3,4.. etc

Who needs to spend lots of money on a duplex scanner hey?

EDIT:
Please feel free to download my automator scripts here