After several conversations with my good friend and now also boss I decided to open a small little Blog which will describe my Linux/SysAdmin/C++/Bash adventures.

I’m Dan, you can learn a lot about me from my website at http://nevela.com . My expertise is Unix/Linux System administration and C++ development. Linux is my home and Bash is my mother’s tongue. In this Blog I hope to share interesting topics, stories and decisions in the mentioned fields.

I’ll start with something fresh and new for me – Puppet (http://reductivelabs.com/trac/puppet).

I believe System administration is an art, just like programming. SysAdmins divide into 2 types, from what I’ve seen:

1. The fireman – most likely the mediocre SysAdmin type you are very familiar with. This type of SysAdmin knows how to troubleshoot and apply the specific fix where’s it’s needed. They get things done, but not more than that. Gradually their workload grows (especially if they do a bad job) and consume all of their time. This is the bad type of SysAdmin. Words like planning are usually not among their jargon (although they might argue with you).
2. The developer – this type of SysAdmin plans before doing anything. I hope I can belong to this group. He thinks as a developer – whenever a problem arises in a system he manages – he treats it as a bug rather then a problem. This type of SysAdmin will always have his configuration stored in a source control repository. He will always solve bugs he encounter by fixing the script or program that caused the so called problem.

If you’re a fireman – then  this post is not for you. However, if you’re the second type, then puppet is exactly for you.

As a SysAdmin in a small startup company, I got to a position where I have to manage 2 types of machines:

1. Production machines
2. In house machines for internal company usage

Luckily managing dozens of production machines is easy for us (for me :)) as the configuration is most likely to be the same on all of them. On the other hand, ironically, managing the in house machines became a more time consuming task, as each machine has many different and diverse jobs. This is where puppet kicks in.

Up until today, we had a few main servers doing most of the simple tasks, among them:

1. LDAP
2. Samba/NFS file sharing
3. SVN
4. Apache
5. In house DNS and DNS caching
6. Buildbot master
7. Nagios
8. And many more other things (you got the point)

Obviously I have all of my configuration stored in our SVN and backed up. Given that one of these servers would die, I could generate a new one within hours. But yet, most of the configuration there was done manually. Who is the crazy SysAdmin that would write a script to meta-generate named.conf? Who would auto create SVN repositories (especially if you have just one repository to manage)?

Yes, this is where puppet goes in. Puppet forces you to work correctly. By working with puppet, you realize that the way you’re going to work is only by writing puppet recipes and automating everything that makes your servers what they are. Puppet is also revolutionary in many other ways, but I’ll focus on the way that it forces the SysAdmin to work in a clean manner.

One thing I have to warn from, puppet has a slightly steep learning curve, but if you’re the serious Linux SysAdmin type – you wouldn’t find that one steep. You’ll struggle with it. You must.

So I said that no one (most likely) would write a script to generate its DNS configuration, but I saved mine in SVN and wrote a recipe for DNS servers to use:

$dns_username = 'named'
$dns_group = 'named'
$dns_configuration_file = '/etc/named.conf'
class dns-server {
	# puppet takes care to install the latest package bind and it's cross platform!
	package { bind:
		ensure => latest
	}
	# configuring named. subscribing it on /etc/named.conf, which means that every time this
	# file would change, named will refresh
	service { named:
		ensure    => running,
		enable    => true,
		subscribe => File[$dns_configuration_file],
		require   => Package["bind"]
	}
	# this is the /etc/named.conf file. in subclasses we'll specify the source of the file
	file { $dns_configuration_file:
		owner   => $dns_username,
		group   => $dns_group,
		replace => true,
		mode    => 640
	}
}

Alright, this puppet code snippet only configures /etc/named.conf for usage with bind – but we never told puppet where to take the configuration from. I’ll share with you my DNS master server configuration:

class dns-master inherits dns-server {
	# if it's a DNS master - lets take the named-master.conf file (obviously stored in SVN as well)
	File[$dns_configuration_file] {
		source => "$puppet_fileserver_base/etc/named/named-master.conf",
	}

	# copy all the zone files
	file { "/var/named":
		source       => "$puppet_fileserver_base/etc/named/zones",
		ignore       => ".svn*",
		checksum     => "md5",
		sourceselect => all,
		recurse      => true,
		purge        => false,
		owner        => $dns_username,
		group        => $dns_group,
		replace      => true,
		mode         => 640
	}

	# subscribe all the zone files in named
	Service[named] {
		subscribe +> File["/var/named"]
	}
}

Great, so now every time I’ll change a zone in my puppet master, named will automatically restart. I believe this is great. Needless to say I do not use dynamic DNS updates, so I care not about zone files being updated.

Yes, so I did it once for DNS. If I ever needed now to change the configuration, I’d do it via puppet and let the configuration propagate. Maybe a small little phase of checking the new configuration, but right afterwards would come the svn commit – and it’s inside. Now I know I’m tidy and clean forever.

Needless to say that if I’ll ever need to configure another DNS server, I’ll use this recipe, or extend it as needed – puppet lets you inherit from base classes and add properties and behavior as needed (I have also a class for a dns-slave as you could have imagined).

OK, I should sum it up. To be honest – I’m glad I’m writing this part of the post after several discussions with a few of my wise colleagues.

Why should I use puppet?!

The answer to this one if not easy – and I’m not the one who should answer it. It is you who should answer it.

Adopting puppet reflects one main thing about you as a SysAdmin:

You are a lazy, strict and tidy SysAdmin, an expert.