The master of puppets   10 comments

After several conversations with my good friend and now also boss I decided to open a small little Blog which will describe my Linux/SysAdmin/C++/Bash adventures.

I’m Dan, you can learn a lot about me from my website at . My expertise is Unix/Linux System administration and C++ development. Linux is my home and Bash is my mother’s tongue. In this Blog I hope to share interesting topics, stories and decisions in the mentioned fields.

I’ll start with something fresh and new for me – Puppet (

I believe System administration is an art, just like programming. SysAdmins divide into 2 types, from what I’ve seen:

  1. The fireman – most likely the mediocre SysAdmin type you are very familiar with. This type of SysAdmin knows how to troubleshoot and apply the specific fix where’s it’s needed. They get things done, but not more than that. Gradually their workload grows (especially if they do a bad job) and consume all of their time. This is the bad type of SysAdmin. Words like planning are usually not among their jargon (although they might argue with you).
  2. The developer – this type of SysAdmin plans before doing anything. I hope I can belong to this group. He thinks as a developer – whenever a problem arises in a system he manages – he treats it as a bug rather then a problem. This type of SysAdmin will always have his configuration stored in a source control repository. He will always solve bugs he encounter by fixing the script or program that caused the so called problem.

If you’re a fireman – then  this post is not for you. However, if you’re the second type, then puppet is exactly for you.

As a SysAdmin in a small startup company, I got to a position where I have to manage 2 types of machines:

  1. Production machines
  2. In house machines for internal company usage

Luckily managing dozens of production machines is easy for us (for me :)) as the configuration is most likely to be the same on all of them. On the other hand, ironically, managing the in house machines became a more time consuming task, as each machine has many different and diverse jobs. This is where puppet kicks in.

Up until today, we had a few main servers doing most of the simple tasks, among them:

  1. LDAP
  2. Samba/NFS file sharing
  3. SVN
  4. Apache
  5. In house DNS and DNS caching
  6. Buildbot master
  7. Nagios
  8. And many more other things (you got the point)

Obviously I have all of my configuration stored in our SVN and backed up. Given that one of these servers would die, I could generate a new one within hours. But yet, most of the configuration there was done manually. Who is the crazy SysAdmin that would write a script to meta-generate named.conf? Who would auto create SVN repositories (especially if you have just one repository to manage)?

Yes, this is where puppet goes in. Puppet forces you to work correctly. By working with puppet, you realize that the way you’re going to work is only by writing puppet recipes and automating everything that makes your servers what they are. Puppet is also revolutionary in many other ways, but I’ll focus on the way that it forces the SysAdmin to work in a clean manner.

One thing I have to warn from, puppet has a slightly steep learning curve, but if you’re the serious Linux SysAdmin type – you wouldn’t find that one steep. You’ll struggle with it. You must.

So I said that no one (most likely) would write a script to generate its DNS configuration, but I saved mine in SVN and wrote a recipe for DNS servers to use:

$dns_username = 'named'
$dns_group = 'named'
$dns_configuration_file = '/etc/named.conf'
class dns-server {
	# puppet takes care to install the latest package bind and it's cross platform!
	package { bind:
		ensure => latest
	# configuring named. subscribing it on /etc/named.conf, which means that every time this
	# file would change, named will refresh
	service { named:
		ensure    => running,
		enable    => true,
		subscribe => File[$dns_configuration_file],
		require   => Package["bind"]
	# this is the /etc/named.conf file. in subclasses we'll specify the source of the file
	file { $dns_configuration_file:
		owner   => $dns_username,
		group   => $dns_group,
		replace => true,
		mode    => 640

Alright, this puppet code snippet only configures /etc/named.conf for usage with bind – but we never told puppet where to take the configuration from. I’ll share with you my DNS master server configuration:

class dns-master inherits dns-server {
	# if it's a DNS master - lets take the named-master.conf file (obviously stored in SVN as well)
	File[$dns_configuration_file] {
		source => "$puppet_fileserver_base/etc/named/named-master.conf",

	# copy all the zone files
	file { "/var/named":
		source       => "$puppet_fileserver_base/etc/named/zones",
		ignore       => ".svn*",
		checksum     => "md5",
		sourceselect => all,
		recurse      => true,
		purge        => false,
		owner        => $dns_username,
		group        => $dns_group,
		replace      => true,
		mode         => 640

	# subscribe all the zone files in named
	Service[named] {
		subscribe +> File["/var/named"]

Great, so now every time I’ll change a zone in my puppet master, named will automatically restart. I believe this is great. Needless to say I do not use dynamic DNS updates, so I care not about zone files being updated.

Yes, so I did it once for DNS. If I ever needed now to change the configuration, I’d do it via puppet and let the configuration propagate. Maybe a small little phase of checking the new configuration, but right afterwards would come the svn commit – and it’s inside. Now I know I’m tidy and clean forever.

Needless to say that if I’ll ever need to configure another DNS server, I’ll use this recipe, or extend it as needed – puppet lets you inherit from base classes and add properties and behavior as needed (I have also a class for a dns-slave as you could have imagined).

OK, I should sum it up. To be honest – I’m glad I’m writing this part of the post after several discussions with a few of my wise colleagues.

Why should I use puppet?!

The answer to this one if not easy – and I’m not the one who should answer it. It is you who should answer it.

Adopting puppet reflects one main thing about you as a SysAdmin:

You are a lazy, strict and tidy SysAdmin, an expert.

Posted July 27, 2009 by malkodan in Linux, System Administration

Tagged with , , , , , , , ,

10 responses to “The master of puppets

Subscribe to comments with RSS.

  1. You wouldn’t need it on solaris 🙂
    Nice blog

  2. Define “steep learning curve” — is it akin to installing linux-ha? Is it closer to configuring Tomcat, on the complexity scale? Maybe it’s like being kicked in the scrotum?…

    Considering who the expression comes from, this may mean significantly more risk than a lot of organizations are willing to take.

    • I’d say it’s quite similar to learning a programming language, but an easy one. Gabel, in your case I don’t think it’s something you’ll find ‘steep’. Seriously. But why don’t you try it for yourself?

      I didn’t know I have a reputation of someone that’s taking risks all the time 😉

  3. It’s not about taking risk, it’s the simple fact that in Israel it’s really hard to find people are actually know their shit around Linux. With Windows servers anyone can figure out how to install just about anything. On Linux, in order to effectively tackle something new you need either a lot of experience or someone with a lot of experience by your side.

    • Don’t blame me for the incompetency of SysAdmins 🙂

      I have always believed that in order to be a good SysAdmin, you first need to be a skilled programmer. Anyway, this was not a Windows vs. Linux post, it was more about the way system administration should be done, in my opinion.

  4. I didn’t intend my response to be the beginning of a flame war (luckily there aren’t any other commenters… yet :-)). Let me rephrase the question: assuming a /beginner/ Linux syadmin who isn’t stupid but fairly inexperienced, how relevant is this solution?

    • I think that a beginner SysAdmin wouldn’t find that one too useful, because a beginner probably wouldn’t need to manage too many servers with heterogeneous, different and dynamic configurations.

      If you are the so called beginner SysAdmin, I’d recommend you to use it, mainly because I know you are a skilled developer who will catch this idea very quickly.

  5. I might look into it, then, thanks…

  6. Great article! I came across it while researching a Puppet tutorial series I’m working on:

    Let me know what you think and if you have any suggestions for future topics to cover. I like your DNS solution and think this would make a good subject for a more advanced article.

  7. Thanks John!

    You can feel free to use this example, but please, if possible, link to me or credit me 🙂
    To be honest recently I haven’t worked much with puppet mainly because it just works!

    Your tutorial seem to cover only the basics now, however, I must admit that I did learn a few things from there.
    To be honest – I’d like to see a puppet configuration that configures a cluster of active/active, or active/passice LDAP servers.
    Perhaps some Facter examples could be nice, I can’t think of anything in particular…
    I can post another puppet example of a Buildbot server configuration (master and slave), if it can be of any interest. Other than that I think that puppet should be well integrated with scripts that can do most of the more complex stuff.

    I’m looking forward to see some new stuff in your tutorial.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: