Auto configuring complex cluster architectures is a task many of you might probably skip, using the excuse that it’s a one time task and it’ll never repeat itself. WRONG!
Being lazy as I could and the need to quickly deploy systems for hungry customers, I started out with a small little infrastructure that helped me along the way to auto configure clusters of two nodes or more.
My use cases were:
1. RedHat cluster
2. LVS
3. Heartbeat
4. Oracle RAC
Auto configuring an Oracle RAC DB is not an easy task at all. However, with the proper infrastructure, it can become noticeably easier.
The common denominators for all cluster configurations I had to carry were:
1. They had to run on more than one node
2. Except from entering a password once for all nodes, I didn’t want any interaction
3. They all consisted from steps that should either run on all nodes, or on just one node
4. Sometimes you could logically split the task into a few phases of configuration, making it easier to comprehend the tasks you have to achieve
Even though it is not the most tidy piece of Bash code I’m going to post here, I’m very proud of it as it saved me countless hours. I give you the skeleton of code, which is the essence of what I nicknamed Bash RPC. On top of this you should be able to easily auto configure various configurations involving more than one computer.
Using it
The sample attached is a simple script that should be able to bake /home/cake on all relevant nodes.
In order to use the script properly, edit it using your favorite editor and stroll through the configuration_vars() function. Populate the array HOST_IP with your relevant hosts.
Now you can simply run the script.
I’m aware to the slight disadvantage that you can’t have your configuration come from command line, on the other hand – when dealing with big, complex, do you really think your configuration can be defined in a single line of arguments?
Taming it
OK, this is the interesting part. Obviously no one needs to bake cakes on his nodes, it is truly pointless and was merely given as an example.
So how would you go about customizing this skeleton to your needs?
First and foremost, we must plan. Planning and designing is the key for every tech related activity you carry. Be it a SysAdmin task or a pure development task. While configuring your cluster for the first time, keep notes of the steps (and commands) you have to go through. Later on, try to logically separate the steps into phases. When you have them all, we can start hacking Bash RPC.
Start drawing your phases and steps, using functions in the form of PHASEx_STEPx.
Fill up your functions with your ideas and start testing! and that’s it!
How does it work?
Simplicity is the key for everything.
Bash RPC can be ran in 2 ways – either running all phases (full run) or running just one step.
If you give Bash RPC just one argument, it assumes it is a function you have to run. If no arguments are given it will run the whole script.
Have a look at run_function_on_node(). This function receives a node and functions it should run on. It will copy the script to the destination node and initiate it with the arguments it received.
And this is more or less the essence of Bash RPC. REALLY!
Oh, there’s one small thing. Sometimes you have to run things on just one host, in that case you can add a suffix of ___SINGLE_HOST for your steps. This will make sure the step will run on just one host (the first one you defined).
I’m more than aware that this skeleton of Bash RPC can be polished some more and indeed I have a list of TODOs for this skeleton. But all in all – I really think this one is a big time saver.
Real world use cases
I consider this script a success mainly because of 2 real world use cases.
The first one is the act of configuring from A to Z Oracle RAC. Those of you who had to go through this nightmare can testify that configuring Oracle RAC takes 2-3 days (modest estimation) of both a DBA and SysAdmin working closely together. How about an unattended script running in the background, notifying you 45 minutes later you have an Oracle RAC ready for service?
The second use case is my good friend and former colleague, Oren Held. Oren could easily take this skeleton and use it for auto configuring a totally different cluster using LVS over Heartbeat. He was even satisfied while using it. Oren never consulted me while performing this task – and this is the great achievement.
I hope you could use this code snippet and customize it for your own needs, continuing with the YOU CAN CREATE A SCRIPT FOR EVERYTHING attitude!!
Have a look at cluster-config-cake.sh to get an idea about how it’s being done.
Bashing Linux 