Giving Your SoftLayer Servers A Personality With Provisioning Scripts

Forward

Personalities no longer only apply to people (and some would argue animals); this terminology has found its way into the Cloud / virtualization space as well. In the Cloud / virtualization space the term personality refers to the act of tweaking or configuring a vanilla server instance (typically via a bootstrapping or automated process) for a particular purpose; the resulting "tweaked" server is said to have a personality. For example, let's say you take a vanilla Ubuntu Virtual Machine (VM), install the eclipse IDE, pydev and a handful of python modules resulting in VM specifically tailored for python development. That VM can be said to have a "python dev" personality.

SoftLayer makes server personalities easy and convenient by means of "provisioning scripts". Not only are provisioning scripts a snap to use, but they also provide a consistent way to bootstrap any SoftLayer server type and image making them a very effective tool in the SoftLayer infrastructure.


In this article we'll cover the following topics:

• An overview of what provisioning scripts are and some common usages.
• Details on how SoftLayer carries out the provisioning script process.
• Subtleties of provisioning scripts on Linux and Windows based instances.
• How provisioning scripts can be hosted, inside SoftLayer and on the public internet.
• Steps on how to create account-wide and one-time provisioning scripts in SoftLayer.
• An example of creating and using an account-wide provisioning script.

SoftLayer provisioning scripts

Giving your servers' a personality in SoftLayer is a straight forward task by leveraging what's called "provisioning scripts". Slightly contrary to their name, provisioning scripts don't need to be a script in the literal sense (e.g. bash shell script or Windows DOS script). Rather, a provisioning script is simply a file you want downloaded and optionally executed on your instance when it's provisioned for you by the SoftLayer infrastructure.

Examples of provisioning scripts include:

• Shell scripts -- a plain old bash (or other shell depending on your instance Operating System) script.
• Python scripts -- a python source script; it must include the shebang path atop the script to ensure the python interpreter is used when running the script file.
• A compiled binary (C code or other) -- an executable in binary format; must be complied for the architecture of your instance's Operating System.
• A non-programmatic text file.
• Any other single file including zips, tarballs, etc.. No size limit is imposed beyond the capacity of the hard disk on the server instance.

SoftLayer supports provisioning scripts for both virtual Cloud Computing Instances (Virtual Servers aka CCIs) as well as for Bare Metal Instances (BMIs). Moreover the provisioning script process works for both Linux and Windows based SoftLayer image templates making the provisioning script process a consistent means to give any SoftLayer server a personality.

Common use cases for provisioning scripts include:

• Setting / customizing user or system preferences.
• Installing software packages.
• Sending a one-time "server ready" message to a remote service or system.
• Downloading a file to the server which will be used at some point in the lifetime of the server instance.

Although SoftLayer doesn't impose restrictions on what you can do in provisioning scripts, my recommendation is to use them for smaller setup / configuration tasks or to transfer an initial file to the server instance. If you find yourself using provisioning scripts for extensive processing it may be worth your while to consider creating a SoftLayer image template. With image templates you can customize a vanilla server instance to your liking and then capture the instance to image making the image available for subsequent provisioning operations. One benefit of the image approach is that the deployed server is already setup to your liking and hence the provisioning process is shorter (no need to wait for your provisioning script to run).

Provisioning script process

Irrespective of your server type or boot image the conceptual process for "running" your provisioning script is the same. The SoftLayer infrastructure performs the following tasks when running your provisioning script at the end of the server provision operation:

• Download the provisioning script specified for your server instance to the host Operating System (OS) as the last step of your instance creation.
• If the provisioning script file fails to download, send an automated email to the SoftLayer account owner indicating such. Note - only the account owner receives this email, not sub-users in the account.
• Save the contents of the file on your server in a predefined location (see subsections below).
• If the script is downloaded via HTTP, the provisioning script process is done; it's up to you to do something with the downloaded file.
• If the script is downloaded using HTTPS (secure HTTP), then change the script's mode to executable (Linux only) and run it without arguments in the default shell of your server's Operating System.
• If the provisioning script fails and exits with a non-zero return code, the SoftLayer infrastructure will not notify you of such. Thus if you need to track return code of you provisioning script you will need to implement the means to do so such as writing it to a file.

An important thing to keep in mind with executable provisioning scripts is that they will be executed for you in the default shell of your server instance's Operating System and therefore must take into account any prerequisite packages which are necessary for execution. For example if you want to execute a JavaScript based provisioning script with node.js, your script would need to install / configure node.js on the system prior to running your actual JavaScript code (assuming the SoftLayer image you are booting your server instance from does not already have node.js).

If you are using a HTTPS based provisioning script to execute during the provisioning process, your script will be run after all devices have come up on the server instance. This means your script can access the network interfaces, work with block devices, access the file system, etc.. Also note that your provisioning script does not get passed any arguments by the provisioning script process and therefore if you need conditional execution your script will have to access a remote service over the network (or some other crafty means) to determine such conditions. Note that SoftLayer provides the ability to pass down arbitrary user metadata for your server instances which could be used for conditional processing. More on that topic in the next post.

There are a few subtleties between the provisioning script realization on Linux instances vs. Windows; let's quickly cover those.

Linux subtleties

On SoftLayer server instances created with Linux based images, the script file downloaded is saved as /root/post_install.xxx where xxxx indicates four random characters. This post install file will contain the contents of your script file as downloaded during the provisioning process. If your script was downloaded via HTTPS this file will additionally be executable and will have been run in the default shell (e.g. bash) of your Linux instance Operating System.

The console listing below shows the /root/ directory of a newly provisioning Ubuntu instance in SoftLayer which specified a provisioning script. As you can see the provisioning script is downloaded and named post_install.t7X6. Since this file is not executable we know it was defined using a HTTP based URL.

root@prov-unsecure-private:~# ll
total 28
drwx------  3 root root 4096 Mar 31 19:27 ./
drwxr-xr-x 23 root root 4096 Apr  1  2014 ../
-rw-------  1 root root    8 Mar 31 19:27 .bash_history
-rw-r--r--  1 root root 3106 Apr 19  2012 .bashrc
drwx------  2 root root 4096 Mar 31 19:27 .cache/
-rw-------  1 root root  446 Mar 31 10:03 post_install.t7X6
-rw-r--r--  1 root root  140 Apr 19  2012 .profile

Windows subtleties

On SoftLayer server instances created with Windows based images, the script file downloaded is saved into the C:\ root directory of the server instance. The result is a file named C:\<script_name> where script_name is the actual name of your script file. If the script was executed (e.g. had a HTTPS based URL) there will also be a file named C:\postInstallScript.log on your server file system. This file contains the output of your scripts execution (e.g. the commands it ran if it was a DOS bat file script). The diagram below depicts the result of an HTTP based provisioning script named secho.bat on a Windows based server instance in SoftLayer.

Note that if your script depends on environment variables during execution, the HOMEDRIVE and HOMEPATH variables are not defined in the environment when your provisioning script runs (at least not on the Windows images I've worked with).

Hosting provisioning scripts

You have numerous options for hosting your provisioning scripts. If the content of your file is not sensitive you can host the file on any publicly accessible web server including a public github repository (using the 'raw' URL of the file), or even on another SoftLayer server instance with a web server listening on the public interface. Moreover the provisioning process can access hosts anywhere on the SoftLayer private network including URLs referring to hosts in different VLANs and in different SoftLayer Data Centers. This cross network segment support enables use cases such as setting up a SoftLayer server instance to host your provisioning scripts on any SoftLayer private network (VLAN) without worry of provisioning co-located instances. If the provisioning script file is sensitive you can elect to serve it using a web server which supports URI based authentication tokens to protect files. For example Apache's mod-auth-token.

Using provisioning scripts for server instances

SoftLayer supports two basic categories provisioning scripts; account-wide and anonymous one-time provisioning scripts. The main difference between the two is that account-wide scripts are defined / saved in your SoftLayer account and available to all users within that account, whereas one-time scripts are specified on-the-fly and not saved in your account. Regardless of which category of provisioning script you use, the script to use will be specified per server instance on the "Order Summary" page of the server order creation process in the SoftLayer web dashboard. Let's cover both categories of provisioning scripts in greater detail below.

Account-wide provisioning scripts

Provisioning scripts can be defined and saved in your SoftLayer account allowing you to easily select and use them during the instance order process. These account wide provisioning scripts are also accessible to any other users in your SoftLayer account permitting reuse and sharing at an account level. Details on creating account level provisioning scripts can be found on SoftLayer's KnowledgeLayer, but let's walk though a few quick topics with some screen shots to exemplify the process.

Creating account-wide provisioning scripts

Firstly, your SoftLayer user must have Manage Provision Scripts permissions within the SoftLayer account in order to add / modify account-wide scripts. If you do not have these permissions, contact your SoftLayer account owner and request such access which can be set via the SoftLayer web dashboard under Administrative > User Administration > username in the User Portal Permissions > Software  section as shown below.

To manage account-wide provision scripts, navigate to Software > Provisiong Scripts in the SoftLayer web portal which will bring-up the provisioning script management portal view. From here you can add, edit and delete account wide provisioning scripts. To add a new provisioning script click the Add Provisioning Script button as shown below.

You will now be presented with the "Add Provisioning Script" panel which allows you to provide the following details for your script:

• Script name -- A name to associate with your script. This can be any name you choose, but will most likely be descriptive with respect to your provisioning script's functionality.
• URL -- The HTTP or HTTPS URL of your script. Remember if you want the provisioning script executed for you, the script must be accessible and specified using HTTPS. HTTP based scripts will only be downloaded (but not executed) on your instance.

For example:



Once you have defined the script name and URL of the script, click 'Create' to create the new provisioning script definition in your account. This script is now associated with your SoftLayer account and can be selected from a drop-down during the server order process. You can also modify and delete existing provisioning scripts in your account by selecting the 'Edit' and 'Delete' links next to the provisioning script listed for your account. The edit and delete operations are self evident so we'll skip their details here.

To specify which provisioning script to use for your SoftLayer server, you will need to navigate to the "Provisioning Scripts" section of the instance order confirmation when creating a new server in the SoftLayer web dashboard. Here you will be presented with a drop-down listing all the provisioning scripts associated with your account. For example, see the screen shot below. Simply select the script to use and you are good to go.



If you are creating a server order for more than one server instance, you will be provided with the option to select a provisioning script per instance as shown below. This allows you to specify a different provisioning script for each server instance providing maximum flexibility.

Anonymous (one-time) provisioning scripts

Anonymous or one-time provisioning scripts are almost identical in usage to account-wide scripts. The only difference is that instead of defining and saving the provisioning script in your account, you only specify the URL to your script during the instance order process. Thus, this approach is useful if you don't have proper permissions to create account-wide provisioning scripts, are testing your script, don't want to share the script with other users in the account or if you only intend to use the script one or a few times.

The screen shot below depicts the "Provisioning Scripts" portion of the instance order process in the SoftLayer web dashboard. As you can see the "Existing Script" dialog is set to (new URL) and the script's HTTP or HTTPS accessible URL is specified in the "URL" text field.

The same rules which apply to the URL of an account-wide script apply to one-time scripts; files specified with HTTP will only be downloaded whereas files specified with HTTPS will be downloaded and executed in the default shell of you server instance OS.

Example: Account-wide provisioning script hosted on github

For this example I have created a basic provisioning script on github which I use to setup my Ubuntu based instances in SoftLayer. My provisioning script is a simple bash shell script which first ensures vim is installed using apt-get, and then sets up my ~/.vimrc preferences. This approach is very handy as it saves me the time of setting up vim manually on each of my new instances. Note that my particular script is tied to Ubuntu as it uses the apt-get command and moreover it assumes bash is the default shell.

In order to leverage the github hosted script as a SoftLayer provisioning script, you need to use the 'Raw' URL of the script as provided by github. To get the raw URL of the file, view the file on github and click the 'Raw' button (see pic below).

Copy the URL from your browser window while viewing the raw version of the file; this is the HTTPS URL to the file contents itself. For this example we end up with the following URL:

https://raw.githubusercontent.com/bodenr/cci/master/apt-init-vimrc.sh

Now we need to create the account-wide provisioning script. To create the new account-wide provisioning script browse to the account provisioning script management panel (Software > Provisiong Scripts) in the SoftLayer web dashboard and click the Add Provisioning Script button. Give the script a name and specify the github 'raw' URL to the script as shown below:

Click "Create" once you have specified the script details to save the new script. This provisioning script is now associated with the account and named ubuntu-vim-init. Other users within my SoftLayer account can now use this script easily from the Instance Order process.

Now we are ready to use the script for a new SoftLayer server instance. To specify the provision script during the instance ordering process, you must select the provisioning script named ubuntu-vim-init from the drop-down on the "Order Summary" page of the server order. All account-wide provision scripts will be shown in the drop-down and you simply need to select the one to use as shown below. For this example select the ubuntu-vim-init script.

If you are provisioning more than one instance in 'batch' you will be presented with a separate drop-down for each instance in the order. This allows you to easily specify a different provisioning script for each instance.

We can now submit the instance order to begin the SoftLayer provisioning process.

Once the instance is provisioned you can verify the provisioning script was executed by looking in your home directory (e.g. /root). The provisioning script will be named post_install.xxxx where xxxx is a random set of characters. As the provisioning script was executed during the instance provisioning process (it used an HTTPS based URL), it will be executable as shown in the sample listing below.

root@htsvr:~# ll
total 40
drwx------  3 root root 4096 Mar 31 10:23 ./
drwxr-xr-x 23 root root 4096 Mar 31 09:42 ../
-rw-------  1 root root 1630 Mar 31 22:18 .bash_history
-rw-r--r--  1 root root 3126 Mar 31 09:43 .bashrc
drwx------  2 root root 4096 Mar 31 09:43 .cache/
-rwx--x--x  1 root root  445 Mar 31 09:43 post_install.IR7A*
-rw-r--r--  1 root root  140 Apr 19  2012 .profile
-rw-------  1 root root 1024 Mar 31 10:21 .rnd
-rw-------  1 root root 1162 Mar 31 10:23 .viminfo
-rw-r--r--  1 root root  323 Mar 31 09:43 .vimrc

We can now verify the provisioning script did it's job. In the case of this example it did -- vim is installed and we can see the .vimrc file created in the console listing above. That's it -- the service instance now has a vim'ish personality and our work is complete (or in reality ready to begin).

Related resources

• http://www.softlayer.com/
• http://knowledgelayer.softlayer.com/topic/provisioning-scripts
• http://knowledgelayer.softlayer.com/topic/image-templates
• http://www.tldp.org/LDP/abs/html/
• http://www.dostips.com/
• http://knowledgelayer.softlayer.com/learning/introduction-virtual-servers
• http://knowledgelayer.softlayer.com/procedure/order-new-device-flex-image-template
• http://knowledgelayer.softlayer.com/gettingstarted/how-to/set-up-a-dedicated-server