So far we’ve deployed Micro BOSH and used it to deploy and scale a release. That’s all well and good, until a release doesn’t exist for something you’re trying to deploy. In this next example, we’ll take a look at creating a release from scratch to deploy redis, which is becoming the sort of standard Hello World of BOSH releases.

Before we get started, a huge thanks and shout-out to Dr Nic Williams for both creating the bosh-gen gem, as well as his tutorial for using it to create a release. His work makes this process an order of magnitude easier, and I’ve very happily helped add vSphere support to his gem. You’ll notice much of this tutorial is the same as his, but I figured I’d rather be redundant and complete rather than leave a step out.

So let’s go ahead and grab the bosh-gen gem…

$ gem install bosh-gen

NOTE: Currently, my pull request is under review from Nic. You can get my fork here, in the vsphere_manifest branch. If you’d like to manually build and install this gem, simply do the following:

$ git clone git://github.com/BrianMMcClain/bosh-gen.git -b vsphere_manifest
$ cd bosh-gen
$ bundle install
$ rake build
$ gem install pkg/bosh-gen-0.6.1.gem

Using this gem, most of the heavy-lifting is done for us, and we focus on package-specific solutions. So let’s dive right in and begin by creating a new release:

$ mkdir ~/releases/redis-on-demand
$ cd ~/releases/redis-on-demand
$ bosh-gen new redis-on-demand
$ cd redis-on-demand

You’ll notice the following things just occurred…

1. A bunch of files were created, this is the basic file-structure of a BOSH release
2. A Git repoistory was initialized
3. The files were added and committed to the git repository

Creating a Package

The first part of a release we’ll look at is a package. A package is essentially a collection source code, along with the scripts to compile and install the software. In this release, our package will be redis, along with the script to compile the redis-server binary. First, let’s create the skeleton for a new package…

$ bosh-gen package redis
      create  packages/redis/packaging
      create  packages/redis/pre_packaging
      create  packages/redis/spec

Let’s take a quick look at what these files are:

• packaging: Script to compile the package. This script is called on the VM that is spun up on a worker VM, that is, a VM specifically created to compile a package
• pre_packaging: Optional. This is ran locally when you run “bosh create release”
• spec: Describes what packages this package depends on, as well as the files to include in this package (more on this later)

Next step is adding the redis source tarball to the package, so that we can use it to compile redis-server:

$ bosh-gen source redis http://redis.googlecode.com/files/redis-2.4.13.tar.gz

Again, you’ll see some files added (src/redis/redis-2.4.13.tar.gz) and a couple files modified (packages/redis/spec and packages/redis/packaging). Looking at the spec file for the redis job, we can see where these dependencies are listed:

--- 
name: redis
dependencies: []

files: 
- redis/redis-2.4.13.tar.gz

As expected, we can see that when this package is compiled, we’ll bring along the redis source tarball. Next step is dropping in the packaging script that’s used to compile the package. Below will be the contents of our packages/redis/packaging file:

tar zxf redis/redis-2.4.13.tar.gz

if [[ $? != 0 ]] ; then
  echo "Failed extracting redis"
  exit 1
fi

(
  cd redis-2.4.13
  make
  make PREFIX=$BOSH_INSTALL_TARGET install
)

Pretty simple, we extract the tarball, make sure it extracted correctly, compile it and install it. You’ll notice the BOSH_INSTALL_TARGET environment variable that is referenced. This is a filesystem path that BOSH expects the binaries and other files required to run the binaries to be placed after it’s been compiled.

Creating a Job

Dr Nic explains it well, we can now install redis, but we don’t know how to run redis. For this, we need to create a job. A job is to running a process as a package is to compiling source code. From the OSS-Docs from VMware, “A job contains the configuration files and startup scripts to run the binaries from a package.” So, let’s go ahead and create one to run redis-server:

$ bosh-gen job redis -d redis
      create  jobs/redis
      create  jobs/redis/monit
      create  jobs/redis/templates/redis_ctl
       chmod  jobs/redis/templates/redis_ctl
      create  jobs/redis/spec

To clarify, we’re saying we want to create a job named “redis” (bosh-gen job redis) that depends on the package “redis” (-d redis). The -d flag takes any number of package names, so if a job requires multiple packages, we would list them out here. We will, once again, see that we’ve created a handfull of files. This is the skeleton for a job, but bosh-gen fills out these files the best it can. Of course, each of these files serve a purpose, and they are:

• jobs/redis/monit: BOSH uses monit to start, stop, monitor and manage a process. The file that bosh-gen generates should suffice for most basic jobs
• jobs/redis/templates/redis_ctl: This is the script that is called in the monit file to actually start the process. This contains the logic to start, stop, etc a process
• jobs/redis/spec: Describes the job

While we’re at it, we’ll make one more file, the template for the redis config file. Again, there’s a handy bosh-gen command to generate this:

$ bosh-gen template redis config/redis.conf
      create  jobs/redis/templates/redis.conf.erb
       force  jobs/redis/spec

We’ve just created the template file (at jobs/redis/templates/redis.conf.erb), and added the file to the spec file, so BOSH knows where to put it when it creates the VM. Let’s toss a basic config into the jobs/redis/templates/redis.conf.erb file:

daemonize no
pidfile /var/vcap/sys/run/redis/redis.pid
timeout 300
loglevel notice
logfile stdout
databases 16
dir /var/vcap/store/redis
maxclients 0
maxmemory-policy noeviction
save 60 1000
appendonly yes

You’ll notice that pidfile matches the PID file we pointed out in out monit file for our job. Finally, we’re going to have to make one quick change to the redis_ctl to tell it how to run redis, let’s take a look at the file:

RUN_DIR=/var/vcap/sys/run/redis
LOG_DIR=/var/vcap/sys/log/redis
STORE=/var/vcap/store/redis
PIDFILE=$RUN_DIR/redis.pid

export PATH=/var/vcap/packages/ruby/bin:$PATH

COMMAND=${COMMAND:-/var/vcap/packages/redis/bin/EXECUTABLE_SERVER}
HOME=${HOME:-/home/vcap}

case $1 in

  start)
    mkdir -p $RUN_DIR
    mkdir -p $LOG_DIR
    mkdir -p $STORE

    echo $$ > $PIDFILE

    exec $COMMAND >>$LOG_DIR/redis.stdout.log 2>>$LOG_DIR/redis.stderr.log
    ;;

  stop)
    PID=$(head -1 $PIDFILE)
    kill $PID
    while [ -e /proc/$PID ]; do sleep 0.1; done
    rm -f $PIDFILE
    ;;

  logs)
    cat $LOG_DIR/*
    ;;

  tail)
    tail -f $LOG_DIR/*
    ;;

  clearlogs)
    rm $LOG_DIR/*
    ;;

  *)
  echo "Usage: redis_ctl {start|stop|logs|tail|clearlogs}" ;;
esac
exit 0

We need to change line 8 to point it to redis-server, and since redis-server accepts a path to a config file as it’s first argument, we’ll pass that in as well, so change the line to the following:

COMMAND=${COMMAND:-/var/vcap/packages/redis/bin/redis-server /var/vcap/jobs/redis/config/redis.conf}

One thing to point out, notice that the executable is in /var/vcap/packages/redis/bin/redis-server where the config file is at /var/vcap/jobs/redis/config/redis.conf. This matches the route we gave when creating the redis.conf file, and in our spec file, we see the line “redis.conf.erb: config/redis.conf”, which tells us where that file will be. Also, notice that the binary is in /var/vcap/packages, and the config file is in /var/vcap/jobs.

Creating the Release

We have our package, which tells BOSH how to compile redis. We also have our job, which tells BOSH how to run redis. Now we’re finally ready to create our release. BOSH seems (and I’m still working out if this is always the case) to want to make sure all of the files and changes are committed into a git repository. bosh-gen luckily does this for us, but we have to add the files we created, and commit the changes we made. Easy enough:

$ git add .
$ git commit -m "First version of the release" -a

And now it’s finally time to turn it over to BOSH, and create the release:

$ bosh create release

BOSH will list the package and job we jut created, along with the path to the release manifest that BOSH creates. Next, upload the release to BOSH:

$ bosh upload release

BOSH will list out the release info again, and push the bits up to your BOSH instance.

Deploying our Release

Remember in Part 2 when the majority of our work was creating the deployment manifest? What if I said bosh-gen could handle all of it with a command-line call and a few flags? Well, for such a basic release like the one we’re creating, it actually can. This is where bosh-gen really shines, and where my work was adding vSphere support. Take a look at the flags before we call this:

$ bosh-gen help manifest
Usage:
  bosh-gen manifest NAME PATH UUID

Options:
      [--force]                    # Force override existing target manifest file
  -c, [--cpi=CPI]                  # Specify the CPI fields to be generated, e.g. "aws" or "vsphere"
  -d, [--disk=DISK]                # Attach persistent disks to VMs of specific size, e.g. 8196
  -a, [--addresses=one two three]  # List of IP addresses available for jobs
  -r, [--range=RANGE]              # IP Netmask
  -g, [--gateway=GATEWAY]          # IP Gateway
      [--dns=one two three]        # DNS Servers
  -w, [--workers=N]                # Number of worker VMs to spawn when compiling packages
  -n, [--network=NETWORK]          # [vSphere Only] Network to bind VMs to

A bit intense, but the mapping will make sense when we actually call it. We’ll need one more piece of information, which is the BOSH UUID. Back in Part 2, we grabbed this with the “bosh status” command as such:

$ bosh status
Target         micro (http://192.168.0.100:25555) Ver: 0.4 (00000000)
UUID           XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX
User           admin

Now, all that’s left is to create the manifest, and deploy our fresh redis release. Here’s an example bosh-gen manifest command, adjust appropriately for your network settings:

$ bosh-gen manifest redisdev001 . XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX -c vsphere -a 192.168.0.200 -r 192.168.0.0/24 -g 192.168.0.1 --dns 192.168.0.1 192.168.0.2 -w 1 -n "Development Network"

Breaking that down, we want to create a manifest named redisdev001, passing the path to our release (.) and our BOSH UUID (XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXX), using the vSphere CPI (-c vsphere), assign it the static IP (-a 192.168.0.200), subnet mask 255.255.255.0 (-r 192.168.0.0/24), our network gateway (-g 192.168.0.1) and our two DNS server (–dns 192.168.0.1 192.168.0.2). We only want one worker VM (-w 1) and in vSphere, we put these types of VMs on the network labeled Development Network, so we tell BOSH to do so (-n “Development Network”). You’ll see bosh-gen spit out redisdev001.yml, and if you look through it, it will look familiar. This is an entire manifest to deploy our release that needs no changes. Currently, for more complex releases, job-specific properties will need to be added, but this is a great start to work off of.

Finally, let’s target this deployment, and spin up our redis server!

$ bosh deployment redisdev001.yml
$ bosh deploy

If we get the all-green, we’re set to go! Let’s open up our redis-cli and point it to our new server and run a few test commands

$ redis-cli -h 192.168.0.200
redis 192.168.0.200> set foo bar
OK
redis 192.168.0.200> get foo
"bar"
redis 192.168.0.200> config get dir
1) "dir"
2) "/var/vcap/store/redis"

Congratulations, you’ve created a release to spin up a redis server. The nice thing is in the future, since the release is already is in our BOSH server, we simply have to point to our manifest, change a few settings, and run “bosh deploy”.

Related Links

• Using BOSH with vSphere – Part 1 – Deploying Micro BOSH
• Using BOSH with vSphere – Part 2 – Deploying a release
• Using BOSH with vSphere – Part 3 – Scaling a deployment
• Create and deploy a new release – Dr Nic
• bosh-gen

In part 1, we took a look at deploying Micro BOSH, the absolute minimum needed to start deploying releases. To keep definitions clear, a release in the BOSH world is a gathering of all of the parts needed to deploy a large-scale, distributed system. They consist of a handful of parts that, when put together, put on a mesmerizing orchestration of automated deployment. As with the last post, I won’t waste your time describing all of the parts, and as usual, I’ll point you to the OSS Docs (look near the bottom).

VMware provides a few releases in their Github organization, one for Cloud Foundry, one for a larger scale BOSH deployment, one for their Gerrit+Jenkins deployment (used in their code review and build process), and the one we’re working with, a nice simple WordPress deployment. This release handles deploying a MySQL server, a few Apache + WordPress servers, and an nginx front-end.

The README in the Github repository does a very good job explaining all of the directories, so I’ll let that shine where it does best. For us, the only part we’re really concerned with is the wordpress.yml file, which describes how to integrate the deployment in with our IaaS layer, and the steps that need to take place before we can deploy a release.

So first thing’s first, let’s go ahead and log into our Micro BOSH. If you followed through with part 1, you should already be logged in. If not, let’s take care of that real quick:

$ cd ~/deployments
$ bosh micro status
Stemcell CID   n/a
Stemcell name  n/a
VM CID         n/a
Disk CID       n/a
Micro BOSH CID [CID]
Deployment     [Path to micro_bosh.yml]
Target         http://[IP Address]:25555

$ bosh target [IP Address]:25555

You will be asked for a username and password, which by default, the username is admin and the password is admin.

Much like when we deployed Micro BOSH, we’ll need to download a public stemcell from VMware’s servers…

$ bosh public stemcells
+-------------------------------+---------------------+
| Name                          | Url                 |                                                                                                                                                                      |                               |                     |
+-------------------------------+---------------------+
| bosh-stemcell-0.4.7.tgz       | https://blob.cfbl...|
| micro-bosh-stemcell-0.1.0.tgz | https://blob.cfbl...|
| bosh-stemcell-0.3.0.tgz       | https://blob.cfbl...|
| bosh-stemcell-0.4.4.tgz       | https://blob.cfbl...|
| bosh-stemcell-aws-0.5.1.tgz   | https://blob.cfbl...|
| bosh-stemcell-0.5.2.tgz       | https://blob.cfbl...|
+-------------------------------+---------------------+

Since we’re looking to deploy to vSphere, let’s grab the most recent one and place it right next to our Micro BOSH stemcell

$ cd ~/stemcells
$ bosh download public stemcell bosh-stemcell-0.5.2.tgz

Once that download is done, let’s push it up to our Micro BOSH

$ bosh upload stemcell bosh-stemcell-0.5.2.tgz

This will push the stemcell to the Micro BOSH server, and then create the VM within vSphere which BOSH will use as a base for both the worker VMs (which compile packages) and the VMs that make up the deployment itself.

Next up is to download and initialize the release, and then upload it to our Micro BOSH

$ cd ~
$ mkdir releases
$ cd releases
$ git clone https://github.com/cloudfoundry/bosh-sample-release.git
$ cd bosh-sample-release

Let’s make sure we have at least the right version, director UUID

$ bosh status
Target         micro (http://[IP ADDRESS]:25555) Ver: 0.4 (00000000)
UUID           [Director UUID]
User           admin
...

Open up the wordpress.yml file and let’s make a few quick changes

1. Change the “director_uuid” field to match the “UUID” field we saw when running “bosh status”. This will match a pattern like “xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx”
2. Change the “name” field under “release” to read “wordpress”
3. Change the “version” field under “release” to read “1″

Before we bother configuring the rest of the wordpress.yml file, let’s go ahead and create and upload the release to our Micro BOSH. Notice the prompt for the release name.

$ bosh create release --force --with-tarball
Syncing blobs...

Building DEV release
---------------------------------
Please enter development release name: wordpress
...
Release version: 1
Release manifest: /home/bosh/releases/bosh-sample-release/dev_releases/wordpress-1.yml
Release tarball (63.0M): /home/bosh/releases/bosh-sample-release/dev_releases/wordpress-1.tgz

$ bosh upload release dev_releases/wordpress-1.tgz

This is where I’d like to remind you, the reader, that I’m still figuring a lot of this out myself, so please excuse any incorrect information that may be provided. What’s next though is a YML file that I used to deploy this 5 VM deployment of WordPress, and how I understood each field. I’ve left some as they were when I deployed the release, others I’ve removed and explained what the field is. This is where most of the work will be, so read carefully, configure carefully, and remember. If you mess up, you can always delete the deployment and try again.

Go ahead and open up the wordpress.yml file in the text editor you’re most comfortable in, we’ll be here for a bit.

---
name: wordpress
director_uuid: (Director UUID as reported from "bosh status")

release:
  name: wordpress
  version: 1

compilation:
  workers: 4
  network: default
  cloud_properties:
    ram: 2048
    disk: 8096
    cpu: 2

# this section describes how updates are handled
update:
  canaries: 1
  canary_watch_time: 30000
  update_watch_time: 30000
  max_in_flight: 4
  max_errors: 1

networks:
  - name: default
    subnets:
    - static:
      - (Range of static IPs to use for this network, ex, "192.168.0.101 - 192.168.0.106")
      range: (Subnet mask for VMs, for above example, this could be "192.168.0.0/24")
      gateway: (Gateway IP, for above example, this could be (192.168.0.1)
      dns:
      - (DNS IP #1)
      - (DNS IP #2)
      cloud_properties:
        name: (VLAN Name in vSphere)
  - name: dmz
    subnets:
    - static:
      - (Static IP to use for this network, we only need one, so this could be "192.168.0.107")
      range: (Subnet mask for VMs, for above example, this could be "192.168.0.0/24")
      dns:
      - (DNS IP #1)
      - (DNS IP #2)
      cloud_properties:
        name: (VLAN Name in vSphere)

resource_pools:
  - name: infrastructure
    network: default
    size: 6
    stemcell:
      name: bosh-stemcell
      version: 0.5.2
    cloud_properties:
      cpu: 1
      disk: 8192
      ram: 4096

jobs:
  - name: mysql
    template: mysql
    instances: 1
    resource_pool: infrastructure
    persistent_disk: 16384
    networks:
    - name: default
      static_ips:
      - (Static IP for MySQL VM, ex. "192.168.0.101")

  - name: wordpress
    template: wordpress
    instances: 4
    resource_pool: infrastructure
    networks:
    - name: default
      static_ips:
      - (Static IPs for Apache/Wordpress VMs, ex. "192.168.0.102 - 192.168.0.105")

  - name: nginx
    template: nginx
    instances: 1
    resource_pool: infrastructure
    networks:
    - name: default
      default: [dns, gateway]
      static_ips:
        - (Static IP for default network, ex. "192.168.0.106")
    - name: dmz
      static_ips:
        - (Static IP for DMZ network, ex. "192.168.0.107")

properties:

  wordpress:
    admin: (Admin login email address)
    port: 8008
    servers:
      - (Apache/Wordpress VM IP #1, ex. "192.168.0.102")
      - (Apache/Wordpress VM IP #2, ex. "192.168.0.103")
      - (Apache/Wordpress VM IP #3, ex. "192.168.0.104")
      - (Apache/Wordpress VM IP #4, ex. "192.168.0.105")
    servername: (Domain name for wordpress site, ex. "examplewordpress.example.com")
    db:
      name: wp
      user: wordpress
      pass: w0rdpr3ss
    auth_key: random key
    secure_auth_key: random key
    logged_in_key: random key
    nonce_key: random key
    auth_salt: random key
    secure_auth_salt: random key
    logged_in_salt: random key
    nonce_salt: random key

  mysql:
    address: (IP Address for MySQL server, ex. "192.168.0.101")
    port: 3306
    password: rootpass

  nginx:
    workers: 1

Whew, ok, so that should take care of all of the configuration. In the future, I’d like to break down ALL of these fields, but for now, this will get our sample release up and running. The nice thing about these YML deployment manifests though is that once we do all the configuration here, in one place, all of the heavy lifting is done. As opposed to hand-crafting 6 VMs, installing and configuring MySQL, Apache, nginx, etc., we simply do the following to actually deploy this release…

$ bosh deployment wordpress.yml
$ bosh deploy

… and with that, we set the machine in motion. BOSH will scan through the release and the deployment manifest, and do the following…

1. Make sure that all of the requested resources (network, etc) are available in vSphere
2. Spin up 4 VMs, as defined as the “workers” field under “compilation”. If you want more, or hitting resource limitations and need less, look at changing this field. This is the max number of simultaneous worker VMs that are created/running at any given time. In this release, we have 6 packages that need built. BOSH will spin up 4 VMs and start compiling the first 4 packages. When one finishes, that VM is destroyed and another is created, which compiles the 5th package, and the same happens with the 6th.
3. Once all of the packages are compiled, all of the worker VMs are destroyed, and 6 VMs are created to run the actual release. This number comes from the “size” field under “resource_pools”. This field must be greater than or equal to the sum of all of the “instances” fields for all of the jobs.
4. BOSH then installs the packages for each of the jobs, applies the specifications, and starts the services in the VMs

If all goes well, you’ll get an all-green in the end, and you can browse to the IP address of the nginx (in this example, either “192.168.0.106″ or “192.168.0.107″, or both) in a web browser and be greeted by the famous WordPress 5-minute setup.

That concludes part 2. For part 3, we’ll take a look at scaling up this deployment (unless someone else has a better suggestion, I’d love to hear it).

Related Links

• Using BOSH with vSphere – Part 1 – Deploying Micro BOSH