IPBabble

My colleague Matt (spinningmatt) posted a really useful article, Submitting Jobs with AMQP on a Condor Based Grid. The article includes an example C++ program that uses a low-latency job submission feature of MRG.

The low-latency feature uses AMQP (MRG's Messaging) to deliver job workloads to a grid's execution nodes.  This bypasses the job scheduler (Condor's schedd).  Instead a special daemon on the execute node consume jobs off an AMQP queue.  It's a pull versus push model.

Why is this useful?  Well originally the intent was for sub-second jobs in the financial services industry.  Consider a grid of index calculation applications. Each calculation may take less than a second to perform.  The applications can't wait for a job scheduler to decide that their execution nodes are available, then schedule a job onto the node, then push the job out there.  Instead jobs are placed on AMQP queues.  As soon as an execute node is free to perform work it pulls the next job form the queue. There is barely any latency between jobs.

Of course as soon as you do this for one specific use, sub-second jobs, then others see the advantages too. This feature doesn't need to be sub-second jobs. MRG customers from various industries now see the advantage of this feature.

There are some considerations. For example, should the entire grid be untilized this way or should a specific portion of the grid be carved off for low-latency workloads?  If the jobs are sub-second and high volume, should they be reported to a management console (this could cause quite a bit of clutter) or just logged? How should failed jobs be managed under different scenarios? e.g. in sub-second transactions a failed job may have missed its window of being useful and therefore there is little point in resubmitting. The answers to these quesitons will depend on the type of low-latency workload.

If you are interested in more information on this topic please remember to check out Matt's post.

So over the weekend I posted an entry that introduced some of the AMQP benefits. That article was very upbeat on AMQP and Qpid.  Two areas of improvement are:

• Potential complexity of feature configuration
• Lack of concrete or industry specific examples
  

Improvement in the second area can really help solve the first issue.

As standards mature they can often get too complex. I remember when CORBA 2.2 came out and I read about Portable Object Adapters.  I was impressed at the flexibility it provided but I was also anxious that the number of configuration options would overwhelm and discourage many developers.  So while it's desirable to have lots of enhanced features, you also don't want lots and lots of features, and their combinations of configurations, to put off the novice.

Of course much of the complexity issue can be solved by providing good examples. However, examples of how to code a feature don't always give you the context. It is better if we can explain the feature with a real world example. When I developed the current C++ training examples for MRG Messaging we (at Red Hat) decided to try and build examples around a banking use case.  And recently I had an example called "tradedemo" committed to the Qpid project.

This week I saw some example code I submitted to the Apache/Qpid project committed. The example demonstrates a possible use case for some Qpid queue features.  I suspect we're going to see more examples with use case focus.  That is a very good thing and I'd encourage Qpid developers to take time to submit such examples.  We need a richer set. I'll discuss this and specifically my example on a later post.

This post is about AMQP and Qpid and is long overdue.

Look, no doubt I'm biased but the more and more I look at AMQP, the more I'm amazed with it's flexibility and it's depth and breadth of features. Its importance in the financial services industry is growing and its profile is rising in other industries (animation, energy, logistics, telecommunications etc.) and government.  

What makes AMQP so important? Well there are three basic characteristics that make it compelling :

• Standard wire protocol
• Multi-platform, multi-language support
• The model - simple, yet flexible
  

Standard wire protocol opens up the ecosystem and provides interoperability and freedom from vendor lock-in. We saw this with IIOP in the CORBA specification. Frankly I'm surprised and disappointed that it has taken this long to get a standard wire protocol for "messaging". I have messaging in quotes because no doubt that there are many that will point to various standards like IIOP, and HTTP and SOAP etc.  Those of you that ave been working on enterprise systems with a more asynchronous messaging approach know what I mean.

Because of the standard wire-protocol it means that anyone can write on any platform or language and still take advantage of AMQP. The Qpid project currently supports C++, Java (including JMS), Python, Ruby, .NET (C#).  In fact Microsoft have joined AMQP and joined Qpid! Hardware vendors like Cisco participate. The point is the ecosystem is growing.

This is very ambitious but as all these various languages and platforms with their various characteristics take advantage of AMQP? Is routing text messages through telecommunications exchanges, large image files to render farms, and low-latency high volume market data a reasonable requirement to be demanded off of one standard? That's where the flexibility comes in.  Unlike other messaging systems that might focus just on a single language programming API, or on a single technology like multicast, or on a single abstraction like a queue, AMQP's model is very flexible.  And yet it remains quite simple.


AMQP has three main abstractions:

• An exchange - where messages are posted
• A queue - where consumers find messages
• A binding - that binds a specific message to one or more queues

The decoupling of message sending from message delivery makes this model very powerful.  Rather than specifying a specific queue or forcing a broadcast across a network, AMQP provides a much more flexible model.

There is a nice explanation of these concepts here as part of the MRG documentation.

And so that is one of the reasons I joined Red Hat last May. AMQP and Qpid represent a new wave in enterprise application communication and I'm very excited in the interest and growth of adoption I'm seeing so far.  This along with the Condor based Grid technology is a very powerful distributed computing platform for government and industry.  It many ways it's what many of use were thinking would happen well over a decade ago.  The growth of higher speed networks and the ubiquity of messaging in every facet of our lives are pushing technology like this to broader adoption.  In the 21st century we are going to need a much more standard and ubiquitous messaging infrastructure.

Tomorrow I'll post about the example I committed to Qpid.

Last week I blogged on Using MRG Grid and EC2.  Since then I've been doing some work testing what we've been calling MRG Grid with EC2 Enhanced. 

Last week I was submitting an AMI as a job in the cloud.  MRG Grid with EC2 Enhanced is different. In this case the user submits an ordinary (vanilla) job but we have a preference to run it in the cloud (EC2).  In this case MRG Grid routes the job from the local grid to the cloud.  There are some routing rules depending on the characteristics specified for cloud instances. e.g. the job may require the use of high CPU instance types or 64bit based instances.

The job actually causes AMIs (specified in the routing rules) to be launched in the cloud and then uses Web services to transfer data in and out of the running instance.

These rules are in hooks that are fired because of certain job characteristics. There are both pre and post job execution hooks.  This allows for the transfer of data in and out.

Obviously the instance(s) running in the cloud need(s) an extra mechanism to get the jobs to run on it's local condor instance.  There is a daemon called caroniad that listens for incoming jobs and uses the AWS Web services.

I haven't had to do anything complex yet but so far the MRG / EC2 Enhanced testing has cost be about $4!

# Example Job Submission for MRG / EC2 Enhanced
universe = vanilla
executable = /home/testuser/ec2_tests/my_ec2_job.sh
output = ec2test.out$(PROCESS)
log = ulog$(PROCESS)
requirements = Arch == "INTEL"
should_transfer_files = yes
when_to_transfer_output = on_exit
transfer_executable = true
+WantAWS = True
+WantArch = "INTEL"
+WantCPUs = 1
queue 

# More CPU requirements get a large instance type
+WantCPUs = 2
queue 2

As I mentioned in the last blog entry this will all be documented in the MRG 1.1 Grid User Guide.  I've been providing some input based on my testing.

Rob Rati is the Red Hat engineer behind caroniad and he's been a great help getting me through the testing.

I've been testing some MRG Grid capabilities recently.  Last week it was using MRG Grid with virtualization and also using MRG Grid with Amazon's EC2.

Using Condor's ability to work with virtualization I was able to bring up a Xen image as a job on my Grid.  This image could have been a complete stack including an application.  I was happy enough just to get an existing RHEL 5.2 guest running. 

Toward the end of the week I was asked to look at our EC2 integration and take it for a test run. My first goal was just to get MRG Grid to launch an image in the cloud.  After I created a new AWS (Amazon Web Services) account  I searched and found a Fedora 6 image already up in EC2 so I decided I'd use that as my first test.  That way I didn't have to go through the task of building my own AMI (Amazon Machine Image).

Next I looked at what changes I'd need to make to MRG Grid's configuration on my laptop - i.e. the local Condor configuration.  The plan was that I'd launch my jobs from my laptop into the cloud.  It turns out that the current MRG Grid configuration (for MRG 1.1 release in December) already has all the default configuration I need in the global configuration file.  So I really had everything that I needed already, in terms of configuration.

Then I had to create a job submission file in order to have MRG Grid's scheduler (Condor's schedd) schedule the job in the EC2 cloud. I used something very similar to the folllowing:

# Note to submit an AMI as a job we need the grid universe
Universe = grid
grid_resource = amazon

# Executable in this context is just a label for the job
Executable  = my_amazon_ec2_job
transfer_executable = false

# Keys provided by AWS
amazon_public_key = cert-ABCDEFGHIJKLMNOPQRSTUVWXYZ.pem
amazon_private_key = pk-AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPP.pem

# The AMI ID
amazon_ami_id = ami-123a456b
amazon_user_data = Hello EC2!

# The keypair file if needed for using ssh etc
amazon_keypair_file = /tmp/keypair
# The security group for the job
amazon_security_groups = MY_SEC_GRP

queue

The key's I got from my AWS account.  The amazon_ami_id is the identifier of an AMI already loaded at EC2. And the security group is only needed if you want to open up certain ports (like 22 for ssh) on the image.  All the values for keys, ids and security groups  above are merely examples.

I then submitted the job and checked using the EC2 command line tools (I had downloaded earlier) and also using Elasticfox - a Firefox plug-in that provides a web condole to view and manage your AMIs.

I did a couple of other steps setting up that security group (above) with the command line tools so that I could ssh into the image. I have to say that running ssh to login to the running image in the cloud made it all seem pretty cool.

I then decided that I would write up my steps and submit them to the MRG Grid User Guide technical writer (God bless her).  So I wrote up the description of the basic steps of using MRG with EC2.  So when MRG 1.1 is released you'll find a more detailed decription of the steps in the MRG Grid User Guide here.

This week I have to work on the "Enhanced" MRG/EC2 integration.  This is where I take advantage of a running AMI in the cloud and use it as a MRG Grid execute node and therefore deploy jobs into it.  Thus making it just another resource available to my Grid, even though it is running externally in the cloud.

Oh, by the way all the EC2 testing on Thursday night and Friday morning cost be about $0.50.  That will be billed to my Amazon account. 

There is a great interview with Jim Whitehurst on ZDNet yesterday that starts on the topic of Cloud computing and virtualization and ends up speaking about the merits of open source.  Jim's ability to explain the value of these innovations is really clear and to the point. 

On the topic of Cloud, Jim explains that he believes that the economics of server farms will win out.  He then says "whether they are internal private grids or clouds, or whether those are public, or semi-private, that's still to be determined, because that's a lot around specific business models."  Therefore flexibility to use private grids or clouds or external clouds will be come very important for many companies.  This flexibility is one of the Objectives of MRG Grid.

Jim then does a great job of explaining how virtualization is becoming a commodity and should be moved to the operating system layer.  Then he explains the Red Hat business model for open source.  Again, he makes it very concise and clear.  Worth a read.

My blog difficulties have delayed this blog entry. I wanted to do a "Farewell IONA" blog (and I might do something later) and then an introduction to my new job at Red Hat.

After over 11 years at IONA (my couple of years at Aurora Technologies are officially included) I moved to Red Hat in May of this year.  Not long after my move I started hearing rumors of IONA's acquistion. And, of course, now IONA is part of Progress Software.

My move was prompted much less by IONA's fate than by what Red Hat is doing with MRG. It was time for change and MRG represents something very exciting going on in the industry. There is a lot I can and will be saying about MRG on this IPBabble and I'll only give a short introduction in this entry.

Red Hat's expansion into emerging technologies is an effort to provide more value to its customers. This work includes, among other things, virtualization, management and Messaging, Realtime and Grid (MRG).  I'll be explaining more about details in later posts but briefly:

• Messaging - an open source (Qpid), open standard (AMQP) based, platform independent, messaging infrastructure.
• Realtime - a predictable, low-latency linux kernel.
• Grid - an open source (Condor) HPC, HTC platform that provides the flexibility of local grids, remotes grids and cloud computing.

Of course there are lots more details but that is the very high level view. I'll be diving in a lot more on ipbabble.com.

I have a hybrid role: I work for Brian Stevens, the CTO, in the Office of CTO focused on MRG. In that way I'm very much an evangelist and customer focused solution architect. But in the same organization I'm an engineer working on MRG. So I have to contribute to the engineering effort too. It is both a very rewarding and very challenging role.  Getting up to speed on all the various aspects of MRG has been a steep learning curve despite my background in distributed computing and messaging. Thankfully I work with a really smart and dedicated team that has been generous with sharing their knowledge.  I'm sure I'll be introducing some of them on this blog in the future.


UPDATE: Thank you to Lana for spotting the "bog" typo.