In a previous post regarding AMQP 1.0 I discussed the differences between AMQP 0.8 and AMQP 1.0.  Recently I was co-presenting at ApacheCon EU and I found a useful way of describing why AMQP 1.0 is so important and different. 

Yesterday the first phase of the OpenMAMA Qpid bridge to provide support for AMQP was completed. Jacob Benet (NYX) , Mike Schonberg (NYX and OpenMAMA) and I have been working on it for a few months.  It wasn't several months effort but unfortunately we got busy and distracted with other work and coordinating time together for calls wasn't always easy.  But yesterday it all paid off.  

Gordon Sim is a senior software architect and developer at Apache Software Foundation and employed at Red Hat. Gordon lives in Scotland. He is one of the main contributors to the Apache Qpid project. Gordon replied on the Qpid users mail list (Aug 29th 2012) to my post The Journey to AMQP 1.0 with Apache Qpid and he was kind enough to give me permission to post it here. Thank you Gordon.

From Gordon Sim:

Nice post, I particularly like the title! It has indeed been a journey and the Qpid community - all the users who have raised JIRAs, highlighted limitations of the software and earlier protocols, all the developers who have contributed features, patches and ideas - has been instrumental in getting AMQP to this very exciting point.

Change is unsettling, but it can also bring great benefits. I'm grateful that you persevered and glad you can now appreciate the greater power, flexibility and reach of the AMQP 1.0 model.

I was more ambivalent about exchanges than you. I felt that because they were the most visible and distinctive feature of early AMQP, they got too much attention at the expense of other important aspects of messaging.

In your post you write:

"The real power of 0.8 was NOT in the exchanges but instead it was in the bindings - the routing [...]  the power of AMQP is that it is a really powerful router of messages"

I would argue that routing is one of the benefits of messaging intermediaries in general (along with store-and-forward capabilities). The power of AMQP is that it enables interoperability between different intermediaries and/or applications. This lets you construct your messaging network from diverse components which gives the system designer a richer more flexible set of tools.

I don't think we are disagreeing there, just providing slightly different emphasis.

Also:

"The broker of 0.8 and it's exchanges, queues and bindings provided a brilliant and dynamic routing capability mainly through decoupling the sender from the endpoint."

The routing capabilities of early AMQP were actually fairly rudimentary and inflexible in my opinion. They also fell a little short in the decoupling since the destination you sent to dictated the only possible routing algorithm you could use for those messages. The shortcomings could be addressed by extensions, but I think 1.0 provides a far better foundation, standardising the simple patterns while allowing freedom for richer patterns to emerge.

I do agree that the addressing syntax as supported by the AMQP 0-10 implementation of the qpid::messaging API is at times a little complex and awkward. I think that was largely due to the fact that it was the control point for all bridging between the conceptual model of AMQP 1.0 and the 0-10 protocol which is actually used underneath.

Again, this was a stage in the journey we have travelled. We wanted simple things to be simple, more complex things to be possible and transition to 1.0 to be as painless as possible.

I am confident the mapping onto 1.0 will be much more direct, transparent and intuitive. I'm working on a 1.0 implementation of the API right now and will post more detail on that for comment as I make progress.

To be honest my journey to AMQP 1.0 with the Qpid implementation was a little bumpy.  My job requires me to be involved in lots of different technologies: from the micro Realtime kernel (no I have not contributed to the kernel - I just had to learn and teach others about realtime use cases), to the macro grid and cloud technologies;  with messaging and virtualization etc. in between.  Switching hats all the time, and often at the same meeting, is challenging.  I do like to keep my hands in the technology - to keep me honest. The area I have most rolled up my sleeves is with messaging but it is often in short bursts.

For those of you that want to get started with AMQP 1.0 look no further than Qpid Proton. It's source repository can be found here. Also at gitub.

*dust dust ..... wipe ... blow ... dust dust ...*

It's been way too long since I blogged. So long, in fact, that I couldn't log back in. With some help from Moveable Type Forums I managed to get logged back in today.

Last week I was at Red Hat Summit 2011 in Boston.  What a buzz.  Red Hat's new PaaS offering, OpenShift, was announced and demonstrated. Really sweet.  I contributed a very small amount of code to the OpenShift Express offering. It was an AMQP plug-in for the management framework MCollective using MRG Messaging with the Ruby/C++ client.  The OpenShift team are a pleasure to work with. I look forward to contributing some more. 

I had the privilege of speaking at Summit with Lans Carstensen from DreamWorks Animation.  We did a session called Red Hat Enterprise MRG at DreamWorks: Grid at Massive Scale, Utilization, and Throughput.  You can download the presentation from the Summit Presentation here.  All presentations are on the Red Hat Summit 2011 Presentation Page.  That version of the presentation contains all the meat but lacks all the fun stuff, including Kung-Fu Panda 2 characters on some of the slides.  We had a lot of fun, as can be gauged from the first of many questions we had, "how many Red Bulls did you guys have?"  He had only been drinking coffee but we were really pumped about the presentation.

My colleague Matt (spinningmatt) posted a really useful article, Submitting Jobs with AMQP on a Condor Based Grid. The article includes an example C++ program that uses a low-latency job submission feature of MRG.

The low-latency feature uses AMQP (MRG's Messaging) to deliver job workloads to a grid's execution nodes.  This bypasses the job scheduler (Condor's schedd).  Instead a special daemon on the execute node consume jobs off an AMQP queue.  It's a pull versus push model.

Why is this useful?  Well originally the intent was for sub-second jobs in the financial services industry.  Consider a grid of index calculation applications. Each calculation may take less than a second to perform.  The applications can't wait for a job scheduler to decide that their execution nodes are available, then schedule a job onto the node, then push the job out there.  Instead jobs are placed on AMQP queues.  As soon as an execute node is free to perform work it pulls the next job form the queue. There is barely any latency between jobs.

Of course as soon as you do this for one specific use, sub-second jobs, then others see the advantages too. This feature doesn't need to be sub-second jobs. MRG customers from various industries now see the advantage of this feature.

There are some considerations. For example, should the entire grid be untilized this way or should a specific portion of the grid be carved off for low-latency workloads?  If the jobs are sub-second and high volume, should they be reported to a management console (this could cause quite a bit of clutter) or just logged? How should failed jobs be managed under different scenarios? e.g. in sub-second transactions a failed job may have missed its window of being useful and therefore there is little point in resubmitting. The answers to these quesitons will depend on the type of low-latency workload.

If you are interested in more information on this topic please remember to check out Matt's post.