In a previous post regarding AMQP 1.0 I discussed the differences between AMQP 0.8 and AMQP 1.0.  Recently I was co-presenting at ApacheCon EU and I found a useful way of describing why AMQP 1.0 is so important and different. 

In the AMQP 0.8 world the focus was on exchanges and queues and bindings and therefore on the broker.  Once you have 'a broker' then the focus becomes very much on the broker.  If a new feature is required, e.g. last-value-queues (LVQs), then you stick it in the broker. The problem is that you can end up with a large Swiss Army knife broker that is jammed with lots of different features. Such a broker becomes difficult to maintain. A change in a major feature like High Availability can break some of the other features, or at the very least requires extensive testing.

So what happens is that developers and users becomes obsessed with the broker.  All the focus is on the broker.  This is not what AMQP is all about.  

So AMQP 1.0 is not about brokers.  It's about the protocol.  And what that allows is projects like Qpid Proton.  Qpid Proton is a AMQP library that focuses on the protocol. It's a toolkit for application developers and third party ISV developers.  It's about embedding AMQP into anything and everywhere it makes sense.  You can even include it in your middleware broker to provide AMQP support.  Indeed that is what Qpid proper is doing in order to provide AMQP 1.0 support. That is what ActiveMQ is doing to provide AMQP support.  I've used Proton to provide AMQP support for OpenMAMA.

But even more importantly, the effect is to take the focus away from artifacts like a broker and back to the protocol. That way, by using Qpid Proton, we can build all sorts of AMQP enabled assets.  Perhaps lightweight routers like I mentioned in that previous post. 

For example, consider a situation where you are using Qpid to provide AMQP messaging for external business partners.  One of the things you may wish to do is filter out illegal AMQP traffic from entering  your intranet.  Before Qpid Proton you might solve this by putting several Qpid brokers in your DMZ behind a VIP.  Those brokers would forward legal messages to your brokers behind the firewall.  But putting the "Swiss Army Knife' broker in the DMZ merely to forward messages seems a bit over kill.  Instead it would be better to have a lightweight router with security filtering in the DMZ.  That's all it does: filter unwelcome traffic out and forward welcome traffic onward. 

Once you focus on the protocol and not on a broker a more flexible network topology becomes possible.  AMQP can become as pervasive as HTTP for more message oriented scenarios.  HTTP for synchronous requests and AMQP for more asynchronous use cases.

If you want to get AMQP support for your application or software product then go git it today at:

Enhanced by Zemanta
Yesterday the first phase of the OpenMAMA Qpid bridge to provide support for AMQP was completed. Jacob Benet (NYX) , Mike Schonberg (NYX and OpenMAMA) and I have been working on it for a few months.  It wasn't several months effort but unfortunately we got busy and distracted with other work and coordinating time together for calls wasn't always easy.  But yesterday it all paid off.  

The first phase involves getting simple messages publishing and being received by a subscriber over the bridge.  It involves message blobs with no rich type support yet. But the cool thing is that it involved AMQP 1.0 support.  We integrated Qpid's Proton library which provides the AMQP 1.0 support.  

This is the third "Open" integration effort I've been involved in with Qpid. Both OpenShift and OpenStack use messaging to manage their infrastructure.  I built plugins for their management layer to incorporate Qpid for AMQP support. As a result of that work I got involved in the OpenMAMA AMQP bridge effort with Mike and Jacob.

Mike provided deep knowledge of the OpenMAMA API and the build system.  Jacob, another experienced OpenMAMA developer, provided bridge documentation and a stubbed out "NULL" bridge so that I could "fill in the blanks" with the Qpid API calls. Both provided lots of hand-holding for me as I navigated through the OpenMAMA APIs. I'll spend some time on a future blog post explaining the process of the integration effort.

There is still plenty of work to do.  First we have to clean up the code - there are lots of extra trace calls in there to be removed. We will need to do lots of testing. Integration of AMQP typed payloads will need to be support. Testing with a broker will be required (but it is cool that you don't actually require a broker with AMQP 1.0 if you don't want one.) 

Having said that, yesterday was a milestone. Watching the subscriber example program receive messages over the bridge from the publisher example was very exciting.  For information on the bridge code check the OpenMAMA/Qpid git repository.  I encourage OpenMAMA folks to pull the code from the repo and play around with the examples and provide some feedback.  

Gordon Sim is a senior software architect and developer at Apache Software Foundation and employed at Red Hat. Gordon lives in Scotland. He is one of the main contributors to the Apache Qpid project. Gordon replied on the Qpid users mail list (Aug 29th 2012) to my post The Journey to AMQP 1.0 with Apache Qpid and he was kind enough to give me permission to post it here. Thank you Gordon.


From Gordon Sim:


Nice post, I particularly like the title! It has indeed been a journey and the Qpid community - all the users who have raised JIRAs, highlighted limitations of the software and earlier protocols, all the developers who have contributed features, patches and ideas - has been instrumental in getting AMQP to this very exciting point.

Change is unsettling, but it can also bring great benefits. I'm grateful that you persevered and glad you can now appreciate the greater power, flexibility and reach of the AMQP 1.0 model.

I was more ambivalent about exchanges than you. I felt that because they were the most visible and distinctive feature of early AMQP, they got too much attention at the expense of other important aspects of messaging.

In your post you write:

"The real power of 0.8 was NOT in the exchanges but instead it was in the bindings - the routing [...]  the power of AMQP is that it is a really powerful router of messages"


I would argue that routing is one of the benefits of messaging intermediaries in general (along with store-and-forward capabilities). The power of AMQP is that it enables interoperability between different intermediaries and/or applications. This lets you construct your messaging network from diverse components which gives the system designer a richer more flexible set of tools.

I don't think we are disagreeing there, just providing slightly different emphasis.

Also:

"The broker of 0.8 and it's exchanges, queues and bindings provided a brilliant and dynamic routing capability mainly through decoupling the sender from the endpoint."


The routing capabilities of early AMQP were actually fairly rudimentary and inflexible in my opinion. They also fell a little short in the decoupling since the destination you sent to dictated the only possible routing algorithm you could use for those messages. The shortcomings could be addressed by extensions, but I think 1.0 provides a far better foundation, standardising the simple patterns while allowing freedom for richer patterns to emerge.

I do agree that the addressing syntax as supported by the AMQP 0-10 implementation of the qpid::messaging API is at times a little complex and awkward. I think that was largely due to the fact that it was the control point for all bridging between the conceptual model of AMQP 1.0 and the 0-10 protocol which is actually used underneath.

Again, this was a stage in the journey we have travelled. We wanted simple things to be simple, more complex things to be possible and transition to 1.0 to be as painless as possible.

I am confident the mapping onto 1.0 will be much more direct, transparent and intuitive. I'm working on a 1.0 implementation of the API right now and will post more detail on that for comment as I make progress.

To be honest my journey to AMQP 1.0 with the Qpid implementation was a little bumpy.  My job requires me to be involved in lots of different technologies: from the micro Realtime kernel (no I have not contributed to the kernel - I just had to learn and teach others about realtime use cases), to the macro grid and cloud technologies;  with messaging and virtualization etc. in between.  Switching hats all the time, and often at the same meeting, is challenging.  I do like to keep my hands in the technology - to keep me honest. The area I have most rolled up my sleeves is with messaging but it is often in short bursts.

As a result I have been less involved in the AMQP specification as I'd like to be.  And that is: not at all - unless you count any influence I have with the involved engineers at Red Hat. So I did not always see the progression to AMQP like those involved and I got very confused along the way.

I "loved" exchanges. They were part of the first AMQP 0.8 specification.  They made so much sense.  When Qpid made changes to the "addressing" I wasn't all that happy.  It seemed confusing and complicated. Why were we moving to a new and simpler API but then forcing a complicated and confusing addressing mechanism?   (Compound that with inconsistencies between the C++ and Java APIs and my frustration grew worse.)  But it was all part of the move to AMQP 1.0 I was told.  And I was getting a bit fearful of what AMQP was turning into.  It was explained that there was a move away from exchanges and queues to "nodes".  I was beginning to get deflated. Why move from something that worked so great and was so cool to explain the value of to people, to something with abstractions like nodes and addresses?

So my colleagues went to work on me.  Understanding that I'm not nearly as foolish as I let on they took the time to invest in convincing me that AMQP 1.0 was way better.  (Or perhaps they thought I was a fool but still needed to be convinced.)  It's been a long road (especially with all the travel interruptions).  But I get it now.

The power of AMQP is in the addressing. In the AMQP 0.8 specifications this was wrapped up in exchanges and queues and bindings.  The real power of 0.8 was NOT in the exchanges but instead it was in the bindings - the routing.  After all, the three (or more) different exchanges were really just specializations of the same abstraction - a router or the default exchange.   I have always maintained that the power of AMQP is that it is a really powerful router of messages.  The broker of 0.8 and it's exchanges, queues and bindings provided a brilliant and dynamic routing capability mainly through decoupling the sender from the endpoint. 

With AMQP 1.0  all that is in the addressing with much more power. My clients/publishers/senders (whatever you wish to call them) do not care how the endpoint is implemented. And by the way that endpoint was always a broker in 0.8.  So you are not sending to an exchange but to an address. How it gets there or where that is is neither here nor there. And with AMQP that can end up to be many different endpoints or applications. And the receivers/subscribers/servers don't care where a message came from (except from a security perspective) or how that was implemented.  

I can still have a broker.  But a broker becomes much more how I used to describe a broker: a router.  I can still implement an exchange in my router.  I can have different flavors of routers instead of a one size fits all broker (0.8).  Routers for various tasks. And, by the way, with the Qpid proton library approach I can stick that in anything and anywhere!

So with AMQP my messaging topology becomes much more of a network.  And that is much more powerful.  AMQP is to messaging networking to what HTTP is for request-response internet application communications.  In fact many internet applications are going to be relieved of some unorthodox HTTP trickery by switching to AMQP. People will use only HTTP where it makes sense and use AMQP everywhere else. (I'll post an HTTP and AMQP comparison at another time.)  Lots of new types of mobile and cloud applications become much more reasonable with AMQP.

Yesterday I pointed you to where you can start playing with AMQP 1.0 at Apache Qpid.  Stay posted for more.

AMQP 1.0 and Qpid Proton

| No TrackBacks
For those of you that want to get started with AMQP 1.0 look no further than Qpid Proton. It's source repository can be found here. Also at gitub.

What is Proton?  From the site:

"Proton is a high performance, lightweight messaging library. It can be used in the widest range of messaging applications including brokers, client libraries, routers, bridges, proxies, and more. Proton is based on the AMQP 1.0 messaging standard."

I've been trying to integrate with Openmama API.  The Messenger API is what I went with and it seems very simple. Trying to integrate with the Openmama API has been more difficult as the Messenger interface is much simplier than the Openmama interface and there arew other considerations. More on that in a later post.



*dust dust ..... wipe ... blow ... dust dust ...*

It's been way too long since I blogged. So long, in fact, that I couldn't log back in. With some help from Moveable Type Forums I managed to get logged back in today.

Last week I was at Red Hat Summit 2011 in Boston.  What a buzz.  Red Hat's new PaaS offering, OpenShift, was announced and demonstrated. Really sweet.  I contributed a very small amount of code to the OpenShift Express offering. It was an AMQP plug-in for the management framework MCollective using MRG Messaging with the Ruby/C++ client.  The OpenShift team are a pleasure to work with. I look forward to contributing some more. 

I had the privilege of speaking at Summit with Lans Carstensen from DreamWorks Animation.  We did a session called Red Hat Enterprise MRG at DreamWorks: Grid at Massive Scale, Utilization, and Throughput.  You can download the presentation from the Summit Presentation here.  All presentations are on the Red Hat Summit 2011 Presentation Page.  That version of the presentation contains all the meat but lacks all the fun stuff, including Kung-Fu Panda 2 characters on some of the slides.  We had a lot of fun, as can be gauged from the first of many questions we had, "how many Red Bulls did you guys have?"  He had only been drinking coffee but we were really pumped about the presentation.


My colleague Matt (spinningmatt) posted a really useful article, Submitting Jobs with AMQP on a Condor Based Grid. The article includes an example C++ program that uses a low-latency job submission feature of MRG.

The low-latency feature uses AMQP (MRG's Messaging) to deliver job workloads to a grid's execution nodes.  This bypasses the job scheduler (Condor's schedd).  Instead a special daemon on the execute node consume jobs off an AMQP queue.  It's a pull versus push model.

Why is this useful?  Well originally the intent was for sub-second jobs in the financial services industry.  Consider a grid of index calculation applications. Each calculation may take less than a second to perform.  The applications can't wait for a job scheduler to decide that their execution nodes are available, then schedule a job onto the node, then push the job out there.  Instead jobs are placed on AMQP queues.  As soon as an execute node is free to perform work it pulls the next job form the queue. There is barely any latency between jobs.

Of course as soon as you do this for one specific use, sub-second jobs, then others see the advantages too. This feature doesn't need to be sub-second jobs. MRG customers from various industries now see the advantage of this feature.

There are some considerations. For example, should the entire grid be untilized this way or should a specific portion of the grid be carved off for low-latency workloads?  If the jobs are sub-second and high volume, should they be reported to a management console (this could cause quite a bit of clutter) or just logged? How should failed jobs be managed under different scenarios? e.g. in sub-second transactions a failed job may have missed its window of being useful and therefore there is little point in resubmitting. The answers to these quesitons will depend on the type of low-latency workload.

If you are interested in more information on this topic please remember to check out Matt's post.

Who is IPBabble

William Henry IPBabble is the personal blog of William Henry.

William has over 20 years experience in software development and distributed computing and holds a M.Sc from Dublin City University. He is currently working in the office of CTO at Red Hat, in the Emerging Technologies team. This weblog is not funded by Red Hat.

Posts are intended to express independent points of view, but understand that there is probably a bias based on the influence of working with standards based middleware for over a decade. (See disclaimer below)

Recent Comments

  • esdee: nice article, thanks for sharing! i also loved 0.8 exchanges, read more

Recent Assets

  • WilliamHenry_crop.jpg
  • DBipbabble.com.gif
  • bannerlanscape1.gif
  • bannerlanscape.gif
  • William_Henry.jpg
  • Logo_Tokyo.gif

January 2013

Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

Disclaimer

The views expressed in this blog are solely the personal views of the author and DO NOT represent the views of his employer or any third party.

Find recent content on the main index or look in the archives to find all content.