processi

about processes and engines

OpenWFEru over Amazon SQS

I already said I wanted to implement something with Amazon’s Simple Queue Service. Now it’s done : OpenWFEru features an SqsParticipant and an SqsListener.

OpenWFE as a workflow engine coordinates the activities of the participants in the enacted business processes. This is done by dispatching a workitem to a participant which will send the workitem back with potential modifications to its payload once the activity is over.

In order to integrate OpenWFEru in a wide range of applications, various participants and listeners have been developed.

The last addition deals with Amazon SQS :

Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed application components performing different tasks, without losing messages or requiring each component to be always available.

The cost of usage for the Amazon SQS is very cheap : USD 0.0001 per message sent (USD 0.20 per GB of data transferred), in short, 1 USD buys you 5000 messages.

An SqsParticipant is a participant sitting behind an Amazon queue. The OpenWFEru engine puts the workitems for this participant in the queue, wrapped in a message. The application represented by the participant will then fetch the message and use the workitem inside of it.

Once the application has finished its activity, it will remove the message from the queue and wrap the [modified] workitem in a new message that will be put in the Amazon queue that the engine is listening to (via a SqsListener).

For human oriented participants (worklists) using directly the Amazon queue as a reliable workitem storage is a ‘natural’ design.

In order to have a maximum portability of the workitems among the participants connected via SQS, the SqsParticipant and the SqsListener by default wrap workitems as simple Ruby YAML hashes.

OpenWFEru uses the SQS REST API to benefit from the upper 256K limit for message size.

Such a process definition :

sqs_procdef.png

could leverage SQS participants in this way :

sqs_part.png

Participants are expected to feed the workitems back to the engine on the “engineq” queue :

sqs_listener.png

(The engine polls the “engineq” every one minute and thirty seconds).

Of course, these participants may be used by multiple process definitions.

What else is there to say ? Amazon does all the heavy work for us, we’re just left with the fun part.

In the medium term, I’d like to release the tools necessary for having a fully AWS backed OpenWFEru engine, no local storage of process instances or workitems, all persistence aspects done via Amazon SQS and Amazon S3. Very lightweight.

Update : the library OpenWFEru uses to connect to Amazon SQS has been released as an independent gem. It also has its own documentation webpage.

Written by John Mettraux

March 13, 2007 at 2:20 am

Posted in amazon, bpm, openwferu, ruby, sqs, workflow

2 Responses

Subscribe to comments with RSS.

  1. […] posted previously on how we use Amazon SQS as a workitem queue […]

  2. […] queue service The OpenWFEru SQS connector got some coverage from Jinesh in his Super Queue Service blog […]


Comments are closed.

%d bloggers like this: