OpenWFEru over Amazon SQS
OpenWFE as a workflow engine coordinates the activities of the participants in the enacted business processes. This is done by dispatching a workitem to a participant which will send the workitem back with potential modifications to its payload once the activity is over.
In order to integrate OpenWFEru in a wide range of applications, various participants and listeners have been developed.
The last addition deals with Amazon SQS :
Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed application components performing different tasks, without losing messages or requiring each component to be always available.
The cost of usage for the Amazon SQS is very cheap : USD 0.0001 per message sent (USD 0.20 per GB of data transferred), in short, 1 USD buys you 5000 messages.
An SqsParticipant is a participant sitting behind an Amazon queue. The OpenWFEru engine puts the workitems for this participant in the queue, wrapped in a message. The application represented by the participant will then fetch the message and use the workitem inside of it.
Once the application has finished its activity, it will remove the message from the queue and wrap the [modified] workitem in a new message that will be put in the Amazon queue that the engine is listening to (via a SqsListener).
For human oriented participants (worklists) using directly the Amazon queue as a reliable workitem storage is a ‘natural’ design.
In order to have a maximum portability of the workitems among the participants connected via SQS, the SqsParticipant and the SqsListener by default wrap workitems as simple Ruby YAML hashes.
OpenWFEru uses the SQS REST API to benefit from the upper 256K limit for message size.
Such a process definition :
could leverage SQS participants in this way :
Participants are expected to feed the workitems back to the engine on the “engineq” queue :
(The engine polls the “engineq” every one minute and thirty seconds).
Of course, these participants may be used by multiple process definitions.
What else is there to say ? Amazon does all the heavy work for us, we’re just left with the fun part.
In the medium term, I’d like to release the tools necessary for having a fully AWS backed OpenWFEru engine, no local storage of process instances or workitems, all persistence aspects done via Amazon SQS and Amazon S3. Very lightweight.