processi

about processes and engines

Archive for the ‘workflow’ Category

ruote 2.2.0 released

Just released version 2.2.0 of ruote, a Ruby workflow engine. It interprets workflow definitions, routing tasks / work among participants.

Ruote.process_definition do
  alice :task => 'prepare offer'
  bob :task => 'revise offer'
  concurrence do
    david :task => 'revise offer'
    fred :task => 'revise offer', :if => '${offer.total} > 1000'
    elie :notification => 'offer for ${customer.name} (${customer.city}) out'
  end
  charly :task => 'submit offer'
  accounting :task => 'emit invoice'
end

 

the changelog

2.2.x

Why a 2.2.x ? It’s not that ruote 2.2.0 is not backward compatible but it now flags all the expression with a sub_id (formerly called a sub_wfid). Previously only the expressions in a subprocess would have a subid, now all the expressions have one. It prevents some nasty issues with concurrent-iterator and forget.

The second justification for a 2.2.x are stateful participants being dropped.

stateless participants

Before 2.2.0, participants could be registered as classes or instances. From now on, only participant classes can be registered. Each time a ruote worker dispatches a workitem to a participant it uses a new instance. Such “stateless” participants cannot share info via instance variables.

Block participants like

engine.register_participant 'total' do |workitem|
  workitem.fields['total'] = workitem.fields['items'].inject(0) { |t, (i, c)|
    item = Item.find(i)
    t = t + c * item.price
  end
end

are a bit harder to make “stateless”. But thanks to the ingenious Sourcify, grabbing the source of the block is not a problem. Small reminder, it grabs the source code of the block, not its closure.

Simply put, stateful participants have been dropped.

Now for the rest of the changes.

composite conditions

Conditions in process definitions were constrained to things like “${customer.level} == gold”, which works OK if you want to keep concise process definitions. Adding a quick ‘and’ should not require too many workarounds. This is now possible :

Ruote.process_definition do
  participant 'alice', :if => "${customer.level} == 'gold'"
  participant 'bob', :if => '${customer.level} == gold and ${customer.country} == Brazil'
end

Other idioms like “${customer_list} is empty”, “${customer} in ${customer_list}”, “${x} is null” are accepted. The tests are probably a more exhaustive source of info about those idioms.

Speaking of the dollar notation, it was all about strings, whatever the values it would turn them into strings. There is a new literal way for dealing directly with non string values.

on_error, on_terminate

The Engine instance has two new setters, on_error= and on_terminate=. For example, this

engine.on_error = 'supervisor'

states that the ‘supervisor’ participant (whatever you registered under that name) will receive a notification (a workitem) each time a process instance emits an error in the engine.

on_error and on_terminate accept participant names, subprocess names or directly process definitions :

engine.on_error = Ruote.process_definition do
  concurrence do
    administrator :msg => 'something went wrong'
    supervisor :msg => 'houston, we have a problem'
  end
end

filters

Thanks to a collaboration with Raphael filters are [back] in ruote.

They come in two forms : the filter expression (one-way filtering of passing workitems) and the :filter attribute (placing a filter around a process region).

The filter attribute may also point to participants (registered like any other participants), focused on workitem filtering (whereas regular participants pass work to the underlying, real, participant).

history

Prior to 2.2.0, ruote had no history, the worker activity was not ‘archived’. A ruote engine will now have a default history keeping in memory the most recent worker operations.

If you really need such a history, you’d better use/customize the StorageHistory class.

(well, you probably don’t need to keep record of all these operations).

sequel

There is a new storage implementation, ruote-sequel. As the name implies, it’s based on the excellent Sequel.

A nice addition to the list of storage implementations.

participants (again)

You register participants, they are all stateless, why not get a copy out for certain interactions ? Engine#participant is the counterpoint to Engine#register_participant.

Participants may provide their own timeout value by implementing the rtimeout method (the timeout given in the process definition, if any, will take precedence though).

expressions

The listen expression now reacts to a process instance entering or leaving a tag (a process instance region). See ruote and tags for an explanation. The Workitem class now has a dedicated __tags__ field containing a list of tag names the process instance (that emitted the workitem) is currently in.

Ruote now has a “switch” statement, it’s a given (also covers the new let expression).

The cancel_process expression has got a new alias “terminate”, concurrent_iterator can be shortened to “citerator” while the when expression can be written “once” (or “as_soon_as”).

There is a new let expression. The main usage is to isolate a set of subprocess definitions in a new scope (case like).

The registerp and unregisterp expressions let you register participants from process definitions (granted, you could do that from the consume method of a participant too).

It was present in ruote 0.9.x but got lost in the way. The lose expression and the :lose attribute are back (thanks Claudio). ‘forget’ and ‘lose’ are ways to put aside certain execution branches of a process instance.

next steps

Next steps ? Workers registering in the storage, pausing individual processes, having process supervise other processes, improved ruote-redis, …

 

Many thanks to all who helped along the way.

 

website : http://ruote.rubyforge.org
source : https://github.com/jmettraux/ruote
mailing list : http://groups.google.com/group/openwferu-users
irc : freenode.net #ruote

 

Written by John Mettraux

March 2, 2011 at 10:46 am

Posted in ruby, ruote, workflow

ruote, process state and tags

A : “What state is this process in ?”
B : “Well, it’s running”
A : “I know, that’s not what I meant. Where is it now ?”
B : “Let’s check which participants have workitems (tasks) about this process now”
A : “No, I mean, the document is it still being reviewed ? What’s its state ?”
B : “Ah you mean, the document state, not the process state ?”
A : “Is there a difference ?”
B : “Well, if a business process deals with the state of multiple resources, you can’t equate process state and resource state”

Initially, for ruote (a ruby workflow engine), the “state of a process” was limited to “the set of visible workitems for that process”.

Ruote.process_definition do
  cursor do
    production
    concurrence do
      qa1
      qa2
    end
    rewind :unless => '${qa_ok}'
    packaging
    delivery
  end
end

For this process, the states are “nil” (not running), “production”, “qa1” and/or “qa2”, “delivery”, and “packaging”.

(note that there is no “terminated” state, since, out of the box, ruote doesn’t do ‘process archiving’ for you).

Asking all the participants about the workitems could get expensive, especially if they’re remote participants. It’s easier to ask the engine :

p engine.process(wfid)
  # =>
  # == Ruote::ProcessStatus ==
  #    expressions : 3
  #      0!69176db85a0651e7a8d8a16426bd93df!20110107-betesuguto : define {}
  #      0_0!be7ac163b2c6ba47d6d4b24bbb83fd8c!20110107-betesuguto : cursor {}
  #      0_0_0!3b19bdf68a953597969bac507229bcf1!20110107-betesuguto : participant {"ref"=>"production"}
  #    schedules : 0
  #    stored workitems : 1
  #    variables :     {}
  #    all_variables : {"0!69176db85a0651e7a8d8a16426bd93df!20110107-betesuguto"=>{}}
  #    errors : 0

p engine.process(wfid).position
  # =>
  # [
  #   [ "0_0_0!3b19bdf68a953597969bac507229bcf1!20110107-betesuguto",
  #     "production",
  #     {} ] ]

(full gist at https://gist.github.com/769362)

What if the “state”, business-wise, has a different granularity than the simple one we derive from the participants ? Something more like a “stage”.

The “tag” attribute could help :

Ruote.process_definition do
  cursor do
    sequence :tag => 'production-stage' do
      production
      concurrence do
        qa1
        qa2
      end
    end
    rewind :unless => '${qa_ok}'
    sequence :tag => 'delivery-stage' do
      packaging
      delivery
    end
  end
end

Our process [instance] can be in “production-stage” or “delivery-stage” (or nowhere).

We can then ask for the tags of a process status :

p engine.process(wfid).tags
  # =>
  #  { "production-stage" => {
  #      "engine_id" => "engine",
  #      "wfid" => "20110107-hopakeze",
  #      "subid" => "0f62bbb3a0a19411aaef1524ebde657c",
  #      "expid" => "0_0_0" } }

(full gist at https://gist.github.com/769371)

Tags were originally meant to be used in conjunction with the undo/cancel and the redo expressions.

They later were used within cursors for jumps (along with participant names) and when “piloting” a cursor from outside.

Asking the engine about the ‘process state’ / ‘stage’ is fine, but what if a [remote] participant wants to know about the stage is in ?

Let’s add a ‘qa_stage’ tag.

Ruote.define do
  cursor do
    sequence :tag => 'production-stage' do
      production
      concurrence :tag => 'qa_stage' do
        qa1
        qa2
      end
    end
    rewind :unless => '${qa_ok}'
    sequence :tag => 'delivery-stage' do
      packaging
      delivery
    end
  end
end

The upcoming ruote 2.1.12 adds a #tags method to its workitems (a shortcut for workitem.fields[‘__tags__’]).

With a participant implementation that looks like

class MyParticipant
  include Ruote::LocalParticipant

  def consume(workitem)
    p [ workitem.participant_name, :tags, workitem.tags ]
    workitem.fields['qa_ok'] = true
    reply_to_engine(workitem)
  end
end

a run of our process will output

["production", :tags, ["production-stage"]]
["qa1", :tags, ["production-stage", "qa_stage"]]
["qa2", :tags, ["production-stage", "qa_stage"]]
["packaging", :tags, ["delivery-stage"]]
["delivery", :tags, ["delivery-stage"]]

(full gist at https://gist.github.com/769406)

There is a difference between asking the engine for the tags of a process intance and the list of tags returned by workitem#tags. The former returns all the tags currently active for the process instance, while the latter returns the list of tags that were traversed to reach the participant expression that emitted the workitem.

Another novelty brought in by ruote 2.1.12 is the possibility to listen to tag events (entering and leaving tags).

Until now, the http://ruote.rubyforge.org/exp/listen.html expression could only listen to participants (workitems reaching and returning from participants). The upcoming ruote 2.1.12 lets us listen to process instances entering and leaving tags.

For example, we could have a monitoring process that listen to process instances entering the ‘final-stage’ tag. Each time it happens, the participant ‘supervisor’ receives a copy of the workitem :

Ruote.process_definition do
  listen :to => 'final-stage', :upon => 'entering' do
    participant 'supervisor'
  end
end

By setting the :wfid attribute of the listen expression to true, we can limit the listening to the process instance to which the listen expression belongs.

It can be useful to approximate the ‘milestone’ workflow pattern :

Ruote.define do

  concurrence :count => 1 do
    # will terminate as soon as 1 branch replies (and cancel the other)

    sequence do
      participant 'a'
      participant 'b', :tag => 'milestone'
      participant 'c'
    end

    listen :to => 'milestone', :upon => 'entering', :wfid => true do

      concurrence :count => 1 do
        # will terminate as soon as 1 branch replies (and cancel the other)

        listen :to => 'milestone', :upon => 'leaving', :wfid => true
          # as soon as the tag 'milestone' is left, this listen will
          # exit (reply to the parent concurrence) and d will get cancelled

        participant 'd'
      end
    end
  end
end

The participant ‘d’ will only receive a workitem when the milestone is reached. As soon as the milestone is left, the workitem of participant ‘d’ is cancelled (removed from him).

Note how this implementation relies on “concurrence :count => 1”, a concurrence that exits as soon as 1 of its branches replies (and cancels the remaining branches).

In conclusion, tags have multiple uses, designating process state / stages, letting undo / redo segments of processes. The next ruote (2.1.12) adds tag information in workitems and lets the listen expression observe tag events (entering or leaving a tag).

 

* web : http://ruote.rubyforge.org
* source : http://github.com/jmettraux
* mailing list : http://groups.google.com/group/openwferu-users
* irc : freenode.net #ruote

 

Written by John Mettraux

January 7, 2011 at 1:24 pm

Posted in bpm, openwferu, ruby, ruote, workflow

ruote and switch

A programming language usually has some kind of super “if”, a switch statement.

Ruote is nothing more that an interpreter, a very patient one, one that can get stopped and restarted. (it also runs multiple process instances concurrently). Since it interprets some kind of high level business process gibberish, a switch statement is nice to have.

This post explores switch statement implementations for ruote, it starts with solutions for ruote 2.1.x and then shows solutions that use two new expressions in the upcoming ruote 2.1.12.

 

before ruote 2.1.12

For ruote [2.1], I wasn’t sure if a switch statement was explicitely needed. With a workitem field or a process variable, it’s easy to switch to a given subprocess.

Suppose we have an item pickup process. The system is onboard, the drivers leaves the depot knowing his immediate pickup point, but not the next, as the system asks after each pickup what to do (via the ‘get_next_task’ subprocess).

Ruote.process_definition do

  define 'coast_pickup' do
    # ...
  end
  define 'mountain_pickup' do
    # ...
  end
  define 'get_back_to_depot' do
    # ...
  end

  sequence do # body of the process

    cursor do
      subprocess 'get_next_task'
      subprocess '${next_task}'
      rewind :unless => '${next_task} == get_back_to_depot'
        # cursor rewinds unless next task is getting back to depot
    end
  end
end

All is well, our switch occurs at “subprocess ‘${next_task}'”. Note that if we wished to, we could use ref instead of subprocess if we wanted to point to participants and/or subprocesses.

But wait… Those subprocesses are defined for the whole process, they are not limited to the switch.

We can go a bit further and isolate the switch and its cases into its own subprocess.

Ruote.process_definition do

  define 'perform_next_task' do

    define 'coast_pickup' do
      # ...
    end
    define 'mountain_pickup' do
      # ...
    end
    define 'get_back_to_depot' do
      # ...
    end

    subprocess '${next_task}'
  end

  sequence do # body of the process

    cursor do
      subprocess 'check_tasks'
      subprocess 'perform_next_task'
      rewind :unless => '${next_task} == get_back_to_depot'
    end
  end
end

This is nice, we have a ‘perform_next_task’ subprocess wrapping the cases, the body of that subprocess calls the right case given the value of the “next_task” workitem field. If there is another subprocess named ‘mountain_pickup’ in the same process definition, the one in the switch subprocess will only shadow it within the case, in other words, the scope of the cases is limited to the switch subprocess.

But, you’ll say, with a real programming language not some toy process definition language, the switch and its cases are all wrapped neatly inline, they are not set apart.

Placing the switch and its cases in the main flow is OK, but it binds subprocesses… It can override subprocesses with the same name.

    cursor do
      subprocess 'check_tasks'
      sequence do
        define 'coast_pickup' do
          # ...
        end
        define 'mountain_pickup' do
          # ...
        end
        define 'get_back_to_depot' do
          # ...
        end
        subprocess '${next_task}'
      end
      rewind :unless => '${next_task} == get_back_to_depot'
    end

We need to look at the next version of ruote to solve that issue.

 

with ruote 2.1.12

For the upcoming ruote 2.1.12, I introduced a let expression :

    cursor do
      subprocess 'check_tasks'
      let do # let's have a new scope, just for our cases
        define 'coast_pickup' do
          # ...
        end
        define 'mountain_pickup' do
          # ...
        end
        define 'get_back_to_depot' do
          # ...
        end
        subprocess '${next_task}'
      end
      rewind :unless => '${next_task} == get_back_to_depot'
    end

The ‘let’ introduces a new scope, where our case subprocesses can get defined without overrides ones with the same name outside of the let block.

Now, how about something that really looks like a switch statement ?

    cursor do
      subprocess 'check_tasks'
      given '${next_task}' do
        of 'coast_pickup' do
          # ...
        end
        of 'mountain_pickup' do
          # ...
        end
        of 'get_back_to_depot' do
          # ...
        end
      end
      rewind :unless => '${next_task} == get_back_to_depot'
    end

With some Ruby/Perl -like magic, all the pickups could get covered by the same case :

    cursor do
      subprocess 'check_tasks'
      given '${next_task}' do
        of /^.+_pickup$/ do
          # ...
        end
        of 'get_back_to_depot' do
          # ...
        end
      end
      rewind :unless => '${next_task} == get_back_to_depot'
    end

The upcoming ruote 2.1.12 has this given expression. Look at its description, it not only covers “given an x of” scenarii but also “given that” ones.

 

conclusion

I still like the first version, it’s vanilla ruote “trigger the subprocess whose name is found in workitem field x”, it’s nice to have subprocesses that can be used as cases or called from other parts of the process definition.

Calling subprocesses doesn’t limit us to processes bound within the same process, processes bound at the engine level (engine variables) or external processes (given by their URI) are callable as well (see the subprocess expression doc for more information).

The given expression is interesting because it covers “given an x of” and “given that” scenarii (see the doc). It also has a “default” part which the first version doesn’t have.

Try to write processes that are concise and that read like english. And test them.

 

Written by John Mettraux

January 3, 2011 at 7:08 am

Posted in bpm, ruby, ruote, workflow

resource lifecycle tuple

I came across this passage :

Moving from design to implementation, we need to think about the protocol in a slightly different way. In a resource-oriented distributed application, an application protocol can be thought of as a function of one or more resource life cycles and the business rules that connect these resources. Because of its resource-centric nature, the Restbucks service does not host an application protocol state machine. In other words, there’s no workflow or business logic for the application protocol as such. Rather, the service governs the life cycles of the orders and payments participating in the application protocol. Any workflows in the service implementation relate to resource life cycles, not the application protocol life cycle. While we’ve been explicit in modeling the business process as an application protocol state machine, we’ve been diligent in implementing it wholly in terms of resource state machines.

REST in practice, page 131, (emphasis mine).

 

I had to link that passage to this blog post :

The bottom line is that a business entity lifecycle represents some business logic that is “business process independent”. This is probably news to many of you and some of you might say that a BEL is a “long running process” but it is not. It is long running. It is a process in the sense of an operating system point of view (nearly), but it is not and will never be a “business process”. The business process is represented by the activities (human or automated) that advance the lifecycle of one or more business entities.

Lifecycle example, from Carnets de Bord

 

Written by John Mettraux

October 27, 2010 at 5:24 am

ruote 2.1.11 released

ruote is an open source workflow engine implemented in Ruby.

It takes as input process definitions and interprets them. It routes work among participants according to the flow described in those process definitions. It can orchestrate ping pong matches as well.

The main motivation behind this release is Torsten being tired of advising people to use ruote edge (thanks Bundler) for their ruote-kit installs, ruote was more than ripe for a new release. This version includes the result of the feedback of numerous people, as the changelog can attest.

I’ll address here two aspects, one in ruote itself, and one on its periphery.

participant in a workflow

Up until now, participants were registered one by one (order of registration matters).

engine.register_participant 'reception', Acme::ReceptionParticipant
engine.register_participant 'support', Acme::SupportParticipant
engine.register_participant '.+', Ruote::StorageParticipant

Torsten came up with an idiomatic solution in ruote-kit, that was soon promoted to ruote itself :

engine.register do
  reception Acme::ReceptionParticipant
  support Acme::SupportParticipant
  catchall
end

All is well in a vanilla world were participants are all known at system startup. People were asking about adding participants on-the-fly. Ruote has always been able to do that :

engine.register_participant 'reception2', Acme::ReceptionParticipant, :site => 'two'

but our reception2 gets registered after the catchall participant, and thus might never receive any workitems. A solution would be to place the participant as second to last :

engine.register_participant(
  'reception2', Acme::ReceptionParticipant, :site => 'two', :position => -2)

But blindly inserting participants isn’t good. Ruote 2.1.11 has a new method for setting the participant list in a single batch :

engine.participant_list = [
  [ '^reception$', 'Acme::ReceptionParticipant', {} ],
  [ '^reception2$', 'Acme::ReceptionParticipant', { 'site' => 'two' } ],
  [ '^support$', 'Acme::SupportParticipant', {} ],
  [ '^.+$', 'Ruote::StorageParticipant', {} ]
]

Very rough, but full control.

ruote in kit

ruote-kit was kind of left behind at version 2.1.8. Torsten and I resumed its development and brought it to 2.1.11.

Ruote-kit is a web administration console for ruote. Kenneth Kalmer’s genius idea was to make it a rack middleware (component), so that it works standalone or fits nicely in a rails application, under /_ruote/

list of processesview on a process errorlist of participantsview on a workitem

Ruote-kit is more than a web administration console, it’s also an HTTP/JSON based interface to ruote, for example : http://gist.github.com/611044. The entry point of the interface is /_ruote/ all the resources are linked to from this “root”. The links are annotated with “rel” attributes that indicate to clients what stands behind the link.

There is more to it, we’re working on conveying the capabilities of both interfaces HTML / JSON from the HTML (the one that is easily navigated by a human and his browser).

storages got updates

ruote-dm, ruote-redis, ruote-couch have been upgraded to 2.1.11.

Note that there is a piece of documentation on how to implement storages that is in the works as it was requested by people who want to have a MongoDB backend.

next steps

Still have to release an updated ruote-amqp, and bring ruote-beanstalk to 2.1.11. Then it will be time to think about 2.1.12. As said, there is still work to do on ruote-kit, and since it’s used by many ruote people, it deserves full attention.

Many thanks to Eric Platon, Nathan Stults, Asier, Hartog de Mik, Brett Anthoine, Eric Smith, Kaspar Schiess, David Greaves, Rich Meyers, Kenneth Kalmer, Don French and David Goodlad (and many others) for their contributions !

source : http://github.com/jmettraux/ruote
mailing list : http://groups.google.com/group/openwferu-users
doc : http://ruote.rubyforge.org
irc : #ruote on freenode.net

 

Written by John Mettraux

October 5, 2010 at 6:07 am

amqp, ruote, pong

Hartog de Mik just wrote an investigation into ruote and AMQP. In this blog post, he makes the long jump from zero to ruote + RabbitMQ participants via the vaguely charted ruote-amqp territory.

But right before clicking away to Hartog’s blog post, please consider this pure ruote pong solution :

require 'rubygems'
require 'ruote'

pdef = Ruote.process_definition do
  repeat do
    ping # mister ping, please shoot first
    pong
  end
end

class Opponent
  include Ruote::LocalParticipant

  def initialize (options)
    @options = options
  end

  def consume (workitem)
    puts @options['sound']
    reply_to_engine(workitem)
  end
end

engine = Ruote::Engine.new(Ruote::Worker.new(Ruote::HashStorage.new))

engine.register_participant :ping, Opponent, 'sound' => 'ping'
engine.register_participant :pong, Opponent, 'sound' => 'pong'

wfid = engine.launch(pdef)

sleep 5 # five seconds of ping pong fun

engine.cancel_process(wfid) # game over

 

http://ruote.rubyforge.org (home)
http://groups.google.com/group/openwferu-users (mailing list)
IRC #ruote on freenode.net

 

Written by John Mettraux

September 27, 2010 at 2:24 pm

Posted in ruby, ruote, workflow

quaderno

My task as a programmer seems mostly to be throwing forms at knowledge workers. I fool myself in thinking that a workflow engine is the panacea when deciding when and what form to present the user.

Workflows are supposed to be adaptable, the forms at the emerged tips even more so. “Can you add a text field here ?”.

In workflow technology, there seems to be this grail where the business user is knowledgeable enough to modify a process definition, and knowledgeable enough to modify a form. Defining a process has a strong taste of programming, and form building hints at hierarchical data structures. It’s a struggle to get right either of them.

I came up with a javascript library named quaderno that takes as input a form template written in a DSL (minilanguage is perhaps a better appellation) and a data dictionary, renders the form, and then, when the user is done, produces the updated data dictionary.

This template :

tabs
  tab "general"
    text_input name
    text_input age
    select gender [male, female]
  tab "appointments"
    box appointments.*^
      date .date
      text_input .goal
      text_area .comment

Will produce a form that looks like :

The second tab is gathering data in an “appointments” array. The suffix .*^ indicates that elements of the array, can be added and removed (+ and – merged into a *) and reordered at will (^).

This is a sample output, once the user has entered his values and the produce() function of quaderno somehow got called :

{ "name": "Toto",
  "age": "33",
  "gender": "male",
  "appointments": [
    { "date":"2010/9/18", 
      "goal":"general checkup", 
      "comment": "(and tell him to quit smoking)" } ] }

This example is visible at http://ruote.rubyforge.org/quaderno/viewer.html?sample=3, there is also another sample at http://ruote.rubyforge.org/quaderno/viewer.html.

I use quaderno for highly adaptable forms, whether they are part of a workflow or not. Perhaps I should not say “adaptable”, but “I trust other people enough to let them modify the forms”.

(at first, I had wanted to have three modes, form use, form view and form edit, but the last mode was a bit hard to maintain and the result was awful to use and to look at. I had come up with something better in terms of edition with the initial version, but I couldn’t get to show the link between data and fields. This 3rd version of quaderno is a back to the basics version, with this tiny DSL that moves away from WYSIWYG and exposes the data/form structure)

Quaderno is a MIT licensed javascript library. It’s not a solution, it’s a tool.

http://ruote.rubyforge.org/quaderno/viewer.html
http://github.com/jmettraux/quaderno

 

Written by John Mettraux

September 22, 2010 at 5:28 am

Posted in dsl, javascript, ruote, workflow