volute

Vos luttes partent en fumée
Vers des flûtes enchantées Et de cruelles espérances
Me lancent Des dagues et des lances En toute innocence

J’cloue des clous sur des nuages Un marteau au fond du garage
J’cloue des clous sur des nuages Sans échafaudage

volutes Alain Bashung (funny auto-translation)

 

This started out in a notebook. I wanted to do something about ‘state’ and ‘lifecyle’, something like a state machine for multiple objects or a rule system for families of resources.

It ended up as something that feels like the subset of an aspect oriented framework. Subset because in its most vanilla usage it only cares about calls to set methods, and there is no “before” (at first glance).

This Ruby gem is named volute. Here is an example of its usage :

require 'volute'

class Package
  include Volute

  attr_accessor :location
  attr_accessor :delivered
end

volute Package do
  # filters non Package instance out

  volute :delivered => true do
    # applies when delivered switches to true
    object.emit_invoice
  end

  volute :location do
    # filters out changes that are not for the :location attribute

    volute 'SFO' => :any, 'FRA' => :any do
      # SFO and FRA are our international hubs, for any package transiting from there,
      # initial 'international' tracking
      TrackingSystem.track_international(object)
    end
  end
end

The “include Volute” reworks the attr_accessor, and triggers the evaluation of the volutes on each set.

It’s not always necessary to include Volute. Here is a example of [business] rules derived from Ruleby (volute is very dumb compared to Ruleby, the example doesn’t do justice to that project).

Here is the diagnosis example with volute :

require 'volute'

class Patient

  attr_reader :name
  attr_accessor :fever
  attr_accessor :rash
  attr_accessor :spots
  attr_accessor :sore_throat
  attr_accessor :innoculated

  attr_accessor :diagnosis

  def initialize(name)
    @name = name
    @symptoms = {}
    @innoculated = false
  end

  def diagnose!
    Volute.apply(self) # trigger evalution of volutes
    return @diagnosis
  end
end

volute Patient do

  volute :fever => :high, :spots => :true, :innoculated => true do
    object.diagnosis = 'measles'
    over # prevent further evals
  end
  volute :spots => true do
    object.diagnosis = 'allergy (spots but no measles)'
  end
  volute :rash => true do
    object.diagnosis = 'allergy (rash)'
  end
  volute :sore_throat => true, :fever => :mild do
    object.diagnosis = 'flu'
  end
  volute :sore_throat => true, :fever => :high do
    object.diagnosis = 'flu'
  end
end

pat = Patient.new('alice')
pat.rash = true

puts "#{pat.name} : diagnosed with #{pat.diagnose!}"

pat = Patient.new('bob')
pat.sore_throat = true
pat.fever = :high

puts "#{pat.name} : diagnosed with #{pat.diagnose!}"

 

The readme for volute is quite extensive. There are a few examples included.

You’ve already seen the diagnosis one, it aimed at using volute for some mini rule engine (nothing fancy at all).

There is an example about a simple (kph = km/h) equation where each time an attribute is changed, other attributes get adjusted.

The light example simply keeps tracks of the last time an attribute got modified.

There is a book in a bookshop state machine example with a variation, to explore state volutes and transition volutes.

Note that state machines built with volute are incomplete, they know nothing about further transitions (could I even call them state machines ? Rather a ‘state transition system’).

Traffic could be fun : it tracks the state of two “lines” and adjust the colour of their lights according to the number of cars waiting. It’s more in line with the “multiple objects” goal I had initially (still here somehow, but examples with 1 class are smaller).

Not sure if it’s worth moving logic out of objects, but it is a fun experiment.

 

http://github.com/jmettraux/volute

 

workflow, bpm, selected resources

I have started to gather posts and blogs I think are worth a read in the workflow, BPM, Adaptive Case Management, rules, etc fields.

The list is at http://ruote.rubyforge.org/resources.html

I hope to list there resources that are sincere and passionate, and that challenge my point of view on workflow engines and enterprisey programming in general. I will try to avoid things that are press release like, that include words like “leading” or “fortune 500”, that are too closed, not leaving place for a slice of doubt.

 

ruote and decision tables

ruote 2.1.7 is out and it felt like it was time to update the CsvParticipant that could be found in ruote 0.9.

Ruote has been relying on rufus-decision for its decision table needs and ruote 0.9 was integrating a CsvParticipant directly. For ruote 2.1, the DecisionParticipant is left in the rufus-decision gem.

It’s not the first time I write something about decision tables. But I noticed that I hadn’t really written anything about how to mix ruote processes and decision tables.

Let’s consider the decision table in the adjacent figure.

Given two input values “type of visit” and “participant physician ?” the level of “reimbursement” is decided.

The CSV version of this decision table is available online.

Clearly this decision has to take place after the customer filled his reimbursement claim and it got reviewed by a clerk in the organization.

Up until now, applying the reimbursement rules was part of the tasks of the clerk (with probably some downstream checks), and our goal is to remove some of that burden.

Our process definition would look like :

Ruote.process_definition :name => 'reimbursement', :revision => '1.0' do
  cursor do
    customer :task => 'fill form'
    clerk :task => 'control form'
    rewind :if => '${f:missing_info}'
    determine_reimbursement_level
    finance :task => 'reimburse customer'
    customer :task => 'reimbursement notification'
  end
end

There are four participants involved in that process definition. Customer, Clerk and Finance our implemented with classical inbox/worklist participants : workitems for them get placed in their workqueue. (With ruote 2.1 I tend to use StorageParticipant for that or I emit the workitem via a ruote-amqp participant towards the target worklist or task manager).

“rewind” isn’t a participant, it’s a ruote expression working with the cursor expression.

That leaves us with the participant “determine_reimbursement_level” to implement.

The straightforward Ruby implementation of the participant would leverage a BlockParticipant and look like :

engine.register_participant 'determine_reimbursement_level' do |workitem|
  workitem.fields['reimbursement'] =
    case workitem.fields['type of visit']
      when 'doctor office'
        workitem.fields['participating physician ?'] == 'yes' ? '90%' : '50%'
      when 'hospital visit'
        workitem.fields['participating physician ?'] == 'yes' ? '0%' : '80%'
      when 'lab visit'
        workitem.fields['participating physician ?'] == 'yes' ? '0%' : '70%'
      else
        '0%'
    end
end

Given the values in the fields ‘type of visit’ and ‘participating physician ?’ this participant sets the value of the field ‘reimbursement’.

All is well, but you may have noticed that the decision table is rather simplistic. And how does it cope with change ? What will it look like when the rules get hairy ? Will the Ruby developer have to intervene for each change ?

(it might not be a bad thing to rely on the Ruby developer, they tend to test carefully their creations)

Those rules are produced by ‘business users’ and one of their favourite tools is the spreadsheet. Rufus-decision can read CSV output of decision tables. The decision participant would look like :

require 'rufus/decision/participant'
  # gem install rufus-decision

engine.register_participant(
  'determine_reimbursement_level',
  Rufus::Decision::Participant,
  :table => {
in:type of visit,in:participating physician ?,out:reimbursement
doctor office,yes,90%
doctor office,no,50%
hospital visit,yes,0%
hospital visit,no,80%
lab visit,yes,0%
lab visit,no,70%
  })

Well, OK, that’s nice, but… We still have to restart the application at each change to the decision table or at least re-register the participant with the updated table. Tedious.

If our business people publish the decision table as CSV somewhere over the intranet (yes, I know, it’s an old-fashioned word), the decision participant can be simplified to :

require 'rufus/decision/participant'
  # gem install rufus-decision

engine.register_participant(
  'determine_reimbursement_level',
  Rufus::Decision::Participant,
  :table => 'http://somewhere.example.com/decision_tables/reimbursement.csv')

Each time a decision will have to be taken, the table will be read. This lets business users modify the rules at will.

The whole example is packaged as reimbursement.rb. It runs the process from the command line and grabs the decision table online. It shouldn’t be too difficult to take inspiration from it and integrate decision tables in web applications (like ruote-kit or barley).

There are many open possibilities when combining a workflow engine and a decision mechanism. One immediate advantage is that they can evolve with different rhythms. They can also be used in isolation : the clerk in our process could talk the claim over the phone with the customer and run blank decisions to help the customer with incomplete forms.

Of course, you can use rufus-decision without ruote, it’s a plain ruby gem.

 

http://github.com/jmettraux/rufus-decision – the source code of rufus-decision
http://github.com/jmettraux/ruote – the source code of ruote
http://ruote.rubyforge.org – ruote’s documentation
http://groups.google.com/group/openwferu-users – the mailing list for ruote and rufus-decision

 

state machine != workflow engine

update (2010) : this resource lifecycle post features two quotes that are enlightening when thinking about state and workflow.

update (2011) : a discussion between two engineers about state machines and workflow (guest post at Engine Yard)

This post is intended for Ruby developers. The idea for it came after numerous discussions with fellow programmers about state machines and workflow engines.

What motivates me for posting is the publication of the 3rd issue of the excellent Rails Magazine. It contains an article named “Workflow solutions with AASM”.

At first a word of warning, I wrote these lines with no intention of minimizing the potential of state machines vs workflow engines. I’m listing workflow engine and state machine implementations for Ruby at the end of this post.

There are a number of open questions in this post, I don’t intend to answer them now or later. There are so many different use cases in the wild.

spark

The article “Workflow solutions with AASM” starts with :

There are two main forms of workflows: sequential and
state-machine. Sequential workflows are predictable. They
utilize the rules and conditions we provide at the beginning
to progress a process forward. The workflow is in control of
the process.

A state-machine workflow is the reverse. It is driven by
external events to its completion. We define the states and
required transitions between those states. The workflow sits
and waits for an external event to occur before transitioning
to one of the defined states. The decision making process hap-
pens externally outside of the workflow – there is a structure
to be followed still like a sequential workflow but control is
passed to its external environment.

I’m clearly in the “sequential workflow” faction. But I refute the ‘sequential’ label. Most of the state/transitions set out there can be qualified as “sequential”. Any workflow engine, in order to present any interest to its users, has to implement a certain number of [control flow] workflow patterns. The number 1 control-flow pattern is named Sequence, granted. But if you take a look at the pattern that immediately follows, it’s Parallel Split. I can’t adhere to the “sequential workflow” appellation.

Note that most workflow engines (you can call then “sequential workflow” engines) strive to implement a large set of those control-flow patterns. This is usually done via their ‘process definition’ language. The goal is to let users express their scenarii / pattern in a not too verbose way. We can argue that state machines may implement any of the control-flow patterns, my guess is ‘yes’, but what’s the cost in code ?

The difference between “sequential workflows” and “state machines” seems to lie not in the predictability of the former, but rather in how the flow between activities or state is weaved, especially how/where it’s concentrated.

Are “sequential workflows” and “state machines” excluding one another ?

The article states “sequential workflow are predictable. They utilize the rules and conditions we provide at the beginning”. It’s the same for state machines. The vast majority of [Ruby] state machine implementations rely on behaviour specified at implementation time (or right before the application’s [re]start).

The second paragraph I quoted up here says that “decision making process happens externally outside of the workflow (…) control is passed to its external environment”. I argue that a “sequential workflow” engine should do the same. A workflow [engine] can’t take a decision by itself, this task is usually handled by a human or a dedicated algorithm (a piece of code, a rule engine, a coin tossing engine wired to internet, whatever).

case

Wait, we’ve been using this ‘workflow’ term quite liberally until now, what is it about ? Why do state machines seem to be the perfect vessel for it, at least in the eyes of the hard-working developer ?

Let’s look at how state machines are sold, at their use cases. Here is a classical example from a [Rails-oriented] state machine library :

  class Document < ActiveRecord::Base
    include StateFu

    def update_rss
      puts "new feed!"
      # ... do something here
    end

    machine( :status ) do
      state :draft do
        event :publish, :to => :published
      end

      state :published do
        on_entry :update_rss
        requires :author  # a database column
      end

      event :delete, :from => :ALL, :to => :deleted do
        execute :destroy
      end
    end
  end

It attaches behaviour (state and transitions) to a resource. If you look at the example of any Ruby state machine library out there, you will see this pattern : a set of states and transitions attached to a resource in the system (a Model).

Let’s move to an adjacent example, it’s also about a document, but it’s in the “sequential workflow” faction (ruote). (The ‘cursor’ expression in the process definition allows for the flow to be rewound or skipped…) :

Ruote.process_definition :name => 'doc review', revision => '0.2' do

  cursor do

    participant '${f:author}', :step => 'finalize document'
    participant '${f:review_team}', :step => 'review document'

    rewind :unless => '${f:approved}'

    concurrence do
      participant '${f:publisher}', :step => 'publish document'
      participant '${f:author}', :step => 'publication notification'
    end
  end
end

There is a document yes, but it could be a folder of documents. The process definition is a separate thing, meant to be turned into multiple process instances.

Workflow engines are mostly process definitions interpreters, graduating as “operating systems for business processes” (think ‘ps’ and ‘kill dash 9’ but with business processes / process instances).

As said, most of the Ruby state machine libraries out there are all about binding behaviour to resources / models. What about moving to the workflow engine camp and instead of binding to a system artefact (document, order, invoice, book, customer, …) why not attach state and transitions to a virtual artefact, a so-called “process instance” ? The state machine would move out of its shell and turn into a full workflow engine. Or is that so ?

cases

Back to the 90% of the cases : the state machine attached to a model. What if there is more than 1 “stateable” model ? Easy. But what if transition in model A provokes a transition in model B ? The “real workflow” now lies at the intersection of two models.

The workflow engine will expose a process definition while, with the state machines, we’ll have to extract it from two or more models. Another advantage of workflow engines is that they have process definition versioning. Most of them run process instances from process definition X at version 2 happily with process instances from version 4, concurrently.

On the other hand, when a process instance goes ballistic it might take some time to repair it (align it with the reality of the business process). It might be easier with state machines, a simple SQL update usually, but what if there are multiple inter-related behaviours ? Pain on both sides.

Whatever the tool, you’ll have to carefully avoid locking yourself into your private hell of a system.

nuances

Kenneth Kalmer is using a mix of state_machine and ruote in their ISP in a box product. The state machine library is in charge of the state of its various models while the workflow engine does the orchestration. “Processes that drives the provision of services”

The two techniques may co-exist. Tinier states / transitions sets, more concise process definitions. Models managing their states, process instances coordinating the states of multiple models.

It’s OK to remove the workflow engine and replace it with a “super” state machine, but then you’re entering in the real workflow territory and the mindset is a bit different and at that level, you’ll be exposed to a flavour of change that directly involves your customer / end-user. Nowhere to hide.

machines

I’ve been linking profusely to my ruote workflow engine. The only other workflow engine I’ve spotted in the wild is rbpm, but its development stopped a long time ago.

Here is a non-exhaustive list of Ruby state machine implementations, in no specific order :

• state-fu
• state_machine
• acts_as_state_machine
• ryan-allen’s workflow
• geekq’s workflow (fork of the previous)
• alter-ego
• stateful
• stonepath covered by an article in the 2009/10 pragprog magazine
• automatic-acts-as-state-machine
• micromachine
• ruby-state-machine
• dm-is-state_machine
• newflow
• state_pattern
• stateflow
• ActiveRecord::StateMachine (announced)
• transitions (what was supposed to become ActiveRecord::StateMachine)
• statemachine
• flojo
• ssm
• state_attr
• state_manager
• stately
• statesman
• finite_machine
• …

There’s a plethora of state machines versus very few (one or two) workflow engines. This could be interpreted in multiple ways.

State machines follow a well defined pattern, while workflow engines have to follow a set of patterns, and workflow engines strive to be “operating systems for business processes”. State machines are more humble.

So Rails Magazine #3 is out. Great read.

rufus-decision 1.1, ruby decision tables

rufus-decision is a small ruby gem for ‘running decision tables’. Decision tables are useful for mapping conditions to actions.

This example decision table considers two conditions : ‘topic’ and ‘region’ (I tend to call them ‘inputs’). Certain combinations of condition yield one or more output value.

In the example, when the topic is about finance and the region is Europe, the team member Donald gets selected.

This is a vanilla decision table, as soon as a ‘match’ is found, the run is over, an output value has been found.

Note that the arrangement of rules, their order, does matter. Ernest who is in charge of finance for the rest of the world, comes after America and Europe, Charly and Donald respectively. If Ernest were placed before them, all the input sets with the topic set to ‘finance’ would go to him, ignoring his two colleagues.

Likewise, our ‘handle all the rest’ Zach has been placed last, collecting all the input sets that didn’t match.

It’s easy to reproduce the table with some code :

if topic == 'sports'
  if region == 'europe'
    team_member = 'Alice'
  else
    team_member = 'Bob'
  end
elsif topic == 'finance'
  # ...
elsif topic == 'politics'
  # ...
else
  team_member = 'Zach'
end

Turning business logic into code is a common task for software developers, but what about simply ‘running’ a decision table directly ?

Decision tables are easy to edit with spreadsheet software and that’s a happy coincidence since many domain experts are Excel jockeys (see lay programmer).

Excel isn’t the only spreadsheet software, see for example our decision table in Google Spreadsheet.

At this point, I have to say I’m sorry, I won’t suggest any mean to run decision tables directly inside of Excel, I will bring back the flow to the Ruby environment, with some code triggering our business logic.

require 'rubygems'
require 'rufus/decision' # sudo gem install rufus-decision

table = %{
  in:topic,in:region,out:team_member
  sports,europe,Alice
  sports,,Bob
  finance,america,Charly
  finance,europe,Donald
  finance,,Ernest
  politics,asia,Fujio
  politics,america,Gilbert
  politics,,Henry
  ,,Zach
}

table = Rufus::Decision::Table.new(table)

p table.run('topic' => 'politics', 'region' => 'america')
  # => {"region"=>"america", "topic"=>"politics", "team_member"=>"Gilbert"}

p table.run('topic' => 'sports', 'region' => 'antarctic')
  # => {"region"=>"antarctic", "topic"=>"sports", "team_member"=>"Bob"}

p table.run('topic' => 'culture', 'region' => 'america')
  # => {"region"=>"america", "topic"=>"culture", "team_member"=>"Zach"}

rufus-decision understands decision tables with rules expressed per column as well as per row :

table = %{
  in:topic,sports,sports,finance,finance,finance,politics,politics,politics,
  in:region,europe,,america,europe,,asia,america,,
  out:team_member,Alice,Bob,Charly,Donald,Ernest,Fujio,Gilbert,Henry,Zach
}

The parameter to Rufus::Decision::Table.new() can be a CSV string, an array of arrays, the path to a CSV file or the URI of a CSV file. With a Google spreadsheet [published] table, our example could be cut to :

require 'rubygems'
require 'rufus/decision' # sudo gem install rufus-decision

table = Rufus::Decision::Table.new(
  'http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv')

# ...

So far so good, our domain expert may publish decision tables on the company information’s system and we can use them directly. But what to do when the number of conditions and rules get large ?

A programmer is expected to test his code, he’s even expected to deliver code with tests. A sane programmer can’t live without tests. Lay programmers should welcome testing as well.

rufus-decision comes with a “rufus_decide” command line tool for running decision tables in batch.

Given this input file (topics_in.csv), where the first row lists the keys and the next rows hold the values :

topic,region
sports,america
politics,europe
culture,
politics,africa

the decision table can be exercised with :

rufus_decide \
-t "http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv" \
-i topics_in.csv

which will yield :

region,team_member,topic
america,Bob,sports
europe,Henry,politics
,Zach,culture
africa,Henry,politics

‘rufus_decide’ ran the decision table for each row in the input data and generated an output table with one row for each input row.

This ‘ideal’ output can saved in a ‘goal.csv’ file and used to ‘test’ the decision table :

rufus_decide \
-t "http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv" \
-i topics_in.csv \
-g goal.csv

Running rufus_decide with a goal will emit an output similar to the one of classical unit test tools.

This was an introduction to rufus-decision, whose version 1.1.0 was just released.

The new release contains some kind of web-based environment for testing rules (see thumbnail on left), but well, I’m not quite convinced it’s useful.

source : http://github.com/jmettraux/rufus-decision/
rdoc : http://rufus.rubyforge.org/rufus-decision/

 

ruby decision table quickstart

Update : there is a new blog post covering this Ruby decision table subject, with updated information.

The information in this current blog post is a bit out of date.

---

At first, install the necessary OpenWFEru gems :

sudo gem install openwferu openwferu-extras

Here is the quickstart :

(code here).

It should output something like :

weather : raining, month : december
=> take umbrella ? yes

weather : sunny, month : december
=> take umbrella ? no

weather : cloudy, month : may
=> take umbrella ? yes

If you take a closer look at the code, you’ll see that it taps directly the CSV view of a Google Spreadsheet. Quite handy.

More documentation *soon* on this page about Decision Tables.

Update : Adapted the head of the post, sudo gem install openwferu openwferu-extras will just work.

tb : one click deploy

Phil Haack wrote an extremely interesting blog post about, well, empowering business users maybe.

Deploying a business process belongs to a business process, it should, even if nowadays, now is too late.

ruby business rules

Pat Cappelaere took over the Rools project from Bob Stanley.

Rule and Workflow engine are related things. I sometimes think workflow engines are General Staff officers, taking care of the details in the long term, positioning the troops, whereas rule engines take the big decisions, Generals for short. It’s not a good parallel.

In business processes, decisions are usually taken by human participants, some decisions can be taken by rule engines (there are simple decisions left to workflow engines, generally routing decisions).
Some ‘automated decision taking’ can be useful outside the execution of a formal business process.

Is it a reduction to consider a rule engine as a decision taking artefact ?

To cut this short, workflow engine + rule engine is a powerful combination.

ruby decision tables

update (2008/01/28) : I’ve packaged this ruby decision table system into its own “rufus-decision” gem.

---

It’s part of OpenWFEru 0.9.4 that will be released this week.

I’ve already blogged about decision tables. Now the implementation is available in Ruby as well.

Disclaimer : this decision table implementation, whether in Java or Ruby is not meant to replace a full rules system with features like forward and backward chaining, RETE implementations and the like. But it covers a lot of cases. Be pragmatic.

For example, in our workflow definition, there’s a point where we’ve got to decide if we should take our umbrella when going to work (please no comments).
In pseudo-code, take would look like :

    if weather == "rainy"
        take_umbrella = "yes"
    elsif weather == "cloudy"
        take_umbrella = "maybe"
    else
        take_umbrella = "no"
    end

As we don’t want to clutter our process definition with such cruft (or we simply don’t want to hardcode that into our source), we fire up our spreadsheet and write down the [business] rule.

That makes for a nice CSV file :

in:weather, in:month, out:take_umbrella?
,,
raining,,yes
sunny,,no
cloudy,june,yes
cloudy,may,yes
cloudy,,no

Leveraging OpenWFEru CSV tables, we can write such Ruby code :

require 'openwfe/util/csvtable'


table = OpenWFE::CsvTable.new("path/to/our/file.csv")


h = { "weather" => "cloudy" }
table.transform(h)

puts h["table_umbrella?"]
    #
    # will yield "no"

h = { "weather" => "cloudy", "month" => "june" }
table.transform(h)

puts h["table_umbrella?"]
    #
    # will yield "yes"

# ...

That feature will be available with OpenWFEru 0.9.4 (or immediately by checking out OpenWFEru). You can already have a look at the unit tests for it.

The documentation for CSV tables as [decision] participants to business processes will be available soon.

pharma wfe

I spent my whole day tweaking the decision tables system for a pharma customer. The decision tables are used there as some kind of ‘expert system’, advising human participants on what to do. The human has the last word for now.

Something like :

<sequence>
    <participant ref="prepare-decision" />
    <participant ref="human-participant" />
</sequence>

The ‘prepare-decision’ applies the rules in the tables to present a workitem pre-filled for human approbation/modification.

The knowledge of the human participant has been captured within those decision tables that he augments iteratively. Could finally the decision table participant supplant the human participant ? At least it will help the human replacements / successors of the human participant. The table itself (mainly its Excel version) will as well be a valuable item in their company knowledge landscape.

I was discussing with Boris about those decision tables. He felt a bit concerned that they were something outside the process definitions. Why not embedding them (some of them) within their process definition ? It’s certainly an idea to investigate, but as participants they make lots of sense : they can be reused among many process definitions.

Pharma WFE ? No “c1al1s” spam intended.