This post is intended for Ruby developers. The idea for it came after numerous discussions with fellow programmers about state machines and workflow engines.

What motivates me for posting is the publication of the 3rd issue of the excellent Rails Magazine. It contains an article named “Workflow solutions with AASM”.

At first a word of warning, I wrote these lines with no intention of minimizing the potential of state machines vs workflow engines. I’m listing workflow engine and state machine implementations for Ruby at the end of this post.

There are a number of open questions in this post, I don’t intend to answer them now or later. There are so many different use cases in the wild.

spark

The article “Workflow solutions with AASM” starts with :

There are two main forms of workflows: sequential and
state-machine. Sequential workflows are predictable. They
utilize the rules and conditions we provide at the beginning
to progress a process forward. The workflow is in control of
the process.

A state-machine workflow is the reverse. It is driven by
external events to its completion. We define the states and
required transitions between those states. The workflow sits
and waits for an external event to occur before transitioning
to one of the defined states. The decision making process hap-
pens externally outside of the workflow – there is a structure
to be followed still like a sequential workflow but control is
passed to its external environment.

I’m clearly in the “sequential workflow” faction. But I refute the ’sequential’ label. Most of the state/transitions set out there can be qualified as “sequential”. Any workflow engine, in order to present any interest to its users, has to implement a certain number of [control flow] workflow patterns. The number 1 control-flow pattern is named Sequence, granted. But if you take a look at the pattern that immediately follows, it’s Parallel Split. I can’t adhere to the “sequential workflow” appellation.

Note that most workflow engines (you can call then “sequential workflow” engines) strive to implement a large set of those control-flow patterns. This is usually done via their ‘process definition’ language. The goal is to let users express their scenarii / pattern in a not too verbose way. We can argue that state machines may implement any of the control-flow patterns, my guess is ‘yes’, but what’s the cost in code ?

The difference between “sequential workflows” and “state machines” seems to lie not in the predictability of the former, but rather in how the flow between activities or state is weaved, especially how/where it’s concentrated.

Are “sequential workflows” and “state machines” excluding one another ?

The article states “sequential workflow are predictable. They utilize the rules and conditions we provide at the beginning”. It’s the same for state machines. The vast majority of [Ruby] state machine implementations rely on behaviour specified at implementation time (or right before the application’s [re]start).

The second paragraph I quoted up here says that “decision making process happens externally outside of the workflow (…) control is passed to its external environment”. I argue that a “sequential workflow” engine should do the same. A workflow [engine] can’t take a decision by itself, this task is usually handled by a human or a dedicated algorithm (a piece of code, a rule engine, a coin tossing engine wired to internet, whatever).

case

Wait, we’ve been using this ‘workflow’ term quite liberally until now, what is it about ? Why do state machines seem to be the perfect vessel for it, at least in the eyes of the hard-working developer ?

Let’s look at how state machines are sold, at their use cases. Here is a classical example from a [Rails-oriented] state machine library :

  class Document < ActiveRecord::Base
    include StateFu

    def update_rss
      puts "new feed!"
      # ... do something here
    end

    machine( :status ) do
      state :draft do
        event :publish, :to => :published
      end

      state :published do
        on_entry :update_rss
        requires :author  # a database column
      end

      event :delete, :from => :ALL, :to => :deleted do
        execute :destroy
      end
    end
  end

It attaches behaviour (state and transitions) to a resource. If you look at the example of any Ruby state machine library out there, you will see this pattern : a set of states and transitions attached to a resource in the system (a Model).

Let’s move to an adjacent example, it’s also about a document, but it’s in the “sequential workflow” faction (ruote). (The ‘cursor’ expression in the process definition allows for the flow to be rewound or skipped…) :

Ruote.process_definition :name => 'doc review', revision => '0.2' do

  cursor do

    participant '${f:author}', :step => 'finalize document'
    participant '${f:review_team}', :step => 'review document'

    rewind :unless => '${f:approved}'

    concurrence do
      participant '${f:publisher}', :step => 'publish document'
      participant '${f:author}', :step => 'publication notification'
    end
  end
end

There is a document yes, but it could be a folder of documents. The process definition is a separate thing, meant to be turned into multiple process instances.

Workflow engines are mostly process definitions interpreters, graduating as “operating systems for business processes” (think ‘ps’ and ‘kill dash 9′ but with business processes / process instances).

As said, most of the Ruby state machine libraries out there are all about binding behaviour to resources / models. What about moving to the workflow engine camp and instead of binding to a system artefact (document, order, invoice, book, customer, …) why not attach state and transitions to a virtual artefact, a so-called “process instance” ? The state machine would move out of its shell and turn into a full workflow engine. Or is that so ?

cases

Back to the 90% of the cases : the state machine attached to a model. What if there is more than 1 “stateable” model ? Easy. But what if transition in model A provokes a transition in model B ? The “real workflow” now lies at the intersection of two models.

The workflow engine will expose a process definition while, with the state machines, we’ll have to extract it from two or more models. Another advantage of workflow engines is that they have process definition versioning. Most of them run process instances from process definition X at version 2 happily with process instances from version 4, concurrently.

On the other hand, when a process instance goes ballistic it might take some time to repair it (align it with the reality of the business process). It might be easier with state machines, a simple SQL update usually, but what if there are multiple inter-related behaviours ? Pain on both sides.

Whatever the tool, you’ll have to carefully avoid locking yourself into your private hell of a system.

nuances

Kenneth Kalmer is using a mix of state_machine and ruote in their ISP in a box product. The state machine library is in charge of the state of its various models while the workflow engine does the orchestration. “Processes that drives the provision of services”

The two techniques may co-exist. Tinier states / transitions sets, more concise process definitions. Models managing their states, process instances coordinating the states of multiple models.

It’s OK to remove the workflow engine and replace it with a “super” state machine, but then you’re entering in the real workflow territory and the mindset is a bit different and at that level, you’ll be exposed to a flavour of change that directly involves your customer / end-user. Nowhere to hide.

machines

I’ve been linking profusely to my ruote workflow engine. The only other workflow engine I’ve spotted in the wild is rbpm, but its development stopped a long time ago.

Here is a non-exhaustive list of Ruby state machine implementations, in no specific order :

• state-fu
• state_machine
• acts_as_state_machine
• ryan-allen’s workflow
• geekq’s workflow (fork of the previous)
• remix-state
• alter-ego
• stateful
• stone-path (in extraction)
• automatic-acts-as-state-machine
• …

There’s a plethora of state machines versus very few (one or two) workflow engines. This could be interpreted in multiple ways.

State machines follow a well defined pattern, while workflow engines have to follow a set of patterns, and workflow engines strive to be “operating systems for business processes”. State machines are more humble.

So Rails Magazine #3 is out. Great read.

With the initial releases of rufus-tokyo, I happily cut corners and went with C strings (ending with NUL).

This isn’t optimal, often you need to store binary data as the value, the resulting ’string’ contains NUL characters and values get truncated at restitution. I was working around that with Base64 encoding. Fine, the perf cost isn’t too heavy. But then, binary data was working fine on the Rufus::Edo side (wrapping the original tokyo{cabinet|tyrant}-ruby), why should Rufus::Tokyo be an exception ?

rufus-tokyo 0.1.13 which just got released fixes that issue.

Thanks to Ben Mabey for pointing out the issue.

Kamal Fariz Mahyuddin was kind enough to fork rufus-tokyo and add the #putkeep(k, v) method which only put if there was no previous value associated with the key. The collaboration with Kamal also prompted me to provide #add_int and #add_double for int/double counters in cabinet/tyrant.

http://github.com/jmettraux/rufus-tokyo

This blog is turning into a release gallery.

Anyway. Just released rufus-scheduler 2.0. It’s a complete rewrite. The only new “feature” is an EventMachine mode.

The plain scheduler uses a thread waking up every 0.330s (by default) to check for jobs to trigger, while the em-based scheduler uses an EventMachine timer.

Other flavour for the core loop are possible, why not something 1.9 fiber based in the short term ?

This release 2.0 is mostly backward compatible. Only some corner cases and advanced usages (block arity) have changed (they have been simplified). Having a look at the readme should help.

require 'rubygems'
require 'rufus/scheduler' # sudo gem install rufus-scheduler

s = Rufus::Scheduler.start_new

will still work, it will return an instance of Rufus::Scheduler::PlainScheduler, but if an EM reactor is running, it will stick to it and return a Rufus::Scheduler::EmScheduler instance.

source : http://github.com/jmettraux/rufus-scheduler/
rdoc : http://rufus.rubyforge.org/rufus-scheduler/
ml : http://groups.google.com/group/rufus-ruby

Many thanks to Klaas Jan Wierenga for contributing nice stress tests (for the 1.0 and the 2.0 branches), thanks as well to Kenneth Kalmer for its daemon-kit + rufus-scheduler effort.

rufus-decision is a small ruby gem for ‘running decision tables’. Decision tables are useful for mapping conditions to actions.

This example decision table considers two conditions : ‘topic’ and ‘region’ (I tend to call them ‘inputs’). Certain combinations of condition yield one or more output value.

In the example, when the topic is about finance and the region is Europe, the team member Donald gets selected.

This is a vanilla decision table, as soon as a ‘match’ is found, the run is over, an output value has been found.

Note that the arrangement of rules, their order, does matter. Ernest who is in charge of finance for the rest of the world, comes after America and Europe, Charly and Donald respectively. If Ernest were placed before them, all the input sets with the topic set to ‘finance’ would go to him, ignoring his two colleagues.

Likewise, our ‘handle all the rest’ Zach has been placed last, collecting all the input sets that didn’t match.

It’s easy to reproduce the table with some code :

if topic == 'sports'
  if region == 'europe'
    team_member = 'Alice'
  else
    team_member = 'Bob'
  end
elsif topic == 'finance'
  # ...
elsif topic == 'politics'
  # ...
else
  team_member = 'Zach'
end

Turning business logic into code is a common task for software developers, but what about simply ‘running’ a decision table directly ?

Decision tables are easy to edit with spreadsheet software and that’s a happy coincidence since many domain experts are Excel jockeys (see lay programmer).

Excel isn’t the only spreadsheet software, see for example our decision table in Google Spreadsheet.

At this point, I have to say I’m sorry, I won’t suggest any mean to run decision tables directly inside of Excel, I will bring back the flow to the Ruby environment, with some code triggering our business logic.

require 'rubygems'
require 'rufus/decision' # sudo gem install rufus-decision

table = %{
  in:topic,in:region,out:team_member
  sports,europe,Alice
  sports,,Bob
  finance,america,Charly
  finance,europe,Donald
  finance,,Ernest
  politics,asia,Fujio
  politics,america,Gilbert
  politics,,Henry
  ,,Zach
}

table = Rufus::Decision::Table.new(table)

p table.run('topic' => 'politics', 'region' => 'america')
  # => {"region"=>"america", "topic"=>"politics", "team_member"=>"Gilbert"}

p table.run('topic' => 'sports', 'region' => 'antarctic')
  # => {"region"=>"antarctic", "topic"=>"sports", "team_member"=>"Bob"}

p table.run('topic' => 'culture', 'region' => 'america')
  # => {"region"=>"america", "topic"=>"culture", "team_member"=>"Zach"}

rufus-decision understands decision tables with rules expressed per column as well as per row :

table = %{
  in:topic,sports,sports,finance,finance,finance,politics,politics,politics,
  in:region,europe,,america,europe,,asia,america,,
  out:team_member,Alice,Bob,Charly,Donald,Ernest,Fujio,Gilbert,Henry,Zach
}

The parameter to Rufus::Decision::Table.new() can be a CSV string, an array of arrays, the path to a CSV file or the URI of a CSV file. With a Google spreadsheet [published] table, our example could be cut to :

require 'rubygems'
require 'rufus/decision' # sudo gem install rufus-decision

table = Rufus::Decision::Table.new(
  'http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv')

# ...

So far so good, our domain expert may publish decision tables on the company information’s system and we can use them directly. But what to do when the number of conditions and rules get large ?

A programmer is expected to test his code, he’s even expected to deliver code with tests. A sane programmer can’t live without tests. Lay programmers should welcome testing as well.

rufus-decision comes with a “rufus_decide” command line tool for running decision tables in batch.

Given this input file (topics_in.csv), where the first row lists the keys and the next rows hold the values :

topic,region
sports,america
politics,europe
culture,
politics,africa

the decision table can be exercised with :

rufus_decide \
-t "http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv" \
-i topics_in.csv

which will yield :

region,team_member,topic
america,Bob,sports
europe,Henry,politics
,Zach,culture
africa,Henry,politics

‘rufus_decide’ ran the decision table for each row in the input data and generated an output table with one row for each input row.

This ‘ideal’ output can saved in a ‘goal.csv’ file and used to ‘test’ the decision table :

rufus_decide \
-t "http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv" \
-i topics_in.csv \
-g goal.csv

Running rufus_decide with a goal will emit an output similar to the one of classical unit test tools.

This was an introduction to rufus-decision, whose version 1.1.0 was just released.

The new release contains some kind of web-based environment for testing rules (see thumbnail on left), but well, I’m not quite convinced it’s useful.

source : http://github.com/jmettraux/rufus-decision/
rdoc : http://rufus.rubyforge.org/rufus-decision/

Just released version 0.1.12 of rufus-tokyo, a Ruby library for accessing Tokyo Cabinet and Tokyo Tyrant (via FFI or via the Ruby classes provided with the Tokyo products).

Tokyo Tyrant, once successfully compiled with –enable-lua, is open to lots of interesting usages.

In his last post on the Mixi dev blog, the author of Tokyo Cabinet / Tyrant is exposing his ideas about an application of map/reduce leveraging Tokyo Tyrant (post in Japanese, machine translation here). This demonstration relies on the embedded Lua runtime for the mapper/reducer implementation.

For this post, I’ll just show the vanilla “incr” function found in the Tokyo Tyrant documentation :

--
-- incr.lua
--
function incr (key, i)
  i = tonumber(i)
  if not i then
    return nil
  end
  local old = tonumber(_get(key))
  if old then
    i = old + i
  end
  if not _put(key, i) then
    return nil
  end
  return i
end

This Lua function increments the current value bound for a key by a given amount.

You start your Lua-enabled Tokyo Tyrant with something like :

  ttserver -ext incr.lua test.tch
 

and then hit it with code like :

# test.rb

require 'rubygems'
require 'rufus/tokyo/tyrant' # sudo gem install rufus-tokyo

t = Rufus::Tokyo::Tyrant.new('127.0.0.1', 1978)

5.times do
  puts t.ext(:incr, 'my-counter', 2).to_i
end

t.close

The “ext” method is available with the latest release of rufus-tokyo, for the Rufus::Tokyo::Tyrant and Rufus::Edo::NetTyrant classes.

I updated my documentation about Tokyo Cabinet / Tyrant install with some –enable-lua information.

This release of rufus-tokyo also adds transaction support for the Rufus::Tokyo::Cabinet class (requires Tokyo Cabinet 1.4.13).

This doesn’t deserve a gem. It’s just a plain file, so it’s packaged as a gist. It’s inspired by Rails’ rake rails:freeze:gems but I made it for my non-rails stuff.

jmettraux@sanma ~/tmp/test $ ruby frigo.rb dm-migrations
 . created dir vendor
 .. from http://gems.rubyforge.org/ got dm-migrations-0.9.10
 .. from http://gems.rubyforge.org/ got dm-core-0.9.10
 .. from http://gems.rubyforge.org/ got data_objects-0.9.11
 .. from http://gems.rubyforge.org/ got addressable-2.0.2
 .. from http://gems.rubyforge.org/ got rake-0.8.4
 .. from http://gems.rubyforge.org/ got rspec-1.2.2
 .. from http://gems.rubyforge.org/ got cucumber-0.2.2
 .. from http://gems.rubyforge.org/ got term-ansicolor-1.0.3
 .. from http://gems.rubyforge.org/ got treetop-1.2.5
 .. from http://gems.rubyforge.org/ got polyglot-0.2.5
 .. from http://gems.rubyforge.org/ got hoe-1.11.0
 .. from http://gems.rubyforge.org/ got rubyforge-1.0.3
 .. from http://gems.rubyforge.org/ got diff-lcs-1.1.2
 .. from http://gems.rubyforge.org/ got builder-2.1.2
 .. from http://gems.rubyforge.org/ got launchy-0.3.3
 .. from http://gems.rubyforge.org/ got configuration-0.0.5
 .. from http://gems.rubyforge.org/ got extlib-0.9.10

jmettraux@sanma ~/tmp/test $ ls vendor/
addressable-2.0.2       dm-core-0.9.10          polyglot-0.2.5
builder-2.1.2           dm-migrations-0.9.10    rake-0.8.4
configuration-0.0.5     extlib-0.9.10           rspec-1.2.2
cucumber-0.2.2          frozen.rb               rubyforge-1.0.3
data_objects-0.9.11     hoe-1.11.0              term-ansicolor-1.0.3
diff-lcs-1.1.2          launchy-0.3.3           treetop-1.2.5

OK, it downloaded dm-migrations and all its required gems under vendor/ but what is this vendor/frozen.rb thing ?

# require this file to get all the frozen/unpacked gems into the load path
#
# ( created via frigo.rb  http://gist.github.com/87639 )

here = File.expand_path(File.dirname(__FILE__))

Dir.entries(here).select { |p|
  p.match(/^[^\.]/) and File.directory?("#{here}/#{p}/lib")
}.each { |p|
  $:.unshift("#{here}/#{p}/lib")
}

“require ‘vendor/frozen.rb’” will add to the loadpath all the lib/ paths under the vendor/ dir. This frozen.rb file is added by the frigo.rb freezer.

frigo.rb is [awaiting obsolescence] at http://gist.github.com/87639

Just released rufus-lua 1.0

There are two main new features : lua garbage collection control (thanks Alain) and the possibility to bind Ruby blocks as Lua functions.

Rufus-lua is available via

sudo gem install rufus-lua

Just released rufus-tokyo 0.1.10. It supports the new setlimit method in Tokyo Cabinet 1.4.10 tables and Tokyo Tyrant 1.1.17 tables.

(I put up a piece of documentation on how to install TC and TT at http://openwferu.rubyforge.org/tokyo.html)

What’s the deal with this setlimit method ? Well, it’s an improvement over setmax which only accepted a count parameter.

require 'rubygems'
require 'rufus/tokyo'

t = Rufus::Tokyo::Table.new('table.tct')

t['pk0'] = { 'name' => 'alfred', 'age' => '22' }
t['pk1'] = { 'name' => 'bob', 'age' => '18' }
t['pk2'] = { 'name' => 'charly', 'age' => '45' }
t['pk3'] = { 'name' => 'doug', 'age' => '77' }
t['pk4'] = { 'name' => 'ephrem', 'age' => '32' }

p t.query { |q|
  q.order_by 'age'
  q.limit(2, 3) # 2 records max, skip 3 records
}
  # => [ {"name"=>"ephrem", :pk=>"pk4", "age"=>"32"},
  #      {"name"=>"charly", :pk=>"pk2", "age"=>"45"} ]

t.close

(Oops, I noticed the max/count parameter is not always respected, submitted a bug report… And not sure if this offset/skip is ‘pagination’ out of the box, it seems more like a ’skip’, literally)

This release of rufus-tokyo supports previous versions of Tokyo Cabinet and Tokyo Tyrant as well.

sudo gem install rufus-tokyo or http://github.com/jmettraux/rufus-tokyo/

Many thanks to Benjamin Yu for pointing out the issue.

I just released ruote 0.9.20. The original release text is on github, but I’m reproducing it here, with more details.

This release is not backward compatible. Changing the engine and restarting with a batch of persisted processes will not work. Start with new processes or migrate them.

Speaking of persistence, this release features revised and new persistence mechanisms. The most common one, the one directly bound to the file system has been revised to use Ruby marshalling instead of YAML with a net performance increase. Tokyo Cabinet and Tokyo Tyrant persistence mechanisms have been added as well, along with a DataMapper one.

Should the need to move from one persistence mecha to the other arise, a pooltool.ru has been included for easy migrations, back and forth.

On the front of expressions themselves, ’sleep’ and ‘wait’ have been merged, as was pointed out to me, “wait ‘2d’” sounds more businessy than “sleep ‘2d’”.

Each expression may now have a on_cancel and/or on_error attribute, pointing to a subprocess or participant called in case of error or cancel respectively. This was suggested by Raphael Simon and Kenneth Kalmer.

There is a new way to define process via Ruby :

  OpenWFE.process_definition :name => 'cfp' do
    sequence do
      prepare_call
      concurrence do
        vendor1
        vendor2
        vendor3
      end
      decide_which
    end
  end

Kenneth Kalmer came up with a JabberParticipant and a JabberListener and Torsten Schoenebaum implemented an ActiveResourceParticipant for modifying Rails resources from Ruote.

Many thanks to everybody who contributed code, feedback, ideas to Ruote !

Ruote’s source : http://github.com/jmettraux/ruote/

With the help from Alain I just released rufus-lua 0.1.0.

This ruby gem talks to Lua via Ruby FFI. One could say it’s “embedding Lua”, as Lua is described as “Lua is a powerful, fast, lightweight, embeddable scripting language.”

I feel tempted by Lua, there’s the speed, the austerity and that javascript feeling.

Rufus-lua is not the first bridge between Lua and Ruby, there are also :

* http://rubyluabridge.rubyforge.org/
* http://raa.ruby-lang.org/project/ruby-lua

They are worth a look, but I was a bit disappointed, they are not available via ’sudo gem install …’.

Rufus-lua is available via Rubygems, but since it’s a FFI thing, it needs to talk to the dynamic library packaging of Lua. Well, it’s not available by default, there’s some work involved (but not too much).

Something great about FFI is that it works for Ruby 1.8.6, Ruby 1.9.1 and JRuby (1.1.6) (and will work soon for Rubinius, its original platform).

Once liblua.dylib is on your system, along with rufus-lua (sudo gem install rufus-lua), you can play with code like :

require 'rubygems'
require 'rufus/lua'

s = Rufus::Lua::State.new

t = s.eval(%{
  return { message = { 'hello', 'from', 'Lua' },
    funk = function (x) return 2^x end }
})

p t['message'].to_a # => ["hello", "from", "lua"]
p t['funk'].call(8) # => 256.0

s.close

Well, it’s only a 0.1.0 release…

As rufus-lua is using FFI it’s slower at the “frontier”, here is a naive benchmark for computing a Fibonacci suite, with Fiber (Ruby 1.9) and Coroutines on the Lua side :

require 'benchmark'
require 'rubygems'
require 'rufus/lua'

RUBYFIBS = Fiber.new do
  n1 = n2 = 1
  loop do
    Fiber.yield n1
    n1, n2 = n2, n1+n2
  end
end

s = %{
  co = coroutine.create(function ()
    local n1 = 1; local n2 = 1
    while true do
      coroutine.yield(n1)
      n1, n2 = n2, n1+n2
    end
  end)
  return co
}

LUA = Rufus::Lua::State.new
LUAFIBS = LUA.eval(s)

N = 10_000
Benchmark.benchmark(' ' * 31 + Benchmark::Tms::CAPTION, 31) do |b|
  b.report('ruby') do
    N.times { RUBYFIBS.resume }
  end
  b.report('lua via ruby') do
    N.times { LUAFIBS.resume }
  end
  b.report('lua') do
    LUA.eval("for i = 0, #{N} do coroutine.resume(co) end")
  end
end

LUA.close

For the original. Running it on my board yields something along the lines of :

$ ruby19 test/bm_fiber.rb
                  user     system      total        real
ruby          0.050000   0.010000   0.060000 (  0.052032)
lua via ruby  0.180000   0.000000   0.180000 (  0.195411)
lua           0.010000   0.000000   0.010000 (  0.006394)

Well, nothing spectacular, that just states that it’s better to not cross the border too often : the “lua via ruby” bench has Ruby resuming the Lua coroutine 10_000 times and it’s costly.

rufus-lua,
on github : http://github.com/jmettraux/rufus-lua/
rdoc : http://rufus.rubyforge.org/rufus-lua/
the rest of rufus : http://rufus.rubyforge.org

lua : http://www.lua.org
ffi : http://kenai.com/projects/ruby-ffi/