Ruedas y Cervezas, first instance

“ruote” is the Italian for “wheels”. It’s the name of our ruby workflow engine. The name was proposed by Tomaso Tosolini, it stuck and we liked how people pronounced it “route” which is a good fit for a tool that routes work among participants.

If you translate “ruote” to Spanish you get “ruedas”. Add a thirsty community and you get “ruedas y cervezas”.

This event was organized and hosted by abstra.cc, a madrilenian company doing wonders with Java, JRuby, Scala and their blend of agile magic.

The meeting was divided in three parts. First, abstra.cc presented their work with ruote. They’ve bound it into their “Blue Mountain ERP” to drive the necessary business processes. Workitems are communicated to participants over XMPP. The platform is Java and the languages are JRuby and Scala. As you might have guessed, they run ruote on JRuby.

Then, Iván Martínez, student in the Universidad Politécnica de Madrid presented his work on a graphical process editor for ruote. It’s based on Flex and looks very promising. The abstra.cc guys encouraged him to share his code on github.com, I hope he will follow this suggestion.

For the part when people really get thirsty, I cooked up a bunch of slides about ruote 2.0. They were presented by Nando Sola. I have embedded them here. The first half of them is about why ruote 2.0, while the latter half is about the enhancements to the ruote process definition language.

I felt it was necessary to communicate a bit about ruote 2.0, especially since, in Madrid, people have tremendously contributed to ruote 0.9 (many thanks guys !). Ruote 2.0 is a small jump from 0.9, literally a jump, not a step, explanations are required.

Then beer ensued. A ruedas y cervezas (rc2) is planned for next month, I wish I could attend. So if you’re in Madrid and you’re interested in ruote or, less narrowly, in Ruby / JRuby / Scala / Java, contact Nando Sola and join the abstra.cc and UPM guys for the rc2.

 

 

rufus-lua 1.1.0

Just released rufus-lua 1.1.0. The original post I wrote about rufus-lua is named ruby to lua.

This release strongly benefited from Scott Persinger’s work on Laminate, a tool for safe user-authored templates for Vodpod.

Scott needed strong support for callbacks from Lua to Ruby and also ways to pass values back and forth. Many thanks to Scott for the great collaboration.

Those of you interested in Lua and web applications should have a look at Norman Clarke’s lua-haml and at Daniele Alessandri’s mercury (sinatra-like web framework for Lua).

 

rufus-tokyo 1.0.1

rufus-tokyo is a ruby-ffi based library for accessing Tokyo Cabinet and Tokyo Tyrant databases. It also feature a Rufus::Edo side where the native ruby/c extensions provided by the TC/TT author are used (for speed) instead of ruby-ffi.

rufus-tokyo contains ffi bindings for Tokyo Dystopia as well, thanks to Jeremy Hinegardner.

This is the changelog for this 1.0.1 release :

== rufus-tokyo - 1.0.1    released 2009/09/18

- todo : add #putcat to Cabinet / Tyrant (Edo and Tokyo)
- todo : implemented search/union/intersection/difference for tables
- todo : added #putdup and #get4 to Cabinet / Tyrant (Edo and Tokyo)
- todo : better dylib 'detection' (Pietro Ferrari)
- todo : aliased lget to mget (thanks Runa)
- todo : proper Abort exception (Kenneth Kalmer)

 

putcat is used to append data to already stored value. getdup / get4 is only relevant with b+ trees structures. They allow for multiple values stored under one key. This get4/getdup returns all the values stored under one key.

Since Tokyo Cabinet 1.4.29 (Tyrant 1.1.30), cabinet tables and tyrant tables have this new metasearch feature. This is what is meant by “search/union/intersection/difference” in the changelog.

Hirabayashi-san is providing us with a way to combine queries. From rufus-tokyo, it looks like :

require 'rubygems'
require 'rufus/tokyo' # sudo gem install rufus-tokyo

t = Rufus::Tokyo::Table.new('table.tct')

t['pk0'] = { 'name' => 'alfred', 'age' => '22', 'hobby' => 'fishing' }
t['pk1'] = { 'name' => 'bob', 'age' => '18', 'hobby' => 'hunting' }
t['pk2'] = { 'name' => 'charly', 'age' => '45', 'hobby' => 'fishing' }
t['pk3'] = { 'name' => 'doug', 'age' => '77', 'hobby' => 'fencing' }
t['pk4'] = { 'name' => 'ephrem', 'age' => '32', 'hobby' => 'swimming' }

rs = t.union(
  t.prepare_query { |q| q.add 'hobby', :equals, 'fishing' },
  t.prepare_query { |q| q.add 'age', :numgt, '20' }
)

rs.each { |r| p r }
  # ==>
  # ["pk2", {"name"=>"charly", "hobby"=>"fishing", "age"=>"45"}]
  # ["pk3", {"name"=>"doug", "hobby"=>"fencing", "age"=>"77"}]
  # ["pk4", {"name"=>"ephrem", "hobby"=>"swimming", "age"=>"32"}]
  # ["pk0", {"name"=>"alfred", "hobby"=>"fishing", "age"=>"22"}]

t.close

This example returns the persons in the table who do fishing for a hobby OR whose age is greater than 20. The other methods understood by the table are intersection and difference.

For more information, see the rufus-tokyo source on github and the rufus-tokyo rdoc.

Many thanks to all the persons who contributed to rufus-tokyo.

 

On the general Tokyo Cabinet / Tyrant front, Flinn Mueller, the author of ruby-tokyotyrant started a couple months ago a mailing list about Tokyo Cabinet / Tyrant. Lots of knowledge about the Tokyo products is exchanged there.

There is also the release of tyrantmanager by Jeremy Hinegardner (the author of the Dystopia bindings in rufus-tokyo). It’s a neat command line tool for managing Tokyo Tyrant instances, individually or in batch.

 

Kenneth’s ruote talk

Kenneth Kalmer uploaded his “ruote in twenty minutes” video.

I liked it a lot, an excellent mix of ruote idealism and south-african pragmatism.

It was 6 months ago though. We’re now working on ruote2.0, ruote-amqp, ruote-dm and the last addition is ruote-activerecord.

 

edo cabinet – rubykaigi2009

Here are my slides for the “edo cabinet” talk at the RubyKaigi2009.

I presented about Ruby-FFI, Tokyo Cabinet|Tyrant and then rufus-tokyo and Rufus::Edo.

These are just slides, here is the english transcript of the actual talk.

 

Many thanks to the RubyKaigi team for organizing this great event ! Special kudos to Leonard Chin for the on-the-fly translations, throughout the 3 days.

You’ll find videos of most of the talks on ustream.tv (thanks to the KaigiFreaks) and most of the slides are available from Slideshare.

 

rufus-tokyo 1.0.0

Just released rufus-tokyo 1.0.0

This is mostly a “cleanup” release (spec reorg, to_s enforcement on key and values, and dropped backward compatibility with older TC/TT releases). Hence the 1.0.0.

There are new features though :

* Matthew King helped me add process method for table queries.

* Jeremy Hinegardner did an initial contribution on the front FFI / Tokyo Dystopia.

* Adam Keys contributed the ext method for the table structures (sorry, I had forgotten).

* full text search operators for table queries (:ftsphrase, :ftsec, :ftsor: ftsand) and the accompanying :token, :qgram and :opt inverted index types. I haven’t played too much with them. Feedback is welcome.

For those of you interested in extending Tokyo Tyrant via Lua functions, I warmly recommend Ilya Grigork’s post about it.

Many thanks to all the people who helped.

There will be a next release soon, as I’m busy trying to catch up with Hirabayashi-san.

rufus-tokyo : http://github.com/jmettraux/rufus-tokyo/

(picture graciously from @michelledasilva)

 

state machine != workflow engine

This post is intended for Ruby developers. The idea for it came after numerous discussions with fellow programmers about state machines and workflow engines.

What motivates me for posting is the publication of the 3rd issue of the excellent Rails Magazine. It contains an article named “Workflow solutions with AASM”.

At first a word of warning, I wrote these lines with no intention of minimizing the potential of state machines vs workflow engines. I’m listing workflow engine and state machine implementations for Ruby at the end of this post.

There are a number of open questions in this post, I don’t intend to answer them now or later. There are so many different use cases in the wild.

spark

The article “Workflow solutions with AASM” starts with :

There are two main forms of workflows: sequential and
state-machine. Sequential workflows are predictable. They
utilize the rules and conditions we provide at the beginning
to progress a process forward. The workflow is in control of
the process.

A state-machine workflow is the reverse. It is driven by
external events to its completion. We define the states and
required transitions between those states. The workflow sits
and waits for an external event to occur before transitioning
to one of the defined states. The decision making process hap-
pens externally outside of the workflow – there is a structure
to be followed still like a sequential workflow but control is
passed to its external environment.

I’m clearly in the “sequential workflow” faction. But I refute the ’sequential’ label. Most of the state/transitions set out there can be qualified as “sequential”. Any workflow engine, in order to present any interest to its users, has to implement a certain number of [control flow] workflow patterns. The number 1 control-flow pattern is named Sequence, granted. But if you take a look at the pattern that immediately follows, it’s Parallel Split. I can’t adhere to the “sequential workflow” appellation.

Note that most workflow engines (you can call then “sequential workflow” engines) strive to implement a large set of those control-flow patterns. This is usually done via their ‘process definition’ language. The goal is to let users express their scenarii / pattern in a not too verbose way. We can argue that state machines may implement any of the control-flow patterns, my guess is ‘yes’, but what’s the cost in code ?

The difference between “sequential workflows” and “state machines” seems to lie not in the predictability of the former, but rather in how the flow between activities or state is weaved, especially how/where it’s concentrated.

Are “sequential workflows” and “state machines” excluding one another ?

The article states “sequential workflow are predictable. They utilize the rules and conditions we provide at the beginning”. It’s the same for state machines. The vast majority of [Ruby] state machine implementations rely on behaviour specified at implementation time (or right before the application’s [re]start).

The second paragraph I quoted up here says that “decision making process happens externally outside of the workflow (…) control is passed to its external environment”. I argue that a “sequential workflow” engine should do the same. A workflow [engine] can’t take a decision by itself, this task is usually handled by a human or a dedicated algorithm (a piece of code, a rule engine, a coin tossing engine wired to internet, whatever).

case

Wait, we’ve been using this ‘workflow’ term quite liberally until now, what is it about ? Why do state machines seem to be the perfect vessel for it, at least in the eyes of the hard-working developer ?

Let’s look at how state machines are sold, at their use cases. Here is a classical example from a [Rails-oriented] state machine library :

  class Document < ActiveRecord::Base
    include StateFu

    def update_rss
      puts "new feed!"
      # ... do something here
    end

    machine( :status ) do
      state :draft do
        event :publish, :to => :published
      end

      state :published do
        on_entry :update_rss
        requires :author  # a database column
      end

      event :delete, :from => :ALL, :to => :deleted do
        execute :destroy
      end
    end
  end

It attaches behaviour (state and transitions) to a resource. If you look at the example of any Ruby state machine library out there, you will see this pattern : a set of states and transitions attached to a resource in the system (a Model).

Let’s move to an adjacent example, it’s also about a document, but it’s in the “sequential workflow” faction (ruote). (The ‘cursor’ expression in the process definition allows for the flow to be rewound or skipped…) :

Ruote.process_definition :name => 'doc review', revision => '0.2' do

  cursor do

    participant '${f:author}', :step => 'finalize document'
    participant '${f:review_team}', :step => 'review document'

    rewind :unless => '${f:approved}'

    concurrence do
      participant '${f:publisher}', :step => 'publish document'
      participant '${f:author}', :step => 'publication notification'
    end
  end
end

There is a document yes, but it could be a folder of documents. The process definition is a separate thing, meant to be turned into multiple process instances.

Workflow engines are mostly process definitions interpreters, graduating as “operating systems for business processes” (think ‘ps’ and ‘kill dash 9′ but with business processes / process instances).

As said, most of the Ruby state machine libraries out there are all about binding behaviour to resources / models. What about moving to the workflow engine camp and instead of binding to a system artefact (document, order, invoice, book, customer, …) why not attach state and transitions to a virtual artefact, a so-called “process instance” ? The state machine would move out of its shell and turn into a full workflow engine. Or is that so ?

cases

Back to the 90% of the cases : the state machine attached to a model. What if there is more than 1 “stateable” model ? Easy. But what if transition in model A provokes a transition in model B ? The “real workflow” now lies at the intersection of two models.

The workflow engine will expose a process definition while, with the state machines, we’ll have to extract it from two or more models. Another advantage of workflow engines is that they have process definition versioning. Most of them run process instances from process definition X at version 2 happily with process instances from version 4, concurrently.

On the other hand, when a process instance goes ballistic it might take some time to repair it (align it with the reality of the business process). It might be easier with state machines, a simple SQL update usually, but what if there are multiple inter-related behaviours ? Pain on both sides.

Whatever the tool, you’ll have to carefully avoid locking yourself into your private hell of a system.

nuances

Kenneth Kalmer is using a mix of state_machine and ruote in their ISP in a box product. The state machine library is in charge of the state of its various models while the workflow engine does the orchestration. “Processes that drives the provision of services”

The two techniques may co-exist. Tinier states / transitions sets, more concise process definitions. Models managing their states, process instances coordinating the states of multiple models.

It’s OK to remove the workflow engine and replace it with a “super” state machine, but then you’re entering in the real workflow territory and the mindset is a bit different and at that level, you’ll be exposed to a flavour of change that directly involves your customer / end-user. Nowhere to hide.

machines

I’ve been linking profusely to my ruote workflow engine. The only other workflow engine I’ve spotted in the wild is rbpm, but its development stopped a long time ago.

Here is a non-exhaustive list of Ruby state machine implementations, in no specific order :

• state-fu
• state_machine
• acts_as_state_machine
• ryan-allen’s workflow
• geekq’s workflow (fork of the previous)
• remix-machine
• alter-ego
• stateful
• stonepath covered by an article in the 2009/10 pragprog magazine
• automatic-acts-as-state-machine
• micromachine
• ruby-state-machine
• ActiveRecord::StateMachine (announced)
• dm-is-state_machine
• …

There’s a plethora of state machines versus very few (one or two) workflow engines. This could be interpreted in multiple ways.

State machines follow a well defined pattern, while workflow engines have to follow a set of patterns, and workflow engines strive to be “operating systems for business processes”. State machines are more humble.

So Rails Magazine #3 is out. Great read.

 

rufus-tokyo 0.1.13

With the initial releases of rufus-tokyo, I happily cut corners and went with C strings (ending with NUL).

This isn’t optimal, often you need to store binary data as the value, the resulting ’string’ contains NUL characters and values get truncated at restitution. I was working around that with Base64 encoding. Fine, the perf cost isn’t too heavy. But then, binary data was working fine on the Rufus::Edo side (wrapping the original tokyo{cabinet|tyrant}-ruby), why should Rufus::Tokyo be an exception ?

rufus-tokyo 0.1.13 which just got released fixes that issue.

Thanks to Ben Mabey for pointing out the issue.

Kamal Fariz Mahyuddin was kind enough to fork rufus-tokyo and add the #putkeep(k, v) method which only put if there was no previous value associated with the key. The collaboration with Kamal also prompted me to provide #add_int and #add_double for int/double counters in cabinet/tyrant.

http://github.com/jmettraux/rufus-tokyo

 

rufus-scheduler 2.0, with em flavour

This blog is turning into a release gallery.

Anyway. Just released rufus-scheduler 2.0. It’s a complete rewrite. The only new “feature” is an EventMachine mode.

The plain scheduler uses a thread waking up every 0.330s (by default) to check for jobs to trigger, while the em-based scheduler uses an EventMachine timer.

Other flavour for the core loop are possible, why not something 1.9 fiber based in the short term ?

This release 2.0 is mostly backward compatible. Only some corner cases and advanced usages (block arity) have changed (they have been simplified). Having a look at the readme should help.

require 'rubygems'
require 'rufus/scheduler' # sudo gem install rufus-scheduler

s = Rufus::Scheduler.start_new

will still work, it will return an instance of Rufus::Scheduler::PlainScheduler, but if an EM reactor is running, it will stick to it and return a Rufus::Scheduler::EmScheduler instance.

source : http://github.com/jmettraux/rufus-scheduler/
rdoc : http://rufus.rubyforge.org/rufus-scheduler/
ml : http://groups.google.com/group/rufus-ruby

Many thanks to Klaas Jan Wierenga for contributing nice stress tests (for the 1.0 and the 2.0 branches), thanks as well to Kenneth Kalmer for its daemon-kit + rufus-scheduler effort.

 

rufus-decision 1.1, ruby decision tables

rufus-decision is a small ruby gem for ‘running decision tables’. Decision tables are useful for mapping conditions to actions.

This example decision table considers two conditions : ‘topic’ and ‘region’ (I tend to call them ‘inputs’). Certain combinations of condition yield one or more output value.

In the example, when the topic is about finance and the region is Europe, the team member Donald gets selected.

This is a vanilla decision table, as soon as a ‘match’ is found, the run is over, an output value has been found.

Note that the arrangement of rules, their order, does matter. Ernest who is in charge of finance for the rest of the world, comes after America and Europe, Charly and Donald respectively. If Ernest were placed before them, all the input sets with the topic set to ‘finance’ would go to him, ignoring his two colleagues.

Likewise, our ‘handle all the rest’ Zach has been placed last, collecting all the input sets that didn’t match.

It’s easy to reproduce the table with some code :

if topic == 'sports'
  if region == 'europe'
    team_member = 'Alice'
  else
    team_member = 'Bob'
  end
elsif topic == 'finance'
  # ...
elsif topic == 'politics'
  # ...
else
  team_member = 'Zach'
end

Turning business logic into code is a common task for software developers, but what about simply ‘running’ a decision table directly ?

Decision tables are easy to edit with spreadsheet software and that’s a happy coincidence since many domain experts are Excel jockeys (see lay programmer).

Excel isn’t the only spreadsheet software, see for example our decision table in Google Spreadsheet.

At this point, I have to say I’m sorry, I won’t suggest any mean to run decision tables directly inside of Excel, I will bring back the flow to the Ruby environment, with some code triggering our business logic.

require 'rubygems'
require 'rufus/decision' # sudo gem install rufus-decision

table = %{
  in:topic,in:region,out:team_member
  sports,europe,Alice
  sports,,Bob
  finance,america,Charly
  finance,europe,Donald
  finance,,Ernest
  politics,asia,Fujio
  politics,america,Gilbert
  politics,,Henry
  ,,Zach
}

table = Rufus::Decision::Table.new(table)

p table.run('topic' => 'politics', 'region' => 'america')
  # => {"region"=>"america", "topic"=>"politics", "team_member"=>"Gilbert"}

p table.run('topic' => 'sports', 'region' => 'antarctic')
  # => {"region"=>"antarctic", "topic"=>"sports", "team_member"=>"Bob"}

p table.run('topic' => 'culture', 'region' => 'america')
  # => {"region"=>"america", "topic"=>"culture", "team_member"=>"Zach"}

rufus-decision understands decision tables with rules expressed per column as well as per row :

table = %{
  in:topic,sports,sports,finance,finance,finance,politics,politics,politics,
  in:region,europe,,america,europe,,asia,america,,
  out:team_member,Alice,Bob,Charly,Donald,Ernest,Fujio,Gilbert,Henry,Zach
}

The parameter to Rufus::Decision::Table.new() can be a CSV string, an array of arrays, the path to a CSV file or the URI of a CSV file. With a Google spreadsheet [published] table, our example could be cut to :

require 'rubygems'
require 'rufus/decision' # sudo gem install rufus-decision

table = Rufus::Decision::Table.new(
  'http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv')

# ...

So far so good, our domain expert may publish decision tables on the company information’s system and we can use them directly. But what to do when the number of conditions and rules get large ?

A programmer is expected to test his code, he’s even expected to deliver code with tests. A sane programmer can’t live without tests. Lay programmers should welcome testing as well.

rufus-decision comes with a “rufus_decide” command line tool for running decision tables in batch.

Given this input file (topics_in.csv), where the first row lists the keys and the next rows hold the values :

topic,region
sports,america
politics,europe
culture,
politics,africa

the decision table can be exercised with :

rufus_decide \
-t "http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv" \
-i topics_in.csv

which will yield :

region,team_member,topic
america,Bob,sports
europe,Henry,politics
,Zach,culture
africa,Henry,politics

‘rufus_decide’ ran the decision table for each row in the input data and generated an output table with one row for each input row.

This ‘ideal’ output can saved in a ‘goal.csv’ file and used to ‘test’ the decision table :

rufus_decide \
-t "http://spreadsheets.google.com/pub?key=rs4Z_1gjtyfgXupxXz_nAYw&output=csv" \
-i topics_in.csv \
-g goal.csv

Running rufus_decide with a goal will emit an output similar to the one of classical unit test tools.

This was an introduction to rufus-decision, whose version 1.1.0 was just released.

The new release contains some kind of web-based environment for testing rules (see thumbnail on left), but well, I’m not quite convinced it’s useful.

source : http://github.com/jmettraux/rufus-decision/
rdoc : http://rufus.rubyforge.org/rufus-decision/