about processes and engines

Archive for the ‘rufus’ Category

rufus-scheduler 2.0.12 released

rufus-scheduler is a thread-based scheduler written in Ruby. It lets you write code like:

require 'rufus-scheduler'

s = Rufus::Scheduler.start_new

s.every '10m' do
  puts 'open the window for the cat'
end '2012-01-01 12:00' do
  puts 'reminder: wife's birthday'

s.cron '0 22 * * 1-5' do
  puts 'activate the security system'

s.join # in case of stand-alone script...

The main addition brought by this release is the :mutex attribute when scheduling blocks of code. I was seeing people misusing :blocking => true to exclude block execution overlapping. It works but the scheduler is blocked as well, and crons might get skipped:

s.every '10m', :blocking => true do
  puts 'doing this...'
  sleep 60 * 60 # 1 hour
  puts 'done.'

# if the scheduler is in the blocking task above, crons will get skipped... '2012-01-01 12:00' do
  puts 'do that.'

My advice was to use mutexes instead:

$m =

s.every '10m' do
  $m.synchronize do
    puts 'doing this...'
    sleep 60 * 60 # 1 hour
    puts 'done.'

# if the scheduler is in the blocking task above, crons will get skipped... '2012-01-01 12:00' do
  $m.synchronize do
    puts 'do that.'

For those of you who use such mutexes and are OK with them wrapping the whole block, rufus-scheduler 2.0.12 introduces the :mutex attribute:

s.every '10m', :mutex => 'my_mutex_name' do
  puts 'doing this...'
  sleep 60 * 60 # 1 hour
  puts 'done.'

# if the scheduler is in the blocking task above, crons will get skipped... '2012-01-01 12:00', :mutex => 'my_mutex_name'  do
  puts 'do that.'

Where rufus-scheduler receives a mutex name and manages it for you.

When one wants more control over the granularity, it’s OK to do:

$m =

s.every '10m', :mutex => $m do
  puts 'doing this...'
  sleep 60 * 60 # 1 hour
  puts 'done.'

# if the scheduler is in the blocking task above, crons will get skipped... '2012-01-01 12:00' do
  puts 'do that'
  $m.synchronize do
    puts 'and that.'


Remember that rufus-scheduler is not a cron replacement. Many thanks to all the people who complained or helped in the development of this piece of software over the years.


mailing list:
irc: freenode #ruote


Written by John Mettraux

October 28, 2011 at 11:27 am

rufus-jig 1.0

By the end of 2007 I had written a gem sitting on top of net/http. It was called rufus-verbs, this extra layer added a mini cache for conditional GETs, basic auth and digest auth, cookie jar and more.

I used it for a while, I got surprised by people using it (IIRC the digest auth was their reason for using it). And then I forgot it.

Since last year, I am working with things like CouchDB and ruote-kit. I need an HTTP client that groks JSON. I still need it to understand conditional GETs (etags and co).

So I built rufus-jig, something on top of Ruby’s net/http, net-http-persistent, patron or em-http-request. It uses rufus-json to select the best JSON library available (in the order yajl-ruby, json, activesupport, json-pure).

A GET would look like :

require 'rubygems'

#require 'net/http/persistent'
  # http backend

require 'yajl' # gem install 'yajl-ruby'
  # json backend

require 'rufus/jig'

h ='')

p h.get('/users/jmettraux.json')['description']
  # => "another fool"

You specify the HTTP and the JSON backend before requiring rufus-jig and that’s it.

Rufus-jig comes with a class to deal with some of CouchDB specifics.

require 'net/http/persistent'
require 'yajl'
require 'rufus/jig'

c ='', 'rufus_jig_test')

# PUT and GET

c.put('_id' => 'coffee0', 'category' => 'espresso')
c.put('_id' => 'coffee1', 'category' => 'instantaneous')

coffee1 = c.get('coffee1')
coffee1['brand'] = 'nescafe'

# attaching

coffee0 = c.get('coffee0')

c.attach(coffee0, 'picture','espresso.jpg'), :content_type => 'image/jpeg')

# fetching all docs

p c.all
p c.all(:skip => 100, :limit => 100)

# fetching a batch of docs

p c.all(:keys => %w[ coffee0 coffee2 coffee7 ])

# querying views

p c.query('my_design_doc:my_view', :key => 'Costa Rica')
p c.query_for_docs('my_design_doc:my_view', :key => 'Colombia')

# bulk operations

docs = c.all(:keys => %w[ doc0 doc1 doc3 ])
c.bulk_delete(docs) # deleting in one go

docs = c.all(:keys => %w[ doc0 doc1 doc3 ])
docs.each { |doc| doc['status'] = 'copied' }
c.bulk_put(docs) # updating in one go

# ...

# listening to CouchDB activity

c.on_change do |doc_id, deleted|
  puts "doc #{doc_id} has been #{deleted ? 'deleted' : 'changed'}"

c.on_change do |doc_id, deleted, doc|
  puts "doc #{doc_id} has been #{deleted ? 'deleted' : 'changed'}"
  p doc


Rufus-jig just reached 1.0. I’ll probably go on with extending it. It’ll probably need digest authentication at some point, gzipping, why not. I’ll leave the CouchDB function in it for now. We’ll see.


source code :


Written by John Mettraux

December 8, 2010 at 6:28 am

Posted in couchdb, http, ruby, rufus

Houston.rb ruote presentation by Wes Gamble

During the last Houston.rb meeting, Wes Gamble talked about ruote. Here are his slides :


Wes does an excellent job at presenting the main concepts in ruote, process definitions, participants, workflow vs state management and so on. I’m lucky to have people evangelizing ruote as I’m a too much a “you don’t understand workflow ? You certainly don’t need ruote” type.

There was a funny coincidence : my friend Alain Hoang was visiting Houston at that time and he joined the meeting. Alain and I were co-workers when I started working on ruote. He helped me with the initial Rakefile and tamed rote to build an initial website for the project. He also came up with the initial suggestion for the rufus namespace idea.

Many thanks to Wes for his great presentation (and to Alain for all his help).


Written by John Mettraux

August 16, 2010 at 12:46 am

Posted in ruby, rufus, ruote, workflow

retiring rufus-tokyo

As you probably know, rufus-tokyo is a Ruby FFI wrapper for Tokyo Cabinet|Tyrant, the fine pieces of software delivered by Hirabayashi Mikio.

Rufus-tokyo is 12 or 13 months old, but it’s time to retire it (maintenance mode).

James Edward Gray II is building his Oklahoma Mixer which will, hopefully, completely overlap rufus-tokyo and simply be better, very soon. Once the mixer covers Ruby 1.9.x, JRuby and Tokyo Tyrant (see todo list), rufus-tokyo will be irrelevant.

This is great for me (and for you), we will get a better Ruby library for TC/TT. Please note that for Tokyo Tyrant we already have a very fast option with Flinn Mueller’s ruby-tokyotyrant, which will always be faster than a FFI Tokyo Tyrant library.

James is exposing his motivations for Oklahoma Mixer on its front page. I have a few clarifications to make about them and rufus-tokyo. I use this blog to make these clarifications more accessible.

Why not just use rufus-tokyo?

There is already a Ruby FFI interface to Tokyo Cabinet and more called rufus-tokyo. I am a huge fan of rufus-tokyo and have contributed to that project. I have learned a ton from working with rufus-tokyo and that code was my inspiration for this library.

That said, I did want to change some things that would require big changes in rufus-tokyo. Here are the things I plan to do differently with Oklahoma Mixer:

* Tokyo Cabinet’s B+Tree Database has some neat features, like custom ordering functions, that are hard to expose through rufus-tokyo due to the way it is designed. I would have had to rewrite pretty much that entire section of the library anyway to add these features.
* There are some places where rufus-tokyo uses more memory than it absolutely must or slows itself down a bit with extra iterations of the data. Again, this is a result of how it is designed. It allows more code reuse at the cost of some efficiency. I wanted to improve performance in those areas.
* I’m working on some higher level abstractions for Tokyo Cabinet that I eventually plan to include in this library. These extra tools are the reason I needed to make these changes and additions.
* Finally, rufus-tokyo paved the way to a Ruby-like interface for Tokyo Cabinet. Previous choices were scary in comparison. I wanted to push that movement even further though and get an even more Hash- and File-like interface, where possible.

It’s important to note though that rufus-tokyo does do some things better and it always will. Its advantages are:

* It offers a nice Ruby interface over a raw C extension which is faster than using FFI. I have no intention of duplicating that interface, so rufus-tokyo will remain the right choice for raw speed when using MRI.
* For now, rufus-tokyo is more full featured. It provides interfaces for Tokyo Tyrant and Tokyo Dystopia. I would like to add these eventually, but I’m starting with just Tokyo Cabinet.
* It offers a pure Ruby interface for communicating with Tokyo Tyrant without needing the C libraries installed. I won’t be copying that either, so it will remain the right choice for a no dependency install.
* For now, it’s more mature. A lot of developers have used it and contributed to it. It’s probably the safer library to trust for production applications at this time.

The “raw C extension” mentioned is provided by Hirabayashi-san. Rufus-tokyo only provides a (mostly) unified interface to it (same interface for FFI and the C exts).

The “pure ruby interface for communicating with Tokyo Tyrant” again is provided by Hirabayashi-san.

That doesn’t leave me much merit. All this wrapping effort was code-named Rufus-Edo and you can read more about the motivations and mechanisms in ‘rufus-tokyo 0.1.9 (Rufus::Edo)’. You will also find more details in my Edo Cabinet presentation.

So I’d advise TC/TT Rubyists to support the effort of James and Flinn. For those of you who want to participate in the larger Tokyo Cabinet|Tyrant community, Flinn has set up the Tokyo Cabinet Users mailing list, where lots of people help and share.

Rufus-tokyo is now in maintenance mode. I will still help people with the occasional bug or mem leak. The mailing list is still open (it’s the one for my rufus libraries) and if you know how to write an issue report, you will always find help (random tweets do not count).

Thanks for the fun on the way.


Written by John Mettraux

February 26, 2010 at 12:25 am

ruote and decision tables

ruote 2.1.7 is out and it felt like it was time to update the CsvParticipant that could be found in ruote 0.9.

Ruote has been relying on rufus-decision for its decision table needs and ruote 0.9 was integrating a CsvParticipant directly. For ruote 2.1, the DecisionParticipant is left in the rufus-decision gem.

It’s not the first time I write something about decision tables. But I noticed that I hadn’t really written anything about how to mix ruote processes and decision tables.

Let’s consider the decision table in the adjacent figure.

Given two input values “type of visit” and “participant physician ?” the level of “reimbursement” is decided.

The CSV version of this decision table is available online.

Clearly this decision has to take place after the customer filled his reimbursement claim and it got reviewed by a clerk in the organization.

Up until now, applying the reimbursement rules was part of the tasks of the clerk (with probably some downstream checks), and our goal is to remove some of that burden.

Our process definition would look like :

Ruote.process_definition :name => 'reimbursement', :revision => '1.0' do
  cursor do
    customer :task => 'fill form'
    clerk :task => 'control form'
    rewind :if => '${f:missing_info}'
    finance :task => 'reimburse customer'
    customer :task => 'reimbursement notification'

There are four participants involved in that process definition. Customer, Clerk and Finance our implemented with classical inbox/worklist participants : workitems for them get placed in their workqueue. (With ruote 2.1 I tend to use StorageParticipant for that or I emit the workitem via a ruote-amqp participant towards the target worklist or task manager).

“rewind” isn’t a participant, it’s a ruote expression working with the cursor expression.

That leaves us with the participant “determine_reimbursement_level” to implement.

The straightforward Ruby implementation of the participant would leverage a BlockParticipant and look like :

engine.register_participant 'determine_reimbursement_level' do |workitem|
  workitem.fields['reimbursement'] =
    case workitem.fields['type of visit']
      when 'doctor office'
        workitem.fields['participating physician ?'] == 'yes' ? '90%' : '50%'
      when 'hospital visit'
        workitem.fields['participating physician ?'] == 'yes' ? '0%' : '80%'
      when 'lab visit'
        workitem.fields['participating physician ?'] == 'yes' ? '0%' : '70%'

Given the values in the fields ‘type of visit’ and ‘participating physician ?’ this participant sets the value of the field ‘reimbursement’.

All is well, but you may have noticed that the decision table is rather simplistic. And how does it cope with change ? What will it look like when the rules get hairy ? Will the Ruby developer have to intervene for each change ?

(it might not be a bad thing to rely on the Ruby developer, they tend to test carefully their creations)

Those rules are produced by ‘business users’ and one of their favourite tools is the spreadsheet. Rufus-decision can read CSV output of decision tables. The decision participant would look like :

require 'rufus/decision/participant'
  # gem install rufus-decision

  :table => {
in:type of visit,in:participating physician ?,out:reimbursement
doctor office,yes,90%
doctor office,no,50%
hospital visit,yes,0%
hospital visit,no,80%
lab visit,yes,0%
lab visit,no,70%

Well, OK, that’s nice, but… We still have to restart the application at each change to the decision table or at least re-register the participant with the updated table. Tedious.

If our business people publish the decision table as CSV somewhere over the intranet (yes, I know, it’s an old-fashioned word), the decision participant can be simplified to :

require 'rufus/decision/participant'
  # gem install rufus-decision

  :table => '')

Each time a decision will have to be taken, the table will be read. This lets business users modify the rules at will.

The whole example is packaged as reimbursement.rb. It runs the process from the command line and grabs the decision table online. It shouldn’t be too difficult to take inspiration from it and integrate decision tables in web applications (like ruote-kit or barley).

There are many open possibilities when combining a workflow engine and a decision mechanism. One immediate advantage is that they can evolve with different rhythms. They can also be used in isolation : the clerk in our process could talk the claim over the phone with the customer and run blank decisions to help the customer with incomplete forms.

Of course, you can use rufus-decision without ruote, it’s a plain ruby gem. – the source code of rufus-decision – the source code of ruote – ruote’s documentation – the mailing list for ruote and rufus-decision


Written by John Mettraux

February 17, 2010 at 4:32 am

Posted in bpm, bpms, decision, ruby, rufus, rules, ruote

rufus-tokyo 1.0.4

東京 from its bay

This release contains a fix for a memory leak cornered by Jeremy Hinegardner and James Edward Gray II.

The gem is available via gemcutter and the source is on github.

Many thanks to James and Jeremy and merry Christmas to you all !


Written by John Mettraux

December 25, 2009 at 12:40 am

rufus-lua 1.1.0

luaJust released rufus-lua 1.1.0. The original post I wrote about rufus-lua is named ruby to lua.

This release strongly benefited from Scott Persinger’s work on Laminate, a tool for safe user-authored templates for Vodpod.

Scott needed strong support for callbacks from Lua to Ruby and also ways to pass values back and forth. Many thanks to Scott for the great collaboration.

Those of you interested in Lua and web applications should have a look at Norman Clarke’s lua-haml and at Daniele Alessandri’s mercury (sinatra-like web framework for Lua).


Written by John Mettraux

September 30, 2009 at 5:04 am

Posted in lua, ruby, rufus

rufus-tokyo 1.0.1

yokohama cabinetrufus-tokyo is a ruby-ffi based library for accessing Tokyo Cabinet and Tokyo Tyrant databases. It also feature a Rufus::Edo side where the native ruby/c extensions provided by the TC/TT author are used (for speed) instead of ruby-ffi.

rufus-tokyo contains ffi bindings for Tokyo Dystopia as well, thanks to Jeremy Hinegardner.

This is the changelog for this 1.0.1 release :

== rufus-tokyo - 1.0.1    released 2009/09/18

- todo : add #putcat to Cabinet / Tyrant (Edo and Tokyo)
- todo : implemented search/union/intersection/difference for tables
- todo : added #putdup and #get4 to Cabinet / Tyrant (Edo and Tokyo)
- todo : better dylib 'detection' (Pietro Ferrari)
- todo : aliased lget to mget (thanks Runa)
- todo : proper Abort exception (Kenneth Kalmer)


putcat is used to append data to already stored value. getdup / get4 is only relevant with b+ trees structures. They allow for multiple values stored under one key. This get4/getdup returns all the values stored under one key.

Since Tokyo Cabinet 1.4.29 (Tyrant 1.1.30), cabinet tables and tyrant tables have this new metasearch feature. This is what is meant by “search/union/intersection/difference” in the changelog.

Hirabayashi-san is providing us with a way to combine queries. From rufus-tokyo, it looks like :

require 'rubygems'
require 'rufus/tokyo' # sudo gem install rufus-tokyo

t ='table.tct')

t['pk0'] = { 'name' => 'alfred', 'age' => '22', 'hobby' => 'fishing' }
t['pk1'] = { 'name' => 'bob', 'age' => '18', 'hobby' => 'hunting' }
t['pk2'] = { 'name' => 'charly', 'age' => '45', 'hobby' => 'fishing' }
t['pk3'] = { 'name' => 'doug', 'age' => '77', 'hobby' => 'fencing' }
t['pk4'] = { 'name' => 'ephrem', 'age' => '32', 'hobby' => 'swimming' }

rs = t.union(
  t.prepare_query { |q| q.add 'hobby', :equals, 'fishing' },
  t.prepare_query { |q| q.add 'age', :numgt, '20' }

rs.each { |r| p r }
  # ==>
  # ["pk2", {"name"=>"charly", "hobby"=>"fishing", "age"=>"45"}]
  # ["pk3", {"name"=>"doug", "hobby"=>"fencing", "age"=>"77"}]
  # ["pk4", {"name"=>"ephrem", "hobby"=>"swimming", "age"=>"32"}]
  # ["pk0", {"name"=>"alfred", "hobby"=>"fishing", "age"=>"22"}]


This example returns the persons in the table who do fishing for a hobby OR whose age is greater than 20. The other methods understood by the table are intersection and difference.

For more information, see the rufus-tokyo source on github and the rufus-tokyo rdoc.

Many thanks to all the persons who contributed to rufus-tokyo.


On the general Tokyo Cabinet / Tyrant front, Flinn Mueller, the author of ruby-tokyotyrant started a couple months ago a mailing list about Tokyo Cabinet / Tyrant. Lots of knowledge about the Tokyo products is exchanged there.

There is also the release of tyrantmanager by Jeremy Hinegardner (the author of the Dystopia bindings in rufus-tokyo). It’s a neat command line tool for managing Tokyo Tyrant instances, individually or in batch.


Written by John Mettraux

September 18, 2009 at 1:47 am

edo cabinet – rubykaigi2009

Here are my slides for the “edo cabinet” talk at the RubyKaigi2009.

I presented about Ruby-FFI, Tokyo Cabinet|Tyrant and then rufus-tokyo and Rufus::Edo.

These are just slides, here is the english transcript of the actual talk.


Many thanks to the RubyKaigi team for organizing this great event ! Special kudos to Leonard Chin for the on-the-fly translations, throughout the 3 days.

You’ll find videos of most of the talks on (thanks to the KaigiFreaks) and most of the slides are available from Slideshare.


Written by John Mettraux

July 28, 2009 at 11:57 pm

rufus-tokyo 1.0.0

climaxJust released rufus-tokyo 1.0.0

This is mostly a “cleanup” release (spec reorg, to_s enforcement on key and values, and dropped backward compatibility with older TC/TT releases). Hence the 1.0.0.

There are new features though :

* Matthew King helped me add process method for table queries.

* Jeremy Hinegardner did an initial contribution on the front FFI / Tokyo Dystopia.

* Adam Keys contributed the ext method for the table structures (sorry, I had forgotten).

* full text search operators for table queries (:ftsphrase, :ftsec, :ftsor: ftsand) and the accompanying :token, :qgram and :opt inverted index types. I haven’t played too much with them. Feedback is welcome.

For those of you interested in extending Tokyo Tyrant via Lua functions, I warmly recommend Ilya Grigork’s post about it.

Many thanks to all the people who helped.

There will be a next release soon, as I’m busy trying to catch up with Hirabayashi-san.

rufus-tokyo :

(picture graciously from @michelledasilva)


Written by John Mettraux

July 23, 2009 at 1:24 am