ruote in 20 minutes

Kenneth Kalmer presented ruote in 20 minutes. The slides are available from his blog post.

I linked to his presentation from the ruote documentation and I also added another example to the ruote quickstart.

What’s great about Kenneth presentation is that he starts with a state machine example and builds on it to reach ruote territory.

Kenneth’s use of ruote in his company has one particularly interesting aspect : it leverages XMPP participants (contributed by Kenneth). The engine communicates with participants to businness processes over XMPP (Jabber / Gtalk). Very nice.

rufus-tokyo 0.1.9 (Rufus::Edo)

I have just released rufus-tokyo 0.1.9 (sudo gem install rufus-tokyo)

Rufus-tokyo provides rubyist friendly classes for accessing Tokyo Cabinet and Tokyo Tyrant structures. It does so by binding Tokyo C libraries via FFI and …

Hirabayashi-san, the author of Tokyo Cabinet and Tyrant provides a ‘native’ Ruby library for Tokyo Cabinet and a pure Ruby library for Tokyo Tyrant. My initial motivation for writing rufus-tokyo was that those classes are very C oriented and I wanted to interact in an elegant way with the Tokyo ‘products’.

Another issue was that the author was not providing his libraries via gem install. But this has changed, Dane Jensen is now mirroring the libraries (cabinet, tyrant) at github and releasing them as gems (sudo gem install careo-tokyocabinet careo-tokyotyrant –source http://gems.github.com)

As Ilya Grigorik pointed out FFI is slower than extconf.rb classical Ruby to C bindings. I wanted to benefit from the speed of the ‘native’ bindings.

I have added to rufus-tokyo a Rufus::Edo namespace with 2 classes (cabinet, table) wrapping the ‘native’ ruby gem and 2 classes wrapping the ‘pure’ Ruby classes (cabinet, table) for Tyrant. The ‘pure’ ruby classes for Tyrant, despite being a bit slow are interesting because they don’t require the installation of the Cabinet + Tyrant libraries.

There is a bit of documentation at :

http://github.com/jmettraux/rufus-tokyo/tree/master/lib/rufus/edo

The Rufus::Tokyo classes and the Rufus::Edo classes have the same methods, making it easy to plug one for the other. For instance, in my workflow engine (ruote), I’ve made sure the native bindings are used if present. At instantiation/connection time, the right class (native vs FFI, Edo vs Tokyo) is chosen, the rest of the code is the same.

Here is a tiny decision table for choosing which ruby lib to use :

If only need a btree or a hash (no tables) served by a Tyrant, a native libmemcached wrapper might be a good option if speed is a must (yes, Tokyo Tyrant speaks ‘memcached’).

I have some very naive benchmarks at http://gist.github.com/71285 (Tokyo Cabinet 1.4.8 and Tyrant 1.1.16). Unfortunately the memcached part is not done with a native libmemcached wrapper.

rufus-tokyo : http://github.com/jmettraux/rufus-tokyo/

rufus-tokyo 0.1.5, hail to the Tyrant

If you go to the the Tokyo Cabinet page, you’ll sometimes be welcome with a banner saying “Nagoya Cabinet” or “Shinjuku Cupboard”, no worries, you’re at the right place.

The two main projects of Mikio Hirabayashi are Tokyo Cabinet (local hash / table) and Tokyo Tyrant (remote Tokyo Cabinet).

After supporting Tokyo Cabinet in rufus-tokyo 0.1.4, Justin and I included support for Tokyo Tyrant in rufus-tokyo 0.1.5.

Getting Tokyo Cabinet and Tyrant is relatively easy, it boils down to getting the source from sf.net and then unpacking them and running “./configure && make && sudo make” for both of them (a more detailed description of the compile/install steps).

#
# start a tyrant server from the command line :
#
#   ttserver -port 45000 data.tch
#

require 'rubygems'
require 'rufus/tokyo' # sudo gem install 'rufus-tokyo'

t = Rufus::Tokyo::Tyrant.new('localhost', 45000)

t['shinjuku'] = 'twin towers'
t['ikebukuro'] = 'pond bag'

p t['ikebukuro'] # => 'pond bag'

t.close

Looks great, but since Tokyo Tyrant speaks kling… memcached, one is probably better served (in terms of performance) by a classic ruby memcached client library.

Still there is one area where rufus-tokyo + Tokyo Tyrant could make sense :

#
# on the command line, launch a tyrant server with a stable structure :
#
#   ttserver -port 45001 data.tct
#
# (note the .tct suffix, it indicates to the Tyrant it has to create
# a table structure)
#

require 'rubygems'
require 'rufus/tokyo/tyrant' # sudo gem install 'rufus-tokyo'

t = Rufus::Tokyo::TyrantTable.new('localhost', 45001)

t['pk0'] = { 'name' => 'jim', 'age' => '25', 'lang' => 'ja,en' }
t['pk1'] = { 'name' => 'jeff', 'age' => '32', 'lang' => 'en,es' }
t['pk2'] = { 'name' => 'jack', 'age' => '44', 'lang' => 'en' }
t['pk3'] = { 'name' => 'jake', 'age' => '45', 'lang' => 'en,li' }

# ...

p t.keys
  # => [ 'pk0', 'pk1', 'pk2', 'pk3' ]

p t.query { |q|
  q.add 'lang', :includes, 'en'
  q.limit 2
}
  # =>
  # [{"name"=>"jim", :pk=>"pk0", "lang"=>"ja,en", "age"=>"25"},
  #  {"name"=>"jeff", :pk=>"pk1", "lang"=>"en,es", "age"=>"32"}]

# ...

t.close

There is one potential and interesting development for rufus-tokyo : Yehuda Katz integrated it into Moneta, his “unified interface to key/value stores”. Looking forward to see how it develops.

Many thanks to Justin, Yehuda and Zev for their work/support/help.

http://github.com/jmettraux/rufus-tokyo

Tested with Ruby 1.8.6 and Ruby 1.9.1p0 (There is a small issue with JRuby 1.1.6, I hope to solve it very soon and release a 0.1.6 for it).
tyrant

rufus-tokyo 0.1.3 with Tokyo Cabinet tables

Just released the Ruby gem rufus-tokyo 0.1.3. It gives easy access to Tokyo Cabinet hashes and tables via Ruby.

  require 'rubygems'
  require 'rufus/tokyo'

  db = Rufus::Tokyo::Cabinet.new('data.tch')

  db['nada'] = 'surf'

  p db['nada'] # => 'surf'
  p db['lost'] # => nil

  db.close

This new release has undergone serious code restructurations (thanks Justin) and it features access to Tokyo Cabinet “tables” :

  require 'rubygems'
  require 'rufus/tokyo/cabinet/table'

  t = Rufus::Tokyo::Table.new('table.tdb', :create, :write)

  t['pk0'] = { 'name' => 'alfred', 'age' => '22' }
  t['pk1'] = { 'name' => 'bob', 'age' => '18' }
  t['pk2'] = { 'name' => 'charly', 'age' => '45' }
  t['pk3'] = { 'name' => 'doug', 'age' => '77' }
  t['pk4'] = { 'name' => 'ephrem', 'age' => '32' }

  p t.query { |q|
    q.add_condition 'age', :numge, '32'
    q.order_by 'age'
  }
    # => [ {"name"=>"ephrem", :pk=>"pk4", "age"=>"32"},
    #      {"name"=>"charly", :pk=>"pk2", "age"=>"45"} ]

  p t.query { |q|
    q.add_condition 'name', :matches, 'a'
    q.order_by 'name', :desc
  }
    # => [ {"name"=>"charly", :pk=>"pk2", "age"=>"45"},
    #      {"name"=>"alfred", :pk=>"pk0", "age"=>"22"} ]

  t.close

Zev and I ran a few benchmarks with this table feature.

Warning : this is only some exploratory benchmarking ! Don’t build religious dogma on top of it !

Yielded results were quite interesting :

• playing with 4 records : http://gist.github.com/54343 (AR (mysql/sqlite3), TC tables)
• 10_000 records : http://gist.github.com/54371 (AR (mysql), TC tables)
• 1_000 records : http://gist.github.com/54409 (AR (mysql/sqlite3), TC tables)

(Thanks Faker)

My integration work / gem is very naive for now, Mikio Hirabayashi’s Tokyo Cabinet project is rather deep, it features many interesting structures (hashes, b-trees, tables) and then there’s the Dystopia…

I’m looking forward continuing this work and enhancing rufus-tokyo to provide further Tokyo Cabinet features / tuning options.

http://github.com/jmettraux/rufus-tokyo/

Tested with Ruby 1.8.6 and JRuby 1.1.6.

ruby-ffi, Tokyo Cabinet

I know, we have a Tyrant but I wanted to focus just on the Cabinet, staying on my side of the network interface.

Tokyo Cabinet is a great [C] library for managing DBM hashes, it’s very efficient, so efficient it’s one of the workhorses behind Mixi, the top japanese social network (damn me, I used the word ’social’…)

I really wanted to use Tokyo Cabinet from Ruby, I could only find the a datamapper related gem, no ’sudo gem install’ 1-stop-only option for me. Hirabayashi-san, the original author of Tokyo Cabinet wrote a Ruby lib, but it’s not packaged as a gem, and then I wanted something that looked like a ruby hash, enumerable included.

ruby-ffi is a ruby extension for binding ruby to dynamic libraries, it originated in the Rubinius project and is now available for MRI Ruby (the ‘ruby-ffi’ gem) and JRuby as well.

So, if you have the Tokyo Cabinet library installed on your system, you can venture into :

sudo gem install rufus-tokyo

(it should install its dependency ruby-ffi in the same run)

and then

  require 'rubygems'
  require 'rufus/tokyo'

  db = Rufus::Tokyo::Cabinet.new('data.tch')

  db['nada'] = 'surf'

  p db['nada'] # => 'surf'
  p db['lost'] # => nil

  5000.times { |i| db[i.to_s] = "x" }

  p db.inject { |r, (k, v)| k } # => '4999'

  db.close

will create/open the hash database in “data.tch” and store all its stuff in there. It does it via the dynamic library for Tokyo Cabinet that it looks up in either /opt/local/lib or /usr/local/lib (or another path specified by the environment variable TOKYO_CABINET_LIB).

OK, nothing exciting, just that it’s fast, handy, works on the major Ruby platforms just a ’sudo gem install’ away.

It’s rather rudimentary for now, but with a bit time…

source : http://github.com/jmettraux/rufus-tokyo
rdoc : http://rufus.rubyforge.org/rufus-tokyo
rest of rufus : http://rufus.rubyforge.org/
mailing list : http://groups.google.com/group/rufus-ruby

Many thanks to the authors of Tokyo Cabinet and ruby-ffi !

rufus-scheduler 1.0.12 released

Just released version 1.0.12 of the rufus-scheduler gem.

It contains a few improvements, like a more generic find_jobs() methods, but the main new feature is the :timeout attribute, which, when present, limits the time allocated for a triggered job.

require 'rubygems'
require 'rufus/scheduler' # sudo gem install rufus-scheduler

s = Rufus::Scheduler.start_new

s.every "10h30m", :timeout => "3h" do
  do_that_long_job()
end

Every 10 hours and 30 minutes the ‘long job’ will get triggered, if after 3 hours it isn’t done, it will get interrupted via a Rufus::TimeOutError.

Previously, users were wrapping the job inside a ‘Timeout’ construct which used a second thread. The :timeout attribute simply schedules a rufus job for the expiration, leveraging its own infrastructure.

Thanks to Xianhang Zhang, K Liu and Tim Uckun for their feedback.

rufus : http://rufus.rubyforge.org
the scheduler : http://rufus.rubyforge.org/rufus-scheduler
source : http://github.com/jmettraux/rufus-scheduler
mailing list : http://groups.google.com/group/rufus-ruby

new ruote quickstart

Ruote has a new website, like the one for Rufus, it’s based on the excellent Webby.

I wrote yesterday about renovating the quickstart, I had initially written an example showing a mini todo tool, but this time I switched to a “mechanical turk” like example where a process instance fetches flickr pictures and presents them to human participants for evaluation and choice.

The process definition boils down to :

class PicSelectionProcess < OpenWFE::ProcessDefinition

  sequence do

    get_pictures

    concurrence :merge_type => 'mix' do
      user_alice
      user_bob
      user_charly
    end

    show_results
      # display the pictures chosen by the users
  end
end

the participant for fetching the pictures is called ‘get_pictures’ :

engine.register_participant :get_pictures do |workitem|

  feed = Atom::Feed.new(
    "http://api.flickr.com/services/feeds/photos_public.gne"+
    "?tags=#{workitem.tags.join(',')}&format=atom")
  feed.update!
  workitem.pictures = feed.entries.inject([]) do |a, entry|
    a << [ entry.title, entry.authors.first.name, entry.links.first.href ]
  end
end

That’s it for the quickstart. Now I have to update the tea tasting team example and the japanese website.

mini todo tool

I’m currently rebuilding the Ruote web site, as I did for the Rufus one.

A quickstart example is important, so I spent some time trying to write a new one, demonstrating the workflow engine in a few lines of code.

I wrote a mini todo tool, a loop that ends when all the tasks in a task template have been performed. The code is there :

http://gist.github.com/35894

I’m not sure if I will keep it as a “quickstart”, it’s fun anyway since its pair process/engine can be stopped/restarted at will.

pilatus for yahoo pipes

I just updated the activity page for Ruote (warning it needs some CSS love). It’s populated with the result of a Yahoo Pipe that gathers the feeds of Ruote’s activity, be it on github, the users mailing list or this blog.

It all happens on the client side. I’ve written a small javascript library called Pilatus to which you give the URL of a pipe and the id of the HTML div element to fill and that’s all.

  <script src='/pilatus.js'><script>

  <div id='feed'></div>

  <script>
    Pilatus.loadAndRender(
      "feed",
      "http://pipes.yahoo.com/pipes/pipe.run?_id=XGwyd_LA3RGNSAMs6icw5g");
  </script>

Nothing special. License is MIT, washing my hands.

the js : http://github.com/jmettraux/pilatus/tree/master/public/pilatus.js
the project : http://github.com/jmettraux/pilatus

ruote nov2008 status

In a few paragraphs, lots of links about the status of Ruote (OpenWFEru), my/our open source ruby workflow engine.

But at first, a must-read : BPM is not software engineering, excellent post by Keith Swenson. I interpret it as a call for humility.

Diego Moreno Naharro, a PhD student at the Universidad Politechnica de Madrid recently presented about Workflows on Rails (spanish). His presentation takes the broad view, it especially starts by discussing “act_as_a_state_machine” which, along with its descendants, is a great Rails plugin for resource lifecycle handling. The presentation then goes on to show how Diego and his team leveraged AtomPub and Ruote to build their workflow solution.

On the Ruote mailing list, Diego announced that his research group intended to continue this effort and to develop various Flex based editors for their solution. I’m looking forward tracking this work.

I’ve been supporting Kenneth Kalmer as he implemented a JabberParticipant and a JabberListener to connect the Ruote engine to communicate with participants over XMPP.

With the help of Raphael Simon, I’m currently adding stronger error handling capabilities to Ruote. The current error journal mechanism is OK when you want to describe only the “happy path” in your process definition, but relies too much on administrator intervention.

I did a lot of work on making ruote-rest and ruote-web2 converge, especially in the way the present resources over HTTP (lots of linking work). Kenneth helped me with Ruote-rest and its authentication configuration.

I’d like to release 0.9.20 before the end of the year, I hope I can do it.

0.9.20 will be slightly incompatible with 0.9.19, what does it mean ? Simply that process instances started in a 0.9.19 engine cannot be migrated directly to a 0.9.20 engine, I strongly recommend starting new processes on a 0.9.20 engine and letting old 0.9.19 processes finish their lives in their 0.9.19 engine (else make sure to migrate successfully to a 0.9.20 test environment at first).

Many thanks to Diego, Kenneth and Raphael !