about processes and engines

Archive for the ‘ffi’ Category

rufus-tokyo 1.0.4

東京 from its bay

This release contains a fix for a memory leak cornered by Jeremy Hinegardner and James Edward Gray II.

The gem is available via gemcutter and the source is on github.

Many thanks to James and Jeremy and merry Christmas to you all !


Written by John Mettraux

December 25, 2009 at 12:40 am

rufus-tokyo 1.0.1

yokohama cabinetrufus-tokyo is a ruby-ffi based library for accessing Tokyo Cabinet and Tokyo Tyrant databases. It also feature a Rufus::Edo side where the native ruby/c extensions provided by the TC/TT author are used (for speed) instead of ruby-ffi.

rufus-tokyo contains ffi bindings for Tokyo Dystopia as well, thanks to Jeremy Hinegardner.

This is the changelog for this 1.0.1 release :

== rufus-tokyo - 1.0.1    released 2009/09/18

- todo : add #putcat to Cabinet / Tyrant (Edo and Tokyo)
- todo : implemented search/union/intersection/difference for tables
- todo : added #putdup and #get4 to Cabinet / Tyrant (Edo and Tokyo)
- todo : better dylib 'detection' (Pietro Ferrari)
- todo : aliased lget to mget (thanks Runa)
- todo : proper Abort exception (Kenneth Kalmer)


putcat is used to append data to already stored value. getdup / get4 is only relevant with b+ trees structures. They allow for multiple values stored under one key. This get4/getdup returns all the values stored under one key.

Since Tokyo Cabinet 1.4.29 (Tyrant 1.1.30), cabinet tables and tyrant tables have this new metasearch feature. This is what is meant by “search/union/intersection/difference” in the changelog.

Hirabayashi-san is providing us with a way to combine queries. From rufus-tokyo, it looks like :

require 'rubygems'
require 'rufus/tokyo' # sudo gem install rufus-tokyo

t ='table.tct')

t['pk0'] = { 'name' => 'alfred', 'age' => '22', 'hobby' => 'fishing' }
t['pk1'] = { 'name' => 'bob', 'age' => '18', 'hobby' => 'hunting' }
t['pk2'] = { 'name' => 'charly', 'age' => '45', 'hobby' => 'fishing' }
t['pk3'] = { 'name' => 'doug', 'age' => '77', 'hobby' => 'fencing' }
t['pk4'] = { 'name' => 'ephrem', 'age' => '32', 'hobby' => 'swimming' }

rs = t.union(
  t.prepare_query { |q| q.add 'hobby', :equals, 'fishing' },
  t.prepare_query { |q| q.add 'age', :numgt, '20' }

rs.each { |r| p r }
  # ==>
  # ["pk2", {"name"=>"charly", "hobby"=>"fishing", "age"=>"45"}]
  # ["pk3", {"name"=>"doug", "hobby"=>"fencing", "age"=>"77"}]
  # ["pk4", {"name"=>"ephrem", "hobby"=>"swimming", "age"=>"32"}]
  # ["pk0", {"name"=>"alfred", "hobby"=>"fishing", "age"=>"22"}]


This example returns the persons in the table who do fishing for a hobby OR whose age is greater than 20. The other methods understood by the table are intersection and difference.

For more information, see the rufus-tokyo source on github and the rufus-tokyo rdoc.

Many thanks to all the persons who contributed to rufus-tokyo.


On the general Tokyo Cabinet / Tyrant front, Flinn Mueller, the author of ruby-tokyotyrant started a couple months ago a mailing list about Tokyo Cabinet / Tyrant. Lots of knowledge about the Tokyo products is exchanged there.

There is also the release of tyrantmanager by Jeremy Hinegardner (the author of the Dystopia bindings in rufus-tokyo). It’s a neat command line tool for managing Tokyo Tyrant instances, individually or in batch.


Written by John Mettraux

September 18, 2009 at 1:47 am

edo cabinet – rubykaigi2009

Here are my slides for the “edo cabinet” talk at the RubyKaigi2009.

I presented about Ruby-FFI, Tokyo Cabinet|Tyrant and then rufus-tokyo and Rufus::Edo.

These are just slides, here is the english transcript of the actual talk.


Many thanks to the RubyKaigi team for organizing this great event ! Special kudos to Leonard Chin for the on-the-fly translations, throughout the 3 days.

You’ll find videos of most of the talks on (thanks to the KaigiFreaks) and most of the slides are available from Slideshare.


Written by John Mettraux

July 28, 2009 at 11:57 pm

rufus-tokyo 0.1.13

shinjukuWith the initial releases of rufus-tokyo, I happily cut corners and went with C strings (ending with NUL).

This isn’t optimal, often you need to store binary data as the value, the resulting ‘string’ contains NUL characters and values get truncated at restitution. I was working around that with Base64 encoding. Fine, the perf cost isn’t too heavy. But then, binary data was working fine on the Rufus::Edo side (wrapping the original tokyo{cabinet|tyrant}-ruby), why should Rufus::Tokyo be an exception ?

rufus-tokyo 0.1.13 which just got released fixes that issue.

Thanks to Ben Mabey for pointing out the issue.

Kamal Fariz Mahyuddin was kind enough to fork rufus-tokyo and add the #putkeep(k, v) method which only put if there was no previous value associated with the key. The collaboration with Kamal also prompted me to provide #add_int and #add_double for int/double counters in cabinet/tyrant.


Written by John Mettraux

June 2, 2009 at 12:47 am

rufus-tokyo 0.1.12, ext(lua)

tokyo-luaJust released version 0.1.12 of rufus-tokyo, a Ruby library for accessing Tokyo Cabinet and Tokyo Tyrant (via FFI or via the Ruby classes provided with the Tokyo products).

Tokyo Tyrant, once successfully compiled with –enable-lua, is open to lots of interesting usages.

In his last post on the Mixi dev blog, the author of Tokyo Cabinet / Tyrant is exposing his ideas about an application of map/reduce leveraging Tokyo Tyrant (post in Japanese, machine translation here). This demonstration relies on the embedded Lua runtime for the mapper/reducer implementation.

For this post, I’ll just show the vanilla “incr” function found in the Tokyo Tyrant documentation :

-- incr.lua
function incr (key, i)
  i = tonumber(i)
  if not i then
    return nil
  local old = tonumber(_get(key))
  if old then
    i = old + i
  if not _put(key, i) then
    return nil
  return i

This Lua function increments the current value bound for a key by a given amount.

You start your Lua-enabled Tokyo Tyrant with something like :

  ttserver -ext incr.lua test.tch

and then hit it with code like :

# test.rb

require 'rubygems'
require 'rufus/tokyo/tyrant' # sudo gem install rufus-tokyo

t ='', 1978)

5.times do
  puts t.ext(:incr, 'my-counter', 2).to_i


The “ext” method is available with the latest release of rufus-tokyo, for the Rufus::Tokyo::Tyrant and Rufus::Edo::NetTyrant classes.

I updated my documentation about Tokyo Cabinet / Tyrant install with some –enable-lua information.

This release of rufus-tokyo also adds transaction support for the Rufus::Tokyo::Cabinet class (requires Tokyo Cabinet 1.4.13).


Written by John Mettraux

April 7, 2009 at 5:32 am

rufus-tokyo 0.1.10, limit(count, offset)

musicJust released rufus-tokyo 0.1.10. It supports the new setlimit method in Tokyo Cabinet 1.4.10 tables and Tokyo Tyrant 1.1.17 tables.

(I put up a piece of documentation on how to install TC and TT at

What’s the deal with this setlimit method ? Well, it’s an improvement over setmax which only accepted a count parameter.

require 'rubygems'
require 'rufus/tokyo'

t ='table.tct')

t['pk0'] = { 'name' => 'alfred', 'age' => '22' }
t['pk1'] = { 'name' => 'bob', 'age' => '18' }
t['pk2'] = { 'name' => 'charly', 'age' => '45' }
t['pk3'] = { 'name' => 'doug', 'age' => '77' }
t['pk4'] = { 'name' => 'ephrem', 'age' => '32' }

p t.query { |q|
  q.order_by 'age'
  q.limit(2, 3) # 2 records max, skip 3 records
  # => [ {"name"=>"ephrem", :pk=>"pk4", "age"=>"32"},
  #      {"name"=>"charly", :pk=>"pk2", "age"=>"45"} ]


(Oops, I noticed the max/count parameter is not always respected, submitted a bug report… And not sure if this offset/skip is ‘pagination’ out of the box, it seems more like a ‘skip’, literally)

This release of rufus-tokyo supports previous versions of Tokyo Cabinet and Tokyo Tyrant as well.

sudo gem install rufus-tokyo or

Many thanks to Benjamin Yu for pointing out the issue.


Written by John Mettraux

March 19, 2009 at 5:50 am

ruby to lua

luaWith the help from Alain I just released rufus-lua 0.1.0.

This ruby gem talks to Lua via Ruby FFI. One could say it’s “embedding Lua”, as Lua is described as “Lua is a powerful, fast, lightweight, embeddable scripting language.”

I feel tempted by Lua, there’s the speed, the austerity and that javascript feeling.


Rufus-lua is not the first bridge between Lua and Ruby, there are also :


They are worth a look, but I was a bit disappointed, they are not available via ‘sudo gem install …’.

Rufus-lua is available via Rubygems, but since it’s a FFI thing, it needs to talk to the dynamic library packaging of Lua. Well, it’s not available by default, there’s some work involved (but not too much).

Something great about FFI is that it works for Ruby 1.8.6, Ruby 1.9.1 and JRuby (1.1.6) (and will work soon for Rubinius, its original platform).


Once liblua.dylib is on your system, along with rufus-lua (sudo gem install rufus-lua), you can play with code like :

require 'rubygems'
require 'rufus/lua'

s =

t = s.eval(%{
  return { message = { 'hello', 'from', 'Lua' },
    funk = function (x) return 2^x end }

p t['message'].to_a # => ["hello", "from", "lua"]
p t['funk'].call(8) # => 256.0


Well, it’s only a 0.1.0 release…


As rufus-lua is using FFI it’s slower at the “frontier”, here is a naive benchmark for computing a Fibonacci suite, with Fiber (Ruby 1.9) and Coroutines on the Lua side :

require 'benchmark'
require 'rubygems'
require 'rufus/lua'

  n1 = n2 = 1
  loop do
    Fiber.yield n1
    n1, n2 = n2, n1+n2

s = %{
  co = coroutine.create(function ()
    local n1 = 1; local n2 = 1
    while true do
      n1, n2 = n2, n1+n2
  return co

LUAFIBS = LUA.eval(s)

N = 10_000
Benchmark.benchmark(' ' * 31 + Benchmark::Tms::CAPTION, 31) do |b|'ruby') do
    N.times { RUBYFIBS.resume }
  end'lua via ruby') do
    N.times { LUAFIBS.resume }
  end'lua') do
    LUA.eval("for i = 0, #{N} do coroutine.resume(co) end")


For the original. Running it on my board yields something along the lines of :

$ ruby19 test/bm_fiber.rb
                  user     system      total        real
ruby          0.050000   0.010000   0.060000 (  0.052032)
lua via ruby  0.180000   0.000000   0.180000 (  0.195411)
lua           0.010000   0.000000   0.010000 (  0.006394)

Well, nothing spectacular, that just states that it’s better to not cross the border too often : the “lua via ruby” bench has Ruby resuming the Lua coroutine 10_000 times and it’s costly.


on github :
rdoc :
the rest of rufus :

lua :
ffi :


Written by John Mettraux

March 16, 2009 at 9:39 am

Posted in ffi, jruby, lua, ruby, rufus