about processes and engines

ruby-ffi, Tokyo Cabinet

tcabinetI know, we have a Tyrant but I wanted to focus just on the Cabinet, staying on my side of the network interface.

Tokyo Cabinet is a great [C] library for managing DBM hashes, it’s very efficient, so efficient it’s one of the workhorses behind Mixi, the top japanese social network (damn me, I used the word ‘social’…)

I really wanted to use Tokyo Cabinet from Ruby, I could only find the a datamapper related gem, no ‘sudo gem install’ 1-stop-only option for me. Hirabayashi-san, the original author of Tokyo Cabinet wrote a Ruby lib, but it’s not packaged as a gem, and then I wanted something that looked like a ruby hash, enumerable included.

ruby-ffi is a ruby extension for binding ruby to dynamic libraries, it originated in the Rubinius project and is now available for MRI Ruby (the ‘ruby-ffi’ gem) and JRuby as well.

So, if you have the Tokyo Cabinet library installed on your system, you can venture into :

sudo gem install rufus-tokyo

(it should install its dependency ruby-ffi in the same run)

and then

  require 'rubygems'
  require 'rufus/tokyo'

  db ='data.tch')

  db['nada'] = 'surf'

  p db['nada'] # => 'surf'
  p db['lost'] # => nil

  5000.times { |i| db[i.to_s] = "x" }

  p db.inject { |r, (k, v)| k } # => '4999'


will create/open the hash database in “data.tch” and store all its stuff in there. It does it via the dynamic library for Tokyo Cabinet that it looks up in either /opt/local/lib or /usr/local/lib (or another path specified by the environment variable TOKYO_CABINET_LIB).

OK, nothing exciting, just that it’s fast, handy, works on the major Ruby platforms just a ‘sudo gem install’ away.

It’s rather rudimentary for now, but with a bit time…

source :
rdoc :
rest of rufus :
mailing list :

Many thanks to the authors of Tokyo Cabinet and ruby-ffi !

Written by John Mettraux

January 23, 2009 at 8:13 am

Posted in jruby, ruby, rufus, tokyocabinet

4 Responses

Subscribe to comments with RSS.

  1. You had the same thought that I had… now I can kill my repo I was working on and possibly work on yours.



    January 23, 2009 at 5:45 pm

  2. Sorry Justin,

    I looked at github and at the available gems, and couldn’t find anything, so I tried ffi and it was so simple that in one hour I had a version.

    Feel free to fork


    John Mettraux

    January 24, 2009 at 12:27 am

  3. Yeah, I have a methodology I follow when attempting to make something better, through experimenting with APIs, and libraries that already exist in some form.

    1. Write my own tests for what I’m exploring, this helps me learn it quicker and find my own use cases.

    2. Develop a rough API spec, of the most basic things I want to achieve.

    3. Optimize the platform for those simple functionality pieces.

    4. Extend in further… rinse and repeat.

    The thing about the Tokyo Cabinet/Tyrant APIs are that they are so simple and well made C functions. Thats the reason you were able to apply them so easily through FFI. I just read about Ruby’s FFI implementation, and haven’t had the time to confirm this, but heard it was still slow and not near production ready.

    I’ll definitely fork and see what I can contribute, I have a few areas I was going to look into, as well as Tyrant support. Though I was going to build pure C extensions, like the one that comes with Cabinet already. Maybe I’ll write some benchmarks for those.


    January 24, 2009 at 12:40 am

  4. […] a comment » With the initial releases of rufus-tokyo, I happily cut corners and went with C strings (ending with […]

Comments are closed.

%d bloggers like this: