about processes and engines

retiring rufus-tokyo

As you probably know, rufus-tokyo is a Ruby FFI wrapper for Tokyo Cabinet|Tyrant, the fine pieces of software delivered by Hirabayashi Mikio.

Rufus-tokyo is 12 or 13 months old, but it’s time to retire it (maintenance mode).

James Edward Gray II is building his Oklahoma Mixer which will, hopefully, completely overlap rufus-tokyo and simply be better, very soon. Once the mixer covers Ruby 1.9.x, JRuby and Tokyo Tyrant (see todo list), rufus-tokyo will be irrelevant.

This is great for me (and for you), we will get a better Ruby library for TC/TT. Please note that for Tokyo Tyrant we already have a very fast option with Flinn Mueller’s ruby-tokyotyrant, which will always be faster than a FFI Tokyo Tyrant library.

James is exposing his motivations for Oklahoma Mixer on its front page. I have a few clarifications to make about them and rufus-tokyo. I use this blog to make these clarifications more accessible.

Why not just use rufus-tokyo?

There is already a Ruby FFI interface to Tokyo Cabinet and more called rufus-tokyo. I am a huge fan of rufus-tokyo and have contributed to that project. I have learned a ton from working with rufus-tokyo and that code was my inspiration for this library.

That said, I did want to change some things that would require big changes in rufus-tokyo. Here are the things I plan to do differently with Oklahoma Mixer:

* Tokyo Cabinet’s B+Tree Database has some neat features, like custom ordering functions, that are hard to expose through rufus-tokyo due to the way it is designed. I would have had to rewrite pretty much that entire section of the library anyway to add these features.
* There are some places where rufus-tokyo uses more memory than it absolutely must or slows itself down a bit with extra iterations of the data. Again, this is a result of how it is designed. It allows more code reuse at the cost of some efficiency. I wanted to improve performance in those areas.
* I’m working on some higher level abstractions for Tokyo Cabinet that I eventually plan to include in this library. These extra tools are the reason I needed to make these changes and additions.
* Finally, rufus-tokyo paved the way to a Ruby-like interface for Tokyo Cabinet. Previous choices were scary in comparison. I wanted to push that movement even further though and get an even more Hash- and File-like interface, where possible.

It’s important to note though that rufus-tokyo does do some things better and it always will. Its advantages are:

* It offers a nice Ruby interface over a raw C extension which is faster than using FFI. I have no intention of duplicating that interface, so rufus-tokyo will remain the right choice for raw speed when using MRI.
* For now, rufus-tokyo is more full featured. It provides interfaces for Tokyo Tyrant and Tokyo Dystopia. I would like to add these eventually, but I’m starting with just Tokyo Cabinet.
* It offers a pure Ruby interface for communicating with Tokyo Tyrant without needing the C libraries installed. I won’t be copying that either, so it will remain the right choice for a no dependency install.
* For now, it’s more mature. A lot of developers have used it and contributed to it. It’s probably the safer library to trust for production applications at this time.

The “raw C extension” mentioned is provided by Hirabayashi-san. Rufus-tokyo only provides a (mostly) unified interface to it (same interface for FFI and the C exts).

The “pure ruby interface for communicating with Tokyo Tyrant” again is provided by Hirabayashi-san.

That doesn’t leave me much merit. All this wrapping effort was code-named Rufus-Edo and you can read more about the motivations and mechanisms in ‘rufus-tokyo 0.1.9 (Rufus::Edo)’. You will also find more details in my Edo Cabinet presentation.

So I’d advise TC/TT Rubyists to support the effort of James and Flinn. For those of you who want to participate in the larger Tokyo Cabinet|Tyrant community, Flinn has set up the Tokyo Cabinet Users mailing list, where lots of people help and share.

Rufus-tokyo is now in maintenance mode. I will still help people with the occasional bug or mem leak. The mailing list is still open (it’s the one for my rufus libraries) and if you know how to write an issue report, you will always find help (random tweets do not count).

Thanks for the fun on the way.


Written by John Mettraux

February 26, 2010 at 12:25 am

8 Responses

Subscribe to comments with RSS.

  1. I think rufus-Tokyo does leave you with a lot of merit considering how solid it is and how it has been the only real option for the Tokyo universe for rubyists beyond rolling your own. We’re using rufus-edo (which I think is a great name) on for raw speed.

    Thanks for making it and I kinda understand your decision.

    Chu Yeow

    February 26, 2010 at 1:41 am

  2. Thank you John for your work on rufus-tokyo. It has been the library that I used to explore Tokyo, implement my graduation project ( and as of recently we’re using it in production on for


    February 26, 2010 at 11:32 am

  3. I have nothing but respect for your attitude towards software development. It’s so refreshing to see someone who really cares more about software quality, cooperation and collegiality over recognition, competition and ego.

    Norman Clarke

    February 26, 2010 at 1:39 pm

  4. @Chu, @Marko @Norman many thanks guys, keep up the great work !

    John Mettraux

    February 26, 2010 at 2:35 pm

  5. John first off thanks for the kind words. Thank you for creating rufus-tokyo, your fantastic API has been my inspiration on ruby-tokyotyrant. I think you are leaving the Ruby+Tokyo community in good hands with @JEG2. I’ll try to connect with him to continue the compatibility effort between our projects. Good luck in the future, hope you keep in touch.



    February 26, 2010 at 2:57 pm

  6. Flinn,

    many thanks ! You’re doing a great job with ruby-tokyotyrant and the mailing list !
    I’m not going away.


    John Mettraux

    February 27, 2010 at 12:37 am

  7. Thanks a ton for your work on this. It really is a great library.

    Jason Seifer

    March 2, 2010 at 5:34 pm

Comments are closed.

%d bloggers like this: