retiring rufus-tokyo

As you probably know, rufus-tokyo is a Ruby FFI wrapper for Tokyo Cabinet|Tyrant, the fine pieces of software delivered by Hirabayashi Mikio.

Rufus-tokyo is 12 or 13 months old, but it’s time to retire it (maintenance mode).

James Edward Gray II is building his Oklahoma Mixer which will, hopefully, completely overlap rufus-tokyo and simply be better, very soon. Once the mixer covers Ruby 1.9.x, JRuby and Tokyo Tyrant (see todo list), rufus-tokyo will be irrelevant.

This is great for me (and for you), we will get a better Ruby library for TC/TT. Please note that for Tokyo Tyrant we already have a very fast option with Flinn Mueller’s ruby-tokyotyrant, which will always be faster than a FFI Tokyo Tyrant library.

James is exposing his motivations for Oklahoma Mixer on its front page. I have a few clarifications to make about them and rufus-tokyo. I use this blog to make these clarifications more accessible.

Why not just use rufus-tokyo?

There is already a Ruby FFI interface to Tokyo Cabinet and more called rufus-tokyo. I am a huge fan of rufus-tokyo and have contributed to that project. I have learned a ton from working with rufus-tokyo and that code was my inspiration for this library.

That said, I did want to change some things that would require big changes in rufus-tokyo. Here are the things I plan to do differently with Oklahoma Mixer:

* Tokyo Cabinet’s B+Tree Database has some neat features, like custom ordering functions, that are hard to expose through rufus-tokyo due to the way it is designed. I would have had to rewrite pretty much that entire section of the library anyway to add these features.
* There are some places where rufus-tokyo uses more memory than it absolutely must or slows itself down a bit with extra iterations of the data. Again, this is a result of how it is designed. It allows more code reuse at the cost of some efficiency. I wanted to improve performance in those areas.
* I’m working on some higher level abstractions for Tokyo Cabinet that I eventually plan to include in this library. These extra tools are the reason I needed to make these changes and additions.
* Finally, rufus-tokyo paved the way to a Ruby-like interface for Tokyo Cabinet. Previous choices were scary in comparison. I wanted to push that movement even further though and get an even more Hash- and File-like interface, where possible.

It’s important to note though that rufus-tokyo does do some things better and it always will. Its advantages are:

* It offers a nice Ruby interface over a raw C extension which is faster than using FFI. I have no intention of duplicating that interface, so rufus-tokyo will remain the right choice for raw speed when using MRI.
* For now, rufus-tokyo is more full featured. It provides interfaces for Tokyo Tyrant and Tokyo Dystopia. I would like to add these eventually, but I’m starting with just Tokyo Cabinet.
* It offers a pure Ruby interface for communicating with Tokyo Tyrant without needing the C libraries installed. I won’t be copying that either, so it will remain the right choice for a no dependency install.
* For now, it’s more mature. A lot of developers have used it and contributed to it. It’s probably the safer library to trust for production applications at this time.

The “raw C extension” mentioned is provided by Hirabayashi-san. Rufus-tokyo only provides a (mostly) unified interface to it (same interface for FFI and the C exts).

The “pure ruby interface for communicating with Tokyo Tyrant” again is provided by Hirabayashi-san.

That doesn’t leave me much merit. All this wrapping effort was code-named Rufus-Edo and you can read more about the motivations and mechanisms in ‘rufus-tokyo 0.1.9 (Rufus::Edo)’. You will also find more details in my Edo Cabinet presentation.

So I’d advise TC/TT Rubyists to support the effort of James and Flinn. For those of you who want to participate in the larger Tokyo Cabinet|Tyrant community, Flinn has set up the Tokyo Cabinet Users mailing list, where lots of people help and share.

Rufus-tokyo is now in maintenance mode. I will still help people with the occasional bug or mem leak. The mailing list is still open (it’s the one for my rufus libraries) and if you know how to write an issue report, you will always find help (random tweets do not count).

Thanks for the fun on the way.