rufus-tokyo 1.0.0

Just released rufus-tokyo 1.0.0

This is mostly a “cleanup” release (spec reorg, to_s enforcement on key and values, and dropped backward compatibility with older TC/TT releases). Hence the 1.0.0.

There are new features though :

* Matthew King helped me add process method for table queries.

* Jeremy Hinegardner did an initial contribution on the front FFI / Tokyo Dystopia.

* Adam Keys contributed the ext method for the table structures (sorry, I had forgotten).

* full text search operators for table queries (:ftsphrase, :ftsec, :ftsor: ftsand) and the accompanying :token, :qgram and :opt inverted index types. I haven’t played too much with them. Feedback is welcome.

For those of you interested in extending Tokyo Tyrant via Lua functions, I warmly recommend Ilya Grigork’s post about it.

Many thanks to all the people who helped.

There will be a next release soon, as I’m busy trying to catch up with Hirabayashi-san.

rufus-tokyo : http://github.com/jmettraux/rufus-tokyo/

(picture graciously from @michelledasilva)

 

rufus-tokyo 0.1.13

With the initial releases of rufus-tokyo, I happily cut corners and went with C strings (ending with NUL).

This isn’t optimal, often you need to store binary data as the value, the resulting ‘string’ contains NUL characters and values get truncated at restitution. I was working around that with Base64 encoding. Fine, the perf cost isn’t too heavy. But then, binary data was working fine on the Rufus::Edo side (wrapping the original tokyo{cabinet|tyrant}-ruby), why should Rufus::Tokyo be an exception ?

rufus-tokyo 0.1.13 which just got released fixes that issue.

Thanks to Ben Mabey for pointing out the issue.

Kamal Fariz Mahyuddin was kind enough to fork rufus-tokyo and add the #putkeep(k, v) method which only put if there was no previous value associated with the key. The collaboration with Kamal also prompted me to provide #add_int and #add_double for int/double counters in cabinet/tyrant.

http://github.com/jmettraux/rufus-tokyo

 

ruby to lua

With the help from Alain I just released rufus-lua 0.1.0.

This ruby gem talks to Lua via Ruby FFI. One could say it’s “embedding Lua”, as Lua is described as “Lua is a powerful, fast, lightweight, embeddable scripting language.”

I feel tempted by Lua, there’s the speed, the austerity and that javascript feeling.

 

Rufus-lua is not the first bridge between Lua and Ruby, there are also :

* http://rubyluabridge.rubyforge.org/
* http://raa.ruby-lang.org/project/ruby-lua

They are worth a look, but I was a bit disappointed, they are not available via ‘sudo gem install …’.

Rufus-lua is available via Rubygems, but since it’s a FFI thing, it needs to talk to the dynamic library packaging of Lua. Well, it’s not available by default, there’s some work involved (but not too much).

Something great about FFI is that it works for Ruby 1.8.6, Ruby 1.9.1 and JRuby (1.1.6) (and will work soon for Rubinius, its original platform).

 

Once liblua.dylib is on your system, along with rufus-lua (sudo gem install rufus-lua), you can play with code like :

require 'rubygems'
require 'rufus/lua'

s = Rufus::Lua::State.new

t = s.eval(%{
  return { message = { 'hello', 'from', 'Lua' },
    funk = function (x) return 2^x end }
})

p t['message'].to_a # => ["hello", "from", "lua"]
p t['funk'].call(8) # => 256.0

s.close

Well, it’s only a 0.1.0 release…

 

As rufus-lua is using FFI it’s slower at the “frontier”, here is a naive benchmark for computing a Fibonacci suite, with Fiber (Ruby 1.9) and Coroutines on the Lua side :

require 'benchmark'
require 'rubygems'
require 'rufus/lua'

RUBYFIBS = Fiber.new do
  n1 = n2 = 1
  loop do
    Fiber.yield n1
    n1, n2 = n2, n1+n2
  end
end

s = %{
  co = coroutine.create(function ()
    local n1 = 1; local n2 = 1
    while true do
      coroutine.yield(n1)
      n1, n2 = n2, n1+n2
    end
  end)
  return co
}

LUA = Rufus::Lua::State.new
LUAFIBS = LUA.eval(s)

N = 10_000
Benchmark.benchmark(' ' * 31 + Benchmark::Tms::CAPTION, 31) do |b|
  b.report('ruby') do
    N.times { RUBYFIBS.resume }
  end
  b.report('lua via ruby') do
    N.times { LUAFIBS.resume }
  end
  b.report('lua') do
    LUA.eval("for i = 0, #{N} do coroutine.resume(co) end")
  end
end

LUA.close

For the original. Running it on my board yields something along the lines of :

$ ruby19 test/bm_fiber.rb
                  user     system      total        real
ruby          0.050000   0.010000   0.060000 (  0.052032)
lua via ruby  0.180000   0.000000   0.180000 (  0.195411)
lua           0.010000   0.000000   0.010000 (  0.006394)

Well, nothing spectacular, that just states that it’s better to not cross the border too often : the “lua via ruby” bench has Ruby resuming the Lua coroutine 10_000 times and it’s costly.

 

rufus-lua,
on github : http://github.com/jmettraux/rufus-lua/
rdoc : http://rufus.rubyforge.org/rufus-lua/
the rest of rufus : http://rufus.rubyforge.org

lua : http://www.lua.org
ffi : http://kenai.com/projects/ruby-ffi/

 

ruby-ffi, Tokyo Cabinet

I know, we have a Tyrant but I wanted to focus just on the Cabinet, staying on my side of the network interface.

Tokyo Cabinet is a great [C] library for managing DBM hashes, it’s very efficient, so efficient it’s one of the workhorses behind Mixi, the top japanese social network (damn me, I used the word ‘social’…)

I really wanted to use Tokyo Cabinet from Ruby, I could only find the a datamapper related gem, no ‘sudo gem install’ 1-stop-only option for me. Hirabayashi-san, the original author of Tokyo Cabinet wrote a Ruby lib, but it’s not packaged as a gem, and then I wanted something that looked like a ruby hash, enumerable included.

ruby-ffi is a ruby extension for binding ruby to dynamic libraries, it originated in the Rubinius project and is now available for MRI Ruby (the ‘ruby-ffi’ gem) and JRuby as well.

So, if you have the Tokyo Cabinet library installed on your system, you can venture into :

sudo gem install rufus-tokyo

(it should install its dependency ruby-ffi in the same run)

and then

  require 'rubygems'
  require 'rufus/tokyo'

  db = Rufus::Tokyo::Cabinet.new('data.tch')

  db['nada'] = 'surf'

  p db['nada'] # => 'surf'
  p db['lost'] # => nil

  5000.times { |i| db[i.to_s] = "x" }

  p db.inject { |r, (k, v)| k } # => '4999'

  db.close

will create/open the hash database in “data.tch” and store all its stuff in there. It does it via the dynamic library for Tokyo Cabinet that it looks up in either /opt/local/lib or /usr/local/lib (or another path specified by the environment variable TOKYO_CABINET_LIB).

OK, nothing exciting, just that it’s fast, handy, works on the major Ruby platforms just a ‘sudo gem install’ away.

It’s rather rudimentary for now, but with a bit time…

source : http://github.com/jmettraux/rufus-tokyo
rdoc : http://rufus.rubyforge.org/rufus-tokyo
rest of rufus : http://rufus.rubyforge.org/
mailing list : http://groups.google.com/group/rufus-ruby

Many thanks to the authors of Tokyo Cabinet and ruby-ffi !

Fujikawa-san on JRuby at Ninjava

I attended a talk by Kouichi Fujikawa at the June Ninjava meeting.

Was an excellent warm-up before the RubyKaigi 2008.

Sorry for the hashed post, “live blogging” before the post-talk beers.

– Fujikawa-san is working for a company developing finance derivatives trading systems
– had the chance to work with ThoughtWorks China
– went to JavaOne 2008
– met Thomas E Enebo, Charles Oliver Nutter, Nick Sieger, Ola Bini

– JRuby supported by Sun Microsystems (especially Tim Bray) and ThoughtWorks
– JRuby is ‘performant’
– good way to get Ruby into the enterprise

– small demo… (classical JFrame demo)

– leveraging JDBC for database support (many drivers, drivers with connection pooling)
– integration of Java libraries (new systems and legacy services)
– at JavaOne08, Ola Bini presented a SpellCheck java library used in a JRuby on Rails application
– Warbler : turns a Rails app into a Java .war
– JRuby Rack (included in Warbler) integration of the Rack Ruby (low level) web framework

– next some slides courtesy of Nick Sieger
— may JRuby interpreters in 1 JVM
— Glassfish [web] app server
— servlet context initialization creates JRuby runtimes and pool them (acquire/dispatch/release)

– more information :
— http://jruby.org
— book by Ola Bini “Practical JRuby on Rails”
— http://d.hatena.ne.jp/fujibee (japanese, about JRuby and Haskell)

– then questions from the audience :
— Zev asking if threads do really magically make things better (more scalable), answer by Lars was mainly about JRuby being able to leverage ‘native’ threads thus multicore
— questions about JRuby and ObjectSpace (various tries between jirb and irb do follow)

The talk by Fujikawa-san was followed by a presentation of iKnow! by Zev Blut :

– “developing and scaling iKnow!”
– (stopping here as he will present this talk on Sunday at the RubyKaigi)

webdavhandler.rb on jruby 1.1[.1]

There is this hidden gem in Ruby land, the webrick webdav handler.

I’ve been using it to power some webdav accesses to an application here. When the time arrived to upgrade from JRuby 1.1 “pre from last fall” to JRuby 1.1[.1], this webdav handler broke.

jmettraux@sonora:~/tmp$ ~/jruby-1.1.1/bin/jruby dav.rb
[2008-04-24 10:09:24] INFO  WEBrick 1.3.1
[2008-04-24 10:09:24] INFO  ruby 1.8.6 (2008-04-22) [java]
[2008-04-24 10:09:24] INFO  WEBrick::HTTPServer#start: pid=3998 port=10080
[2008-04-24 10:09:37] ERROR ArgumentError: wrong number of arguments (4 for 3)
        /home/jmettraux/tmp/./webdavhandler.rb:135:in `chk_utf8'
        /home/jmettraux/tmp/./webdavhandler.rb:135:in `detect'
...

Two small changes to webdavhandler.rb fixed this issue and a subsequent issue.

The updated webdavhandler.rb is available.

on the JRuby front

Nicolas has been tracking JRuby and testing OpenWFEru on this platform. He had submitted a few patches that turned “OpenWFEru on JRuby” from a big NO to a firm Yes. A business process engine is quite a demanding beast.

This week, JRuby seems to have made big progresses (we test with JRuby trunk, JRuby 1.0.1 is a “no go”), OpenWFEru almost runs perfectly on it (except for a few baroque test cases). The performance are good (slightly lesser).

See Nicolas’ post for the full details.

Time to think about giving true flexibility to some business processes, currently hard-coded in Java here at work.