Friday, April 11, 2008

Iliad iRex note taking and hand-writing recognition

I recently bought an Iliad iRex, a pretty awesome eBook reader. Among the cool features, it's running linux, an sdk is available, and it's really easy to hack stuff for it (for instance I wrote a simple script for downloading the 24h edition of The Guardian). Also, as shown with the previous link, the community is quite active :)

Just a couple of words about the iliad itself... the hardware is pretty awesome, with wifi, ethernet, wacom tablet, usb, mmc card and compactflash, audio jack...
the e-ink display is quite amazing too -- 768x1024 makes it precise enough to be able to read A4 PDF without too much problem (a great thing to review lots of research papers, believe me !).

The software side on the other hand... is a bit disappointing. Don't get me wrong: it's good enough, and some aspects are pretty cool. But you really unlock the possibilities of the devices by getting the root access and adding applications developed by the community (notably, the PDF viewer hacked by the community is fantastic, with gestures, etc.). Which means it's fine if you are a geek and not afraid to hack your device, but more annoying for your average consumer :-/
The other disappointing aspect is the (comparatively) low battery life: about 12-15 hours depending on the model (mine is a v1, the v2 do better), which is mostly caused by the fact that no sleep mode is available. Other eBook readers perform much better on that metric. To be fair they don't have wifi nor a wacom tablet :D ... so it's really a matter of choice.

All in all, it's a bit of a shame as really the platform is very nice, and with a bit more effort on the software side, Iliad would have a killer product on their hands. Oh, and yes the e-ink display refresh rate is slow, but curiously it's not that annoying, and having its full library in such small factor is absolutely fantastic.

Anyway... one of the really, really cool feature of the iliad is the presence of a stylus (i.e. the iliad display sports a wacom tablet), which allows you to annotate PDFs, take notes, etc.

I started to experiment a bit with the note taking feature of the iliad; the idea is that you can open a PNG image in the notes folder, and a copy will automatically be made where you can write on it (the image is being used as a background, so it's trivial to have customized backgrounds). But I then wanted to generate a PDF from those notes (i.e. combining the scribbles+PNG).

Iliad do provide a windows application to do all that, but it's a windows app, not really useful for me... there is a nice java application written by the community that allows merging scribbles with the PDF as well. Alas, the java scribble merging application only seems to work for PDF scribble; I guess it would be trivial to modify the java app, but I had a look at the xml scribbling format, and I saw that the format was really simple.

So I quickly wrote myself a MacOSX viewer for the notes, using the png image as a background, letting me print notes easily or convert them to PDF.

But then... I suddenly remembered the Ink handwriting recognition engine.

This thing comes straight from the ill-fated Apple Newton PDA (such a loss!), but what is nice is that it is available and installed by default on OSX.

Turns out it's not too difficult to feed Ink a set of custom datapoints, and after some tweaking it doesn't work too bad apparently, as can be seen on the screenshot...





So far, this is only highly experimental code, and it'll probably take a bit more time to have a really usable application. Still, pretty cool!

8 comments:

Wes said...

That's really great. I didn't realise the hand writing recognition in Mac OS X was accessible through APIs.

Nicolas said...

Only through Carbon sadly. But it's not really hard to use that in a Cocoa application. Now if I could find a neat HWR engine for linux...

Anonymous said...

Thank you this is invaluable. Just bought one, waiting for it to arrive. Mac user (OS 10.5) but non-techy, so frustrated by lack of native apps. I expect to be taking a lot of notes on this thing, have downloaded that Java app and will try to get to grips with it- will you be releasing what you put together for it? Are there any Mac/Iliad apps fora you know of? Many thanks again.

Nicolas said...

There's the mobile read forum, which is quite nice. Plus the forums on the Iliad website.

I will release the app but I'm quite busy for the next few weeks. Ideally I'd like to implement the segmentation described in this paper (I added some code building the link model, but that's all for now).

Anonymous said...

Any news / updates?
Thanks in advance.

Jan

Mara Uvadio said...

would you give me the tool's name?
Thanks

Piero Campanelli said...

Have you tried reading A4 technical ebooks? (es. APRESS, OReilly, ...). ??

Nicolas said...

Piero: I didn't try specifically those ones, but yes, I read quite a few A4 PDFs. With the reader's gestures, it's trivial to crop the margins on a document, so even in full page it's readable (though that depends on the person I guess -- the text is not very large, but readable). Else there is still the possibility to read things in half/page (rotating the page).

For complex A4 layouts though, I guess the recent "iRex Digital Reader 1000" will be a better fit, as the screen is larger and the resolution better. But damn are they expensive!