Getting lxml 2 into my gnarly old Tiger; going on ROA

In case anyone might google for something like:

lxml "Symbol not found" _xmlSchematronNewParserCtxt

, I'd like to reiterate the steps I just took to get lxml 2 (2.1 beta 1 to be precise) up and running on OS X 10.4 (which I haven't yet purged in favour of the shiny Leopard for reasons of fear, uncertainty and doubt. And while pleasurable, quite mixed impressions from that cat when running my wicked new iMac (and my funky old iBook, in case you're dying to know)).

In short: you need a fresh MacPorts (1.6, I had 1.3.something):

$ sudo port selfupdate

, and then, wipe your current libxml2 (along with any dependent libraries, MacPorts will tell you which):

$ sudo port uninstall libxml2@2.6.23_0

(You might not need the version number; I had tried to get the new libxml2 before this so there was an Activating libxml2 2.6.31_0 failed: Image error: Another version of this port thingy going on.) Then just:

$ sudo port install libxml2 libxslt

and you'll be ready to do the final (after removing any misbehaving lxml (compiled with the old libxml2) from site-packages) :

$ sudo easy_install lxml

But why?

Because lxml 2 is a marvellous thing for churning through XML and HTML with Python. There was always XPath and XSLT, C14N in lxml 1.x too (admittedly also in 4Suite as well; Python has had strong XML support for many, many years). But in lxml 2, you also get:

• CSS selectors
• much improved "bad HTML" support (including BeautifulSoup integration)
• a relaxed comparison mechanism for XML and HTML in doctests!

And more, I'm sure.

So, why all this Yak Shaving tonight? Just this afternoon I put together an lxml and httplib2-based "REST test" thing. Doesn't sound too exciting? I think it may be, since I use it with the old but still insanely modern doctest module, namely its support for executing plain text documents as full-fledged tests. This gives the possibility (from a tiny amount of code), to run plain text specifications for ROA-apps with a single command:

Set up namespaces:

 >>> xmlns("http://www.w3.org/1999/xhtml")
 >>> xmlns(a="http://www.w3.org/2005/Atom")

We have a feed:

 >>> http_get("/feed",
 ...     xpath("/a:feed/a:link[@rel='next-archive']"))
 ...
 200 OK
 Content-Type: application/atom+xml;feed
 [XPath 1 matched]

Retrieving a document entry with content-negotiation:

 >>> http_get("/publ/sfs/1999:175",
 ...         headers={"Accepts": "application/pdf"},
 ...         follow=True)
 ...
 303 See Other
 Location: /publ/sfs/1999:175/pdf,sv
 [Following..]
 200 OK
 Content-Type: application/pdf
 Content-Language: sv
 [Body]

That's the early design at least. Note that the above is the test. Doctest checks the actual output and complains if it differs. Declarative and simple. (And done before, of course.)

Ok, I was a bit overambitious with all the "ROA"; this is mostly for checking your content-negotiation, redirects and so on. But I think it'll be useful for a large system I'm putting together, which will (if things work out well) at its core have a well-behaved Atom-based resource service loaded with (linked) RDF payloads. Complete with feed archiving and tombstones, it will in turn be used as fodder for a SPAQRL service. (Seriously; it's almost "SOA Orchestration Done Right", if such a thing can be done.) But this I'll get back too..

(.. I may call the practice "COURT" — Crafting Organization Using Resources over Time. It's about using Atom Entries as manifests of web resources with multiple representations, fully described by RDF and working as an over-time serialized content repository update stream. (Well, that's basically what Atom is/works perfectly for, so I'm by no means inventing grand things here — grand as they IMHO are.)

Old Bad Breaks Improved by Dividing

I stopped blogging quite a while ago. Partly because I cooked up long articles about Atom-based services with RDF payloads in my head that never got written down (I shall get back to this). Partly because I use this "convert newlines to breaks" in blogspot. If I turned it off, I hadto edit all my old posts. Simple in itself, but I couldn't bear touching those old "modified" times.

So all my posts became these horribly formatted blocks of old, stale, ill-parsable html.

I don't care anymore. I'll use this to post random stuff, and put proper articles (if any) at Neverspace.net or Oort.to. And in a couple of decades from now, move my blogging to a a construct of my own, full of Atom-based services with RDF payloads.

Oh. Just-in-Time Update: Switching to edit-mode in the blogspot editor I just realize that there are no horrible little bloody breaks anymore. They are div:s now! Perhaps they could even become p:s. Nice! Thank you blogspot/google people! Then I don't have to move, since apart from RDF payloads, all Google services are indeed Atom-based (RSS 2.0 too, but I don't care for that).

(Thus I shall "soon" attempt to post from Vim to here, possibly with the python-based GData stuff I haven't been using for anything productive yet. Or some of the other TODO:s that clog my creative pipe far too often.)

Now for another post, possibly with some useful content in it as well.

Out-of-Sync Update: Typically, those pesky breaks still show up. Well, at least they are funnily isolated in div:s too..

Out To Transmogrify Resource Descriptions

Finally, I've bundled another release of Oort. And OortPub! I've modularized the package into two Eggs, separating the core O/R things from the (experimental) WSGI web stuff.

At the cheeseshop you'll now find them both, at

• Oort (0.4), and
• OortPub (0.1)

respectively. I did this not the least since the core Oort package is very useful on its own, in conjunction with other tools altogether. I've used it for a lot of stuff lately, none of which involved OortPub. Check out e.g. the QueryContext for some new features..

I hope this will become useful to others as well.

The website got a little overhaul as well, to reflect this change, and to include some more documentation (not so much more yet, but the basic stuff is in place). Oh, by the way, it's generated using Oort objects (among other things like Genshi, ReStructuredText, google-code-prettify and of course RDFLib).

OMG! Snakes on a Phone!

So, I got this new cell phone from work. A Nokia 6120 Classic, which has just the size and features I wanted (and more, I'm sure). I picked the white/silver model, to go with my kitchen supplies (in honour of the fact that with Series 60, you can basically install the Kitchen Sink if you so choose to).

Going further, I of course wanted to play Monkey Island and such on the amazing ScummVM. Well, admire the running of; actually playing through them might be a little straining.. Though great screen clarity makes it quite possible to discern the orange and pink on the red herring you steal from the Loom™ seagull outside the Scumm Bar kitchen, it's quite thumb-numbing to actually snatch it in time..

And then, surely, I couldn't wait to get to the real goodies: Python. Specifically, The Python for Series 60 distro by Nokia (open source). I knew it was quite capable, but seriously; it basically lets you p0wn your phone.

(Half a decade ago I had fun running Python on my old Psion Revo; but that didn't go much further than some file tinkering and exploration of new-style classes (metaprogramming and whatnot) during commuting. But this is different. The same. But different.)

You go ahead by following the instructions, basically downloading the PythonForS60_1_4_0_3rdEd.sis and PythonScriptShell_1_4_0_3rdEd.sis from sourceforge, wiring or beaming them to the phone and running them to install (py, then shell). I also got the PythonForS60_1_4_0_doc.pdf, and marvelled at its beauty. I mean, apart from greater parts of the Python Standard Library, a mere glance at the modules provided for S60 stuff gives you an impression of the power at hand:

graphics, e32db, messaging, inbox, location, sysinfo, camera, audio, telephone, calendar, contacts, keycapture, topwindow, gles, glcanvas

Now, I haven't gotten very far yet; mostly tinkered with urllib to download some stuff from the web (which just worked (after a nice confirmation dialogue from the phone to actually access the net)), rendering the obligatory mandelbrot and then peeking at the real deal by looking at the SMS inbox contents using the interactive console.

Or actually, using

$ screen /dev/tty.bt_console

from my OS X Terminal (iTerm really). Magically set up by simply following this excellent guide.

Perhaps screen scraping wasn't really what I thought of when wanting Python on the phone, but you can't deny it's cool to be able to (perhaps awaiting flatrate subscription to not drain my wallet in the process though).

Speaking of screens and webs again; damn that WebKit-based browser in the phone really worked! It actually ran the JavaScript-based SyntaxHighlighter, thus rendering the Oort Tutorial quite faithfully. I did not expect that (nor my rekindled desire for highly variable width web sites..).

[And speaking of Oort; I've added some somewhat nifty stuff (if I may say so) lately; accessible in the svn repo at google code for now. A release should be coming up. Pre- or post my planned split into core and web/publication stuff I haven't decided yet. There really is a working RDF-to-object mapper underneath the touted WSGI stuff (which is just a minimal layer).]

Now I should go on to do something productive (like shaving a yak).

A Mind must Become be4 it can Go

Now, I do not understand why the Web 4.0 page at Wikipedia has been protected to prevent creation. It is the fear of the machine? Because as everyone must know, Web 4.0 is the peak of our civilization, when mankind will unite in celebration, marvelling at our own magnificence as we give birth to AI. The Web at 4.0 will also affectionately be known as "Web H4L", to honor its predecessor, born on January 12, 1997 at the HAL Plant in Urbana, Illinois.

Of wheels and fortunes

[I wrote this comment a while ago, but left it hanging. Rather than letting it die, I just throw it into the mix (to show a life sign if nothing more).]

I was inspired by this post about reinvention (stumbled upon it, probably via reddit). An interesting point, addressing many states of affairs of the past, present and future.

To me "gratuitous reinvention" is a "mixed curse". In general much can be lost by the reinvention of wheels. But it is definitely healthy with competing ideas, problem formulations and solutions. In times of surplus, the risk is unfortunately quite big to overlook the fact that thorough thought processes have already addressed situations that seem novel when first encountered. When this leads to incompatible diversification (as opposed to interchangeable alternatives), much is devalued. This is something one should always be aware of, both when choosing among existing solutions and embarking on own adventures of invention. With that in mind though, quests of the latter type may very well lead to a specification of what needs to be solved, and how. At that point, (re)visiting existing work is a wise course.

Keeping it simple (enough but not more) is a fine ideal. But keeping it integrable is also very important for the evolution of infrastructure — where complexity can evolve without necessarily leaking down into the component parts.

Choosing right tools for jobs reasonably means taking paths that lead to a minimizing of energy expenditure — over time. Time is where the real trickiness comes in. Sometimes, when your intuition tells you to, you have to make leaps of faith. This is true in life as well as in art. Whatever those two things are. Wheels and fortunes.

Point of Data

(Read all of this, it is a Zen Koan.)

Been thinking about data lately (possibly "in" as well). And information, content, metadata, meaning, knowledge, taxonomies, ontologies.. Ontology. Flashbacks of never-ending philosophical debates, epistemology, Plato's Ideals, empiricism, positivism, all of that. Well not so much of the latter really, I guess I've learnt to recognize mental swamps before I go trekking nowadays.

But data. I think a little clarification could be in order. I'd say it goes like:

Data

Particulars (atoms if you will) of that which compose our impressions. Not the world, but that stuff from which we "get the world".

Information

I just quote Gregory Bateson: "a difference that makes a difference". The part of data we can use, as opposed to "noise" or "void".

Content

A tricky thing. "Composited information with a bound context" perhaps, with varying (often hard to measure) complexity.

Metadata

Let's say "added bits used to externally correlate content". I'm not much of a fan of the word anymore. I may deprecate it in favour of just "context-providing statements". The stuff that turns data we don't get into data we get.

I often just call content and metadata "data in" and "data about" nowadays.

Of Content

Content is perhaps the most widely used and less defined stuff. It's abundant, the substance which we structure (and in so doing "contentifying" it further). It is the composition from which richer meaning can be derived. By it's virtue of having a context. This article is "content". That last statement is information (this was a reification, but I digress). Possibly "metadata". Now it's just getting funny. Anyway, content is stuff which can be molded to gain shapes and shades, color and tone; somewhat "synergetical" effects which may or may not add meaning — more often than not depending on cultural aspects (part of the implicit context).

It is stuff which we leave semi-formal at best, the information we hardly can process by anything less than our own neural networks (brains). Somewhat pessimistically perhaps, information which during interpretation gain illusory qualities (akin to optical illusions) which either enhance or corrupt the bringing of meaning.

I won't get into information theory nor semantical discussions. Neither into a discussion of techniques of how to "fluff" expressions to become more sympathetical, making the receiver prone to interpret them as "meaningful".

To the Point

If you're still reading, this is the part where I intend to make a point. Binding the context by which this becomes meaningful.

This content bears no inherent meaning. It's all in your head. Perchance it may have conveyed a structure which made sense, put your mind into a state of "ah, ok". I really just needed to externalize that first part of categorizing some terms that continue to come up in discussions, and for which I needed some binding interpretations.

Perhaps you detected an "anomaly" if you tried to interpret the terms in a hierarchical fashion. "Data in" and "data of" was my differentiation of Content and Metadata, seemingly "instances of Information". But Data isn't necessarily Information. The thing is, that "thing that makes the difference" is where the "meaning", "semantic relevance", "precise form" comes in, and what that is my current state of mind can only grasp by intuition.

We use Content, and the more fine-grained and "to the point" context-providing statements (Metadata), one as compositions within a context, the other to bind both particulars and contexts by means of relations and characteristics. To have a feel for the difference is important, for it is the key by which to understand why Knowledge Representation is the missing piece in many Information Technology issues today.

And this is where How To Tell Stuff To a Computer, and then The Semantic Web FAQ comes in, as my recommended reading of the week, to get you going. Those sources of information will hopefully make the difference that eluded you here.

That's the point.

(Or is it?)