OpenCore Software

This past Tuesday I attended the NYC Python User’s Group meeting at the offices of DayLife.

The presentation this week was by Peter Fein about GrassyKnoll, a text search engine written in Python.

From TOPP’s point of view there are several interesting things about it:

• It could provide us with a sitewide search solution for openplans.org and livablestreets.com.
  
• It can use any of a number of pluggable back ends (notably PyLucene).
  
• You interact with it entirely via a simple REST API.
  
• A GrassyKnoll client can also trivially be a GrassyKnoll server. (Peter gave a live demo of this.) In theory this could allow for fun things like smart clustering, where one server gets your query and dispatches queries to N other servers, and then merges the results appropriately. The end client still sees the same simple rest API.
• It’s written entirely in Python and is Free.
• They’re looking for more people to help out and would love for us to get involved.

The major down side is that they’re still some months away from a production-ready release. (There were a few glitches in the live demo.) When I pressed him about this, Peter said “definitely by the end of the year.” I got the impression that more hands helping would speed that up.

Also, it doesn’t define any kind of common query language; it just passes them along to the back end. So you do have to know what you’re really talking to.

Peter said he was hopefully coming to the next NYCPUG meeting that we’re hosting here at TOPP on July 15.

So, Zope’s handling of XML page templates is kind of broken.

As a result, all of opencore’s page templates are parsed in ZPT’s “html mode”. This makes for some surprising validation behavior (e.g. unclosed tags is not an error, but putting content in a “link” tag is an error… which sucks if you happen to be trying to generate georss and not HTML …)

I ended up not even using ZPT for my georss feed, but for posterity, there are at least two possible workarounds:

1. Put an xml declaration at the top of your template (this makes the parser to use XML mode). If  your template is only for a snippet and not for a whole document, you don’t want the declaration showing up in the middle of the final document. To avoid that, use METAL macros.  It’s a bit tricky to do this from python, so we have a handy render_tal function in opencore.browser.tal.  And you’ll need to manually encode to your output encoding if you want to use this with response.write().  An example of this that I later reverted can be seen here.

2. You could, if you don’t want to use macros and can’t have the xml declaration in your output, instead subclass Five’s ZopeTwoPageTemplateFile like so to force the TAL parser to use the right mode. (Thi sis also in the example linked above.)

 class XmlPageTemplateFile(ZopeTwoPageTemplateFile):

    """This class exists because the process whereby PageTemplateFile
    decides whether to use HTML or XML mode for parsing and validation
    is completely stupid.
    """

    content_type = 'text/xml'

    def _cook_check(self):
        # In PageTemplateFile, this gets called potentially multiple
        # times (if you're in dev mode), sniffs the file's content,
        # and if it doesn't start with an XML declaration, uses HTML
        # mode.  We don't want that.  So here, we duplicate the same
        # code without the sniffing.
        if self._v_last_read and not DevelopmentMode:
            return
        __traceback_info__ = self.filename
        try:
            mtime = os.path.getmtime(self.filename)
        except OSError:
            mtime = 0
        if self._v_program is not None and mtime == self._v_last_read:
            return
        t = self.content_type  # Take my word for it, dammit!
        f = open(self.filename, "rb")
        if t != "text/xml":
            # For HTML, we really want the file read in text mode:
            f.close()
            f = open(self.filename, 'U')
            text = ''
        text = f.read()
        f.close()
        self.pt_edit(text, t)
        self._cook()
        if self._v_errors:
            logger.error('Error in template: %s' % 'n'.join(self._v_errors))
            return
        self._v_last_read = mtime

For my georss feed, I ended up deciding this was all too much bother and I just went back to using Python string interpolation. There just wasn’t enough to the “template” to justify the pain of getting TAL to work properly under these constraints.

This showed up on my radar today. It’s a cute little hack, if a bit rough. (I’ve already posted a couple patches).

It could easily be generalized, there’s nothing really zope-specific about it, all it really cares about is a path to a common-apache-format access log and the PID of a process to watch memory. The result is a nice little Flot graph where you can select a range and see what URLs were hit during that time.

Here’s a screenshot of an example from a flunc run against opencore (the “basic” suite):

So what does this tell us really? I’m not sure. Multiple runs show that the graph generally starts around 400 MB and ends around 500 MB, but what happens between does not seem consistent. So I’m not sure if this little app is really that useful for opencore.

One of the things that’s plagued our buildbot config for a while is how to know when the web servers are really up and ready to take requests. For a while, I had buildbot configured to just sleep for a while and then start running flunc. This would eventually fail when the box was under heavy load; the flunc tests would start when things weren’t ready . I suppose I could keep doubling the sleep time, but I wanted something more meaningful.

I finally came up with a hack that seems to be reliable - use wget to actually request a page from the app and block until it returns or times out. Something like this:

fac.addStep(ShellCommand,
            command=('%s/bin/supervisord && sleep 10 && wget '
                     '--retry-connrefused --tries=10 -T 100 --spider '
                     'http://localhost:%d' % (OCBASEDIR, ports[1])),
            description=['start services', 'for functional tests'],
            haltOnFailure=True)

The full config (or rather, a template for it) can be seen in our source control here.

Another problem was that the services sometimes didn’t get shut down properly. I hacked a bit on jeff’s very handy portutils package until it seemed to do the trick.

1. If you are running python that was not compiled with readline (e.g. Mac OS X system Python), you may first have to run the following:

sudo easy_install -f http://ipython.scipy.org/dist/ readline 

2. Create a file in your home directory named .pythonrc.py
3. Add the following lines to it:

import readline, rlcompleter
readline.parse_and_bind('tab: complete')

4. Set your PYTHONSTARTUP environment variable to point to this file on shell startup, e.g.:

$ echo 'PYTHONSTARTUP="$HOME/.pythonrc.py"' >> .bashrc

5. Prosper.

1200 male geeks in one windowless conference center reminded me a bit of engineering school at the worst of times. I’m sort of kidding here Really Pycon was a great bunch of people with a lot of talent and a lot of passion for what they are doing.

One talk that was interesting was “What Zope 2 did wrong.” It really shed some light on the weakness (and strengths) of Zope 2 and also suggested characteristics to look for when choosing among web frameworks.

I think the talk that most moved me was Ivan’s talk on his recent trips bringing the OLPC to schoolchildren in Peru and Uruguay. There, I could really see how technology was improving people’s lives, and such cute and innocent people at that. That Ivan chap turned out to be quite a character I later gathered at Ian’s party. He’s got a reputation for being able to solve problems that are unsolvable. Very impressive!

This inspired me to sprint with the OLPC bunch. I first got Ubuntu up and running in VMWare on my system and was able to install the sugar emulator, that’s the OLPC’s GUI. To help out with the procedure for porting python apps to the XO (the OLPC laptop), I ported the great game of Asteroids. This, no doubt, will come in handy the next time Peruvian children get lost in an asteroid belt armed only with forward phasers and a bomb. Actually, I really do have doubts as to whether Asteroids was a great thing to export to the world’s children. I also wonder about digital games in general, however, I do like that I was able to improve the porting process somewhat for the OLPC bunch.

So Pycon was fun. One thing I really enjoyed was just being able to hang out with our bunch of TOPPers in a different environment. I really do appreciate being able to work with such great (and a little eccentric) people.

pycon was really fun.  It was the first time I’ve gotten to see a good sampling of the python community all in one place and it was a good experience to see the diversity of the pythonistas, doing all sorts of things (though web programming and scientific use seemed the most prominent, at least from my vantage point). 

The talks were pretty cool.  I especially liked Guido’s talk about py3k.  I guess if I had been following more closely, none of what he said would have been much a surprise, but for someone that does not have as much time to pay attention to the vast world that is python the talks were good updates to many things that I’ve had half an eye on but not my full attention.

 I liked the lightning talks better than the regular talks.  Many of them were the obligatory corporate “we are hiring” talks, but some of them were pure jewels.  After seeing how not a big deal they were the first day, I signed up for one on Saturday which ended up happening on Sunday basically plugging http://bitsyblog.biz/ .  Being five minutes and completely unrehearsed, I didn’t get much to “how bitsyblog works”, but probably more importantly I covered “why bitsyblog?” pretty well IMHO.

Then came Ian’s party.  I probably learned more and talked about python at a much higher level than at the conference.  I met plenty of folks during the conference, and it was good to meet them, but at the party people were, um, loose enough to really say their minds.  (Spare me abuse regarding “geeks” and “parties”) There was much (not undeserved) ripping on the rails community, and more high level talk about how python works, why python is awesome, and what could be better.  Most of it transcended “just” python.  Okay, it was a party.  No world-breaking programs will come out of it, but it is good to put things in perspective now and then.

 Then came the sprinting.  I didn’t do as well as at the snow sprint, partially because it was over as soon as it began (or so it seemed).  I didn’t get up until late Monday, then I marched ahead until 4a getting bistyblog to 1.0 until only me, a redhead, and some other guy were in a room.  Tuesday I played with Mel and Ian’s commenting webob middleware, but other than that was thoroughly unproductive (I ended up spending the afternoon starting at a screen waiting for a brain reboot).  I wish I had done the repoze sprint, or maybe got bitsyblog packaged for the OLPC.  ::sigh::

 And then it was over.  Bye, PyCon!  Maybe I’ll see you next year!

During the Pycon sprint I ended up writing some nasty tests for a simple feature.

Let’s say you’re writing a test for a class like this:

class Foo:
    def foo():
        if os.environ['FOO'] == 'bar':
            ...
        elif os.name != 'posix':
            ...
        elif somemodule.some_function():
            ...

It’s the usual isolation problem that leads people to using mocks and stubs. What’s bothering me is the need to replace those imported names (os.environ, os.name, etc.)  for the duration of the test, and cleanly restore them after the test.

One approach, which I used in the patch I submitted for that feature,  is to monkeypatch during test setup, replacing the problematic values and functions; and then restore the originals during teardown.

This just smells bad to me; it stinks in proportion to the number of things you monkeypatch. I ended up with four, which is plenty of stink.  Why? If your setup and teardown code drift out of sync, or your teardown raises an exception before it finishes, or or or … you get very obscure failures in unrelated parts of your system.

My crazy idea is this: What if there were a way to deep-copy the global namespace seen by the object under test, and inject things into this fake globals copy, without modifying the real globals? That would be totally safe: other instances and classes would see the world as usual, and when the test exits, the copy would just get thrown away.

But I think it’s just not possible. Oh well.

Off to France! I’ll be back at TOPP on 3/27.

We’re at PyCon! The lion’s share of the openplans / opencore developers are here:

Ian Bicking, Josh Bronson,  Mel Chua, Jeff Hammel, Anil Makhijani, Robert Marianski, Doug Mayle, Whit Morris, and me (Paul Winkler). Look for us and say hi!  We are friendly and mostly harmless. Oh, and we might have a job or two open.

I’m giving a talk Friday at 3:55, and Ian’s doing one on Sunday at 11 AM

I learned something useful on the zope-dev mailing list today… Gary Poster said this:

For future reference, if you put the set_trace on the same line,
you *can* use pdb.  For instance, this would let you walk into
the "do_foo" function.

    >>> import pdb; pdb.set_trace(); do_foo()