a, (b, c) = d, e = 1, (2, 3)

This would get translated to:

LOAD_CONST 1
LOAD_CONST (2, 3)
BUILD_TUPLE 2 # builds the tuple (1, (2, 3)
DUP_TOP       # duplicates the top item on the stack, since
              # it's going to be assigned to two targets
UNPACK_SEQUENCE 2 #now, push each element of the tuple we just
                  #built onto the stack
STORE_FAST a
UNPACK_SEQUENCE 2 #unpack (2, 3)
STORE_FAST b
STORE_FAST c
UNPACK_SEQUENCE 2 #unpack the duplicate
STORE_FAST d
STORE_FAST e

This is stupid, because it does a bunch of packing and unpacking of tuples. Tom Lee’s patch improves the situation by recognizing (1, (2, 3)) as a constant, and storing it in the constants table so that it does not need to be created on the fly. In other words, it replaces the first three operations of the above with:

LOAD_CONST (1, (2, 3))

This is much faster, but it’s still not optimally fast. The above Python code is equivalent to:

a = 1
d = 1
b, c = e = 2, 3

This produces the following bytecode:

LOAD_CONST 1
STORE_FAST a
LOAD_CONST 1
STORE_FAST b
LOAD_CONST (2, 3)
DUP_TOP
UNPACK_SEQUENCE 2
STORE_FAST b
STORE_FAST c
STORE_FAST e

My latest patch to the optimizer does this conevrsion. At a slight space cost, the sequence could be further reduced by replacing the DUP_TOP UNPACK_SEQUENCE with a LOAD_CONST for each element. I decided not to do this, because not everyone will want to make time-space tradeoffs, and I’m not 100% sure it would be faster.

This came up because I had an error similar to the one in  this thread, while trying to reload rolemap.xml to add my permission to a Plone site via GenericSetup; it wouldn’t load until I restored a commented-out view declaration that used the permission.

Now you know. I found it surprising.

From a designer’s POV, the most important files are templates, CSS, and javascript.  They are the three pillars of web design.  The sun rises and sets with them.  They are best friends.  So, ideally, they should be as close together as possible.

For example: here’s how files are laid out in a typical Pylons application:

/path/to/application/:

  public/
    images/
    javascripts/
    stylesheets/
  templates/
    index.mako
    specific-group-of-templates/
    etc.

This is a really nice layout — the files are close together, and there’s never any question where something is or where it should go.  New images go in public/images, new javscript goes in public/javascripts, etc.  This is the same from one application to another.  Ruby on Rails follows a similar pattern.

By contrast, here’s the file layout in the Almanac project:

/path/to/application/:

  parts/
    client/
       resource/
           lib/
              Almanac.js
           theme/
              almanac.css
              img/

   src/
      CommunityAlmanac/
         opengeo/
            almanac/
               templates/
                   mytemplate.pt

When I first saw this I was like AAAAH!  Not to pick on the Almanac, though; I love the Almanac.  Really, I do.  But, it can be frustrating to work with. Not only are the files in far flung branches of the filesystem, but they’re in separate checkouts.  This means lots of time spent switching back and forth between areas of the site (I ended up using two separate Coda profiles to work on these files), and two commits for nearly every change.

Could this have been avoided?  I’m not sure.  If we keep laying out apps this way, will the sky fall?  Probably not.  But if it’s possible, we should could consider the plight of the poor, defenseless designer when laying out our apps in the future.

First, some general impressions. The code is very clean C code. It actually uses Python objects internally for things that are annoying to safely represent in C, such as strings, vectors (Python lists), and hash tables (Python dicts). This means that readers can piggyback on their existing understanding of Python. The bytecode interpreter is also straightforward and readable.

The Python virtual machine is stack-based. Python bytecode is simply a list of operations with either zero or one sixteen-bit operand, depending on the operation. For example, a common section of code is:

...
LOAD_FAST 0
GET_ITER
FOR_ITER
STORE_FAST 1
...

LOAD_FAST and STORE_FAST give access to local variables. Every function has an array of locals, and the operand of LOAD_FAST/STORE_FAST is an index into that array. LOAD_FAST puts the value of the local onto the stack, and STORE_FAST pops a value off the stack and puts it into the local. GET_ITER gets an iterator for an iterable — it replaces the top of the stack with the iterator. FOR_ITER pushes the next value from the iterator on top of the stack on to the stack. You’re expected to pop it — the next operation is nearly always STORE_FAST. In fact, the Python bytecode interpreter has a predictor which quickly checks if the next operation is STORE_FAST, and skips some tests if so. So that snippet of code is the preamble of a for loop. Local variable 0 is the iterable, and local variable 1 is the iteration variable.

If you want to see the bytecode for a function, the dis module will print it. Unfortunately, that’s all it will do — it won’t give it to you in a manipulable form. If you want to write your own functions in Python bytecode, you need peak.util.assembler, which does not in any way interoperate with dis. To check how Python interprets a given opcode, check in the Python source code, Python/ceval.c.

The python compiler works in a fairly standard fashion — the source code is tokenized, the tokens become a concrete syntax tree, the CST becomes an abstract syntax tree, and the AST is compiled to bytecode. I started looking at the peephole optimizer, which operates on compiled bytecode (that is, an array of chars). I implemented a minor optimization for the case:

r = ...
return r

The bytecode looked like:

STORE_FAST n
LOAD_FAST n
RETURN_VALUE

My patch replaced it with just:

RETURN_VALUE

My patch was rejected, because apparently the additional six lines of code didn’t buy enough of a speed improvement for an uncommon case. In retrospect, however, it also didn’t work for all bytecode (although it may have worked for all bytecode produced by the Python compiler). Imagine if there were a jump to the LOAD_FAST from somewhere else in the code — for example, in the code

if x:
  r = 1
else:
  r = 2
return r

The last seven bytes of the bytecode would still initially look like:

...
STORE_FAST n
LOAD_FAST n
RETURN_VALUE

But if the store and load were removed, then the jump would lead directly to the return, which would result in a stack underflow.

This neatly illustrates why doing anything on the bytecode level is a huge hassle. I went through a number of schemes to try to eliminate unnecessary store/load pairs at the bytecode level, but trying to figure out variable lifespans in the presence of forward and backwards jumps is not easy.

I decided instead to take a suggestion made when I submitted my (broken) patch, and work at the AST level. Thomas Lee has been working on an AST-level optimizer. He wrote a set of functions that walk the AST and perform optimizations at each level. I also changed my focus, because I was a bit bored of thinking about local variables, and I wanted something easy for my first look at high-level C code in a long time.

Python bytecode includes at least one operation which is not directly accessible from Python code: LIST_APPEND, which pops a value and a list off the stack and appends the value to the list. This is used in generating code for list comprehensions:

[x for x in somelist]

generates the following code for the body of the loop:

...
LOAD_FAST 1#a local is generated to hold the new list
LOAD_FAST 2 #the iteration variable, x
LIST_APPEND
...

Whereas if you were to try to write the comprehension with a for loop, you would get:

LOAD_FAST 1
LOAD_ATTR 0 #an index into the constant table: "append"
LOAD_FAST 2
CALL_FUNCTION 1 #remove x and the append function from the top of the stack and push the result of the call
POP_TOP

This is much slower. In part, it’s the cost of doing the attribute lookup each time, which is why it’s common to see code like

a = []
ap = a.append
for x in .....:
  ap(x + ...)
return a

But the function call is naturally slower than the simpler LIST_APPEND operation. My new patch tries to determine statically if a local is all circumstances holds a list, and if so, generates LIST_APPEND whenever user code would call local.append. It’s not perfect — in particular, I could track local types on a more fine-grained level than per-function. But it’s a start. I submitted it today.

Next, I think I’ll consider local elimination again — but at the AST level.­

The packages in question are repoze.bfg.httprequest  and r­epoze.bfg.restrequest­.  restrequest is easier to understand because it is more concrete: basically it adds marker interfaces to your request object for the request method, and you can add request_type=IGetRequest to your view declarations, hence declaring that a view is only applicable to that request method.  You can do this with a decorator like:

@bfg_view(request_type=IGETRequest, for_=models.IMyModel)
def my_view_get(context, request):
    return Response("GETn")

Right now repoze.bfg (a web framework) uses interfaces to do it’s dispatch to views, so this adds the ability to dispatch on request method.  The existence of these packages bothers me a great deal, and it took a while to figure out why.  These are entirely packages, with docs, tests, etc., that express something trivial.  If repoze.bfg supported conditionals on views — functions that would indicate if the view was applicable to a request — then you could express this like:

@bfg_view(for_requests=lambda req: req.method=='GET', for_=models.IMyModel)
def my_view_get(context, request):
    return Response("GETn")­

This is basically what Routes has done, though it additionally added request-method-dispatch as a primitive (just a keyword like method=’GET’).  I think this is a much nicer construct, and no extra packages are required.  More importantly, it doesn’t just read well (arguably IGETRequest reads a bit better), but it is easy to understand the mechanism and to extend or adjust that mechanism.  If you want to dispatch based on the Accept header, you just use for_requests=lambda req: ‘application/xml’ in req.accept.  If you don’t want to write lambdas, you can just abstract this in the obvious way in a separate function or a class with a __call__ attribute.

OK, so that’s simple enough: repoze.bfg should support this construct, and it’s just an unfortunately and hopefully temporary awkwardness that caused these libraries to be written.  In theory anyone could, right now, write an alternative implementation of @bfg_view that accepts this.  Of course, anyone could spend an hour or two and write a new framework from scratch, which makes this a less compelling argument: if someone decides to use repoze.bfg instead of writing their own framework or using a different framework, it’s not because they want to write or maintain their own infrastructure, or be in the situation where they’d ask “can you help me, but oh by the way I’m not actually using your stuff”?

The other reason for this, I think, is ZCML.  Here’s another way of expressing method-based dispatch:

<bfg:view for=".models.IMyModel"
      request_type="repoze.bfg.restrequest.interfaces.IGETRequest"
      view=".views.my_view_get"
      />

There’s no place to put functions into ZCML, but there is room for interfaces.  And more generally, interfaces are the tool that comes to mind most commonly when using the Component Architecture.  I think it leads to this kind of anti-pattern all too often, the use of marker interfaces and the accompanying indirection when a simple function would do.  There’s a claim that interfaces allow for more flexibility, but this is simply false: functions are completely general and ameniable to any kind of abstraction you want.  Interfaces add indirection, and sometimes that’s useful, but in many cases like this the indirection is added before there’s any need for it, and when there might never be a need for it.

It can be hard for me sometimes to articulate what bothers me about Zope 3-style architectures, because it’s hard to point to what exactly the alternative would be.  Usually the alternative is: a few lines of code.  Code is a powerful and extremely general abstraction, and especially in a dynamic language like Python the amount of abstraction you can add is substantial.  And you don’t need to add the abstraction until you need it.  So usually the alternative I see to the Component Architecture is simply a few lines of code, written for the specific case that is at hand.

Although I haven’t tried it myself, the process looked easy, simple, and straightforward. Kapil made a conscious effort to keep it simple, and work for the most common use cases. Other scenarios will require using custom adapters.

It is a one way sync however. Any writes back to the relational database will not be reflected in the plone site. And, by default it will only catch archetypes. If you have other content types, you’d have to write custom adapters to get it working.

Using contentmirror can be useful for plone sites where you’d like to share the data with other applications. This opens up some exciting possibilities, like being able to use the django admin to look at the plone data. Anything that works with sql should be workable. It’s fairly new, but it’ll be interesting to see what people do with it.

I found that seeing it there and reading it really upset me.  Jackie saw me get upset and helped me realize that there were actually some broadly useful points to share about why it did and what this means in a greater context, so at her suggestion I’m going to try to unpack them here.

The referenced post — and the post that *it* references — is basically contentless.  Some popular blogger is linking to someone’s writeup of their personal impressions of a single panel at a tech conference.  He’s not adding any particular commentary of his own so presumably it’s an approving reference.  So for the essential content we go to the  original reference itself.  That post is equally contentless: as Kottke correctly observes with his summary headline, the post is meant to put forth a general theory about technical conferences and the “geeks” who organize and attend them  via a proof by anecdote.  It says: “I attended this one event, I felt this way about it, therefore I can offer general observations on a whole class of events and people.”  Well, we all know how rigorous that line of argument is.

Just for fun, though, let’s take it to its logical conclusion.  What, actually, is being put forth here?  That panels at tech conferences suck — because some guy on the internet went to one conference panel with an absurdly broad topic and didn’t get anything out of it except a lousy blog post — and so, I suppose, we should conclude that we shouldn’t be spending our time, energy and money on tech conferences?  Is this really a conversation worth having?  Of course there are good conferences and less valuable conferences in every field and discipline, from geology to nursing to quilting.  And at each conference, the individual sessions and panels will have a whole range of quality and outcomes.  To draw any conclusions about the broad category of tech conferences from a single failed panel at a single event is ignoring this reality and generalizing to what should be an obvious point of absurdity.  Come on.

So, essentially, it feels like this tagged item simply doesn’t contribute anything to a conversation; it’s a dead-end post with no information to impart and no worthwhile lessons to be drawn.  Its appearance on PlanetDev is an invasion of our communal space, and we all individually waste our time discovering its lack of value.

The world is full of those little distractions, though, and while I don’t like them, I don’t usually get too emotional when I see a Google Sponsored Link.  But of course this post’s content is not just unproductive; it’s unabashedly, gleefully insulting, playing to offensive stereotypes.  Har, har, tech “geeks” have no social skills, have no grounding in reality, get excited about techno-fantastical topics, aren’t good at explaining themselves.  Oh, and let’s mock their various supposed developmental disorders and drug indulgences!

This type of content sort of bothers me personally, but it points to something broader than just that.  While I’m sure this was totally unintentional, by putting this on PlanetDev Rollie effectively just pushed this type of stereotyping and mockery out into our group.  And, honestly, I really hope that we’re better than that, that we can create a culture of respect and collaboration here at TOPP, a safe space where no one will be mocked by a coworker for his interests or through implied association with a stereotype.  I can’t imagine anyone here is interested in descending into petty warring tribes based on our job descriptions.  So, please, let’s not get into the business of trading insults between designers and engineers, or any other “subgroups” at TOPP; let’s be respectful of each other and of everyone’s individual skills and interests, let’s work together without rivalry, and let’s respect our public spaces.

­Have you ever been frustrated by easy_install? Yeah, me neither. But I have heard whispers of discontent.

So I’m introducing a new tool for the installation of Python packages: pyinstall.

pyinstall is mostly easy_install compatible. That means it finds distributions in the same way as easy_install and it installs packages via setuptools. If you are familiar with easy_install you’ll know how to use pyinstall right away. This is not a repudiation of the mechanisms of easy_install, but a refinement. What does it refine?

First, it is a more user friendly easy_install. It collects everything it needs before it installs anything. The console output is intended to be concise and helpful. It says *why* it is doing things, tracking the set of dependencies that led to the installation of each package, and some of how it found the packages. It knows a little more about Subversion than easy_install, and I plan to add native support for other version control systems directly to it (this is easier now than it would have been a few years ago, since it seems there’s a clear and finite set of viable version control systems).

It also has some added features that I think are important for version management. Specifically the features of its predecessor PoachEggs, which you may not have heard of - but pyinstall at this point is at least a third-generation implementation of these ideas, if not more. You can install a set of packages from a “requirements file”, which really is little more than a list of packages to install. This is a seemingly small improvement, but the idea is to move requirements out of setup.py and into something that is managed separately from any library or package. By separating out this management you can control the application environments without having to touch the applications or libraries themselves. In addition you can generate new requirements files from a working installation: just run pyinstall --freeze=requirements.txt and it will write out a file that can be used to install the exact version of everything that is installed. One of the most common complaints about easy_install once you get second-order dependencies is that you can’t easily reproduce working environments, and this fixes that.

Another feature is a “pybundle”, which is just an archive of a set of libraries. Here’s something you can try:

$ virtualenv my-app
$ cd my-app
$ source bin/activate
... set up my-app environment just how you want it ...
$ pyinstall.py --freeze=my-app-req.txt
$ pyinstall.py --bundle=my-app.pybundle -r my-app-req.txt
... then on another machine ...
$ pyinstall.py my-app.pybundle

The pybundle format is pretty simple: it’s a lot of source files in one zip file. It isn’t a binary package format at all, it’s just source, and all it saves you is the finding and downloading of packages. But I think that simple process is a big part of why no-dependency libraries and frameworks are sought after (not to say that’s the only reason).

Despite the many rewrites preceeding this, pyinstall is still very young — but for the most part if you get something installed, then it worked, and if it didn’t get something installed then you can always fall back on easy_install (and submit a bug report).  If you have suggestions, also submit a ticket or ask about it on the virtualenv list.

To get started: easy_install pyinstall­ or jus­t grab the single-file executable.

As some of you know, I’ve been doing the brainpower project as a Django admin application.

The reasons for this decision were:

• Django admin is touted as a very quick way to build CRUD applications, since it generates a UI from your model that in many cases is good enough for end users. No forms to write, maybe just a little tweaking. Brainpower is nothing other than a simple CRUD application, so this sounded like a perfect match.
  
• Good excuse to learn a little about Django.
  
• Get me to do something other than Zope for once in my freakin’ life.

So how did that pan out?

Well, in one month since our first requirements-gathering meeting with our “customer” (Liz and Robin from Streetfilms), in addition to all the other stuff I’ve been doing, I built something they said was good enough to start using. I did in really just a week or so of work, alone, from scratch, with almost zero advance knowledge of Django. I even spent some time testing and tuning performance (just enough that I feel confident we won’t ever have a problem with it). This also includes a full suite of Flunc tests; a random content generation script that I used for the performance tests; and a build script for development & deployment using Fassembler.

The core code is tiny: the bulk of it is in a 250-line models.py module that is reasonably clean.

As usual, writing the core code is only a small part of the story. A large portion of my time went to things like figuring out how to conveniently run external functional tests against django with a scratch database, writing and fixing the build script, and troubleshooting my initial attempts at deployment (tripping on a django bug.).

I do have some general early impressions of Django.

Things I liked

• The admin interface is, indeed, pretty slick (with a few minor oddnesses like a pretty useless Time widget).
• If I had another little CRUD app to build that seemed well-matched for Django, I could probably do it ridiculously fast.
• The Django docs are pretty decent for the most part, much better than the current state of, say, Zope 2 docs, and more extensive and thorough than the Pylons docs. (Too bad I had to quickly un-learn a bunch of stuff when I had to switch to developing against more recent django checkouts.)
• The stable release’s way to create an admin UI — by writing an inner class named Admin inside your Model — smelled really bad to me. Thankfully, this is gone in the newforms-admin branch. Similarly, you used to wire up custom validators inside your Model class and do cleanup in its save() method; now you can do custom validation in a ModelForm subclass, and you can do data cleanup in the same place. Newforms-admin is pretty nice!
  
• Got a multiple choice field that you need users to be able to extend with new choices?  Just add a foreign key field to your model referencing another model, and it just plain works in exactly the way you’d hope.  Slick!
  

Gripes

• I wish they had used an existing template language, I don’t see anything compellingly different about Django’s.
  
• I wish Django’s setup.py had a “develop” command.
• I wish the tools for syncing your models to your database were more developed. You seem to be entirely on your own for migrations. For example, if you modify the type of a field in a Model, you’ll likely need to either drop your tables and recreate them, or if you have production data to preserve, either do a dump-modify-restore cycle or hand-hack the database in place to keep your app working. At one point I ended up needing to do a dump-modify-restore using xml exports and lxml transforms; it took longer than I would have liked and I might try another strategy next time. For other model modifications such as adding a field, you might be able to get away with re-running manage.py syncdb; unfortunately I don’t know how to predict when this will just work and when it won’t.
• I wish all Django manage.py operations could be performed non-interactively. For example, for my flunc tests I wanted to reliably create and destroy a scratch database with a test admin user. Django provides a way to do most of this, but the commands that create an admin user must be run interactively. I tried using a database fixture, but that didn’t work reliably — I’m wildly guessing there’s some salt that gets reset and I don’t know when? I ended up having my test script use pexpect to drive the interactive commands.
• It seems I arrived at an inconvenient time. Django 0.96.2 may be the “latest stable release”, but it’s aging. The developers are all focused on a “newforms-admin” branch, which is cleaner and more extensible, but this work hasn’t landed on trunk yet. I don’t know when that will happen, or what else will change before another stable release. I opted to develop against the old stable release. I thought I was being smartly conservative :-p Unfortunately I soon hit a really irritating bug. While trying to understand the admin code enough to find a workaround (and feeling like I was on a familiar learning curve), I was advised to check if trunk still had the same problem, which meant an hour or two adjusting my code to work against trunk. Trunk turned out to fix only half of the problem. Then a week later the bug got fully fixed… on the newforms-admin branch. So much for trying to stick to a stable release! Maybe I was just unlucky. If I didn’t need to search fields from a join, I wouldn’t have hit this bug. (But isn’t that a common need?)
  
• Customizing and extending the admin UI is commonly done in a way that also seems vaguely familiar: You just make a copy of the thing you want to modify and hack away on it. I’ve already had a hint of pain with this — my two modified template copies broke on both Django upgrades. That’s how things typically go in the Plone world: Every single skin override you did on a Plone site would add a drop of future pain to every Plone upgrade you ever tried to do.

  This is a hard problem to solve. Just as Plone did, Django admin seems to be gradually adding more plug-in extension points so you don’t have to override the core templates as often; instead you just flip some configuration switch and/or add another template that magically gets slurped into the right place. Which has its own headaches, as every one of those bits of flexibility adds to the learning curve.  Let’s hope Greenspun’s Tenth doesn’t grow a corollary substituting Plone for Lisp :-]