OpenCore Software

Robert and I have been working on the opencore feeds. Currently, these are only used by the project summary page, but looking at the site, we have many things that could be feeds: Latest Projects, Latest Members, Recently Updated Projects, just to name a few. This blog post is sponsored by the word ‘agnostic’, where agnostic means “optimistically not incestuous code” and “incestuous” means “code that unfairly presumes coupling or functionality that destroys perceived modularity and forces binding to a framework even when the desired functionality should not depend on the framework”. Okay, those are just words. The idea is that incestuous code is badly coupled code. Its a word I use alot to describe code I don’t like, because something can (perhaps wrongly) be described as modular but be completely incestuous. Agnostic is the antidote (and the antonym) to incestutous.

Sorry for that gratuitous introduction. The point is that I think what we did with feeds is not a bad model:

[So I tried to upload this image through wordpress’s xinha.  Of course, horrible things happened, so now I’m just linking to it.  GUIs are my bane; did i mention I’m now editting this in html mode, as I almost always do, even though basically i want text ]

This is a horrible diagram with symbolism that only makes sense to me. So let me describe. You have a bunch of content (projects, mailing lists, etc) that provide feed data. They may provide feed data in more than one way, but that’s another story. Instead of having the wild west, I make an interface called IFeedData that forces the feed in the standard format. In this way, I have decoupled what the feed comes from from what a feed is. IFeedData (and its items) have some things that are mandatory to define — the essence of what a feed is — and what is optional.

Moving further right on the chart, there are a bunch of templates. Any of these templates can be used to render any feed. Some of them will work better with particular type of feed than others — for instance, if the feed doesn’t have icons, the portrait_feed_snippet.pt is pretty silly; but each can render the basic functionality of “what a feed is”, and may support optional items, conditionally displayed if they exist. To extend functionality, add a new template.

So we have feed providers, which give us feeds, and templates, which display feeds. How are they used?

So we have this viewlet manager, which has a bunch of viewlets. The viewlets have some sort of context which gets adapted to give a feed. The viewlets *also* have something that says “use *this* template”. In other words, the viewlets decide how the feed is displayed.

I’m not writing this to say “ooooh! look at this cool pattern i helped make!” I’m writing this as an illustration of how something that isn’t (amazingly) incestuous can work. Notice you have decoupling on every level. You have content providers which have rules to transform them to feeds. You have feeds, which have various templates that can be used to display them. You have viewlets, which contain the display logic, tell how to adapt their context to the feed, and choose the template. On top of that, its a simple system: abstract enough to handle anything I can imagine atm, but concrete enough to actually use.
There are problems going forward that I haven’t conquered yet, but I am at least comfortable with the pattern.

Last night Google announced Google App Engine.  Since then I’ve been pretty obsessed with it.  (If you are interested in trying it out, you should sign up and put in a request — they seem to be sending out invites periodically).

Since last night I’ve been reading about it, and some of the commentary around it.  I haven’t actually tried to run anything on it yet.  But here’s my first impressions:

It’s not as peculiar system as some people have suggested.  It runs CGI-ish scripts, and it is easy to turn that into WSGI.  Processes can be reused, but they are short lived and single-threaded.  You can’t have any background processes.  It’s a lot like the PHP process model.  An SDK is provided that allows you to run stuff locally.  The SDK is open source (Apache licensed), so at least a minimal environment is available with no proprietary ties, and it provides a model for reimplementing proprietary parts of the stack.

There’s no database, but there is Google BigTable stuff, with a database-like API.  It accepts queries that look like SQL, and they give a Django-like ORM.  It’s close enough to be familiar, but it’s not exactly a database.  Like the ZODB, you have to add indexes for any queries you want to run outside of the most obvious queries.  It might be more accurate to look at BigTable as ZODB-like than RDBMS-like.  But I don’t know; it’s a big topic and I’ve only read that part of the docs lightly.  This is probably the biggest proprietary tie that an application written for appengine will have.  But I think that making this API work over the ZODB would be feasible.  You won’t get the same scaling properties of BigTable, but the application would still be viable.  There’s also some APIs for email, authentication, and doing URL requests.  Only authentication really seems concerning from the perspective of tie-in.  Google, unfortunately, has not been particularly progressive with respects to OpenID.  But people have already used OpenID with this, so it’s not a hard constraint.  Given Google’s involvement in Open Social stuff, I also imagine that their stance on authentication will improve.

That’s just a short description, but I post it here because I’ve been thinking about what if anything this means for TOPP development.

Generally, I think this could be a very big thing for Python web development.  This offers a development environment that is on par with PHP (which I think has a very good and accessible development environment, IMHO the most significant reason for its success).  Well, better than PHP really, as you get most of the deployment advantages, but they’ve structured the system in a way to make good development practices easy (private staging deployments, you can update and revert versions of an application, and probably other stuff I haven’t yet noticed).

This relates to us because of what this could do to the general ecosystem for open source web applications.  Right now deployment is hard.  Hard enough to seriously stunt the success of open source web application projects.  Even the most successful applications — for example, Trac and MediaWiki — seem to be a flash of activity until they are superseded by easier-to-manage hosted services.  Google Code’s issue tracker is a far cry from Trac, and not extensible, but it’s so much easier to manage that I have a hard time recommending going through the trouble of setting up Trac (and dealing with spam and other problems) when that time could be better spent actually writing code.  Supporting deployment on systems also is a lot of overhead for open source developers, and the overhead has very little return.  Supporting more deployment options doesn’t actually improve a product.

So as a result the emphasis has been on hosted services, and open source development effort has been focused more on tools for building those services than on the services themselves.  In some ways this seems reasonable: with public, free, hosted services (which is a lot of services these days) why do you need more than one implementation?  But of course that’s a simplification, and there are many reasons you might want to modify a web application at the code level.  And while mashups offer some extensibility without code sharing, and with closed/hosted services, they can be quite limited.  Unless you write your own applications, which is too difficult for many people to do reliably because deployment is hard, expensive, and hard to scale in response to the surge in traffic a successful mashup can get (a surge which might not turn into any means of economic stability, as the surges themselves are unstable).

Google App Engine could change this.  This is a hosted platform that appears that it will, when it gets out of beta, offer basically free hosting to small web applications.  (The quota limits, which they say will remain for the free service after it gets out of beta, are quite generous.)  So, for instance, if the Trac developers make a port to this environment (which seems quite possible) then installing Trac, with whatever plugins you want, and any local modifications you care to make, suddenly seems very feasible.  Even feasible for people who could only marginally be classified as “developers” — a class of people who have almost entirely gone to PHP up until now.

Internally we’ve always struggled with how functionally open source our products can be.  That is, while people are allowed to use the stuff we write (via licensing), that does not necessarily make the stuff we write compelling or useful.  Deploying our entire stack is somewhat challenging, and even putting aside the technical points of that deployment, there’s the concern about whether it is a useful basis for someone outside of our team to make a site.  This might be resolvable, but when open source web applications have such a limited potential for success it makes it hard to justify resolving problems.  This could substantially increase the motivation to create reusable applications.

Lately I’ve been hearing people talk about a service-based architecture to make our UI easier to work with.  To summarize my understanding of a service based architecture:

• Back end applications have explicit over-the-wire APIs.  That is, you fetch data from them, and render pages based on that data.  The data forms a stable API.  The actual data might be XML or JSON, and applying changes might be RESTful PUTs, or might be RPCish POSTs.
• We develop a single application that is the skin on all our backend applications.  All the UI goes there.  We can use coding conventions to apply UI conventions universally in our site.  Other sites could implement other applications using our backends but with their own frontend UI application.

I believe (strongly) that this is a bad path to go down, and it’s a huge architecture change in response to resolvable development issues.

The fronting UI application is perceived of as simple.  But how simple is it really?  How easy can it be to write an edit screen in this UI application?  If it’s going to give us flexibility in our UI work, that means that often the frontend requests will not be isomorphic to the backend requests — for one save, or even one page render, there could be a large number of backend requests, and complicated ways to aggregate and join the results of those requests.  Does Latest Activity get the last 10 items from everything that could produce items, then choose the real top 10?  How do you configure all the items together?  More XML configuration?  TOPPCML?  If you implement a form that allows editing data from several backends at once, what happens when one of those backend applications gives an exception on save?  When you consider validation the surface area of these individual applications starts to become large.  Given a page, how do we find the code that needs to be changed, if a change can’t be done solely in the UI layer?  How do we we manage the (*very* common) case when both frontend and backend have to be updated in concert?  All these seem like the hard parts of application development.  We don’t want to increase our hard work.

There’s three reasons I see for the interest in a service-based architecture:

1. You can mock out the backends and get a certain amount of stability between frontend and backend, and development can occur in parrallel.  I’m suspicious of whether this is really possible, but it’s a common desire.  Mocking backends is in substantial part a workaround to avoid development deployments, and I hope we don’t need to avoid development deployments anymore.
2. It centralizes the integration of our applications.  Rob Marianski talks about the issue with transclusion and Twirlip.  This is a valid problem.  I’m not sure about the solution, though in some cases we have tools to resolve this (e.g., with Cab).  If our applications have certain bindings to each other, or special configuration or plugins that we have to implement to make them less closely bound, I think this is easier than a service-based architecture.  Basically I’m advocating ad hoc extensibility in our applications rather than a single framework that encompasses all kinds of potential means of extension.
3. It centralizes UI work, because there is a canonical location for templates, CSS, class definitions, etc.

We’ve been working on 1.  2 I admit is hard.  I think 3 is quite resolvable, and that we should resolve it.

To handle UI work it is important to know (a) what files are used to generate the UI, (b) what those files do, and (c) how those files can be changed. 

(a) can be resolved if we included template filenames in all template renderings when some development switch is turned on.  I can tell you how to do this in Pylons applications, and I’m sure someone can figure something out for Zope too, and hopefully even PHP.  Additionally Transcluder and Deliverance would do the same things for their page composition.  Maybe a bookmarklet would reveal the names (since they can mess up rendering if always displayed).

(b) is similar: we include information on the objects used to render the pages (the Zope view, the variables in the pylons.c object, and… all global variables in PHP).

(c) is probably just a matter of implementing (a) and relying on people to edit files in the proper place.  It would also be possible to implement External Editor support.  This is a Zope ZMI product, but its API is actually extremely simple — serve a page with a magic content type that points to the URL where the page source is located (plus an optional cookie auth token), External Editor fetches that and opens an editor, and whenever you save External Editor will PUT the new content to that same location.  I used External Editor a lot when I was doing Zope ZMI development, and it made the experience tolerable.

But that editing only applies to changing the main-line code.  We also want to change UI for local deployments.  To facilitate local modification (e.g., nycstreets.org) we should have a search path for all template loading so that templates can be copied for per-deployment customization.  Of course these copies can also go in svn, and we can use normal merge techniques to keep them working as the original templates are modified.  I also believe strongly that it should be possible to put Real Code into templates; while this isn’t a good smell, it’s an entirely acceptable way to do local modifications.  I remember hearing that even ZPT has the potential to run arbitrary Python with <script language=”python”>, though it is disabled in nearly everyone’s setup.  But we could enable it, if the rumor turns out to be true.  (If it turns out to be false, we could also just look for a neighboring .py file and exec it.)

Service architectures take XML or JSON documents and then transforms and aggregates them into something you render.  As the underlying documents change the transformations have to be updated.  But lacking a clear canonical original transformation (like the original template) this seems harder to maintain.  And ultimately what the transformations are doing is basically equivalent to a template.  Templates transform the input variables into rendered HTML.  The original templates have access to richer objects and methods than what a document transformation can do.  We have the ability to improve our stack to support Rollie and ourselves when doing UI work.  We shouldn’t be shy about making these kinds of infrastructure improvements.

Robert pointed out that our “microapp” architecture has the potential to become extremely hard to work with and isn’t really giving us any benefits in the process, since we’re not actually decoupling any of the components. I’d like to think more about that, but in the meantime, I’ve been thinking about what I think would be a really compelling use for proper microapps on our site that wouldn’t be increasing our administrative burden in the way Robert talks about.

(Incidentally, just to be clear, when I talk about proper microapps, I mean simple UI-less applications with a very stable API over HTTP that provide a single service intended to be plugged in to other contexts.)

So, we’re trying to use (quasi-)microapps to build our site on the backend, but I think microapps would be most valuable as services we provide *for users* instead. It seems more compatible with the microapp tendency towards runtime configuration over HTTP; while we end up bending the code and our builds to configure these applications to set up our site at “build and compile” time, users could just configure the applications for themselves on an already-running site to gain additional and personalized functionality.

Offered as user facing tools, these microapps would provide extremely flexible functionality beyond what we offer universally — ways for individual users to layer their own custom configurations on top of the site. Transcluder is a simple example; by just turning on a sitewide transcluder, we would allow users to build very flexible componentized webpages. I think an HTTP request hub like Cabochon could be another really powerful service for users: while we handle our own events internally with code and configuration, users could configure a set of individually saved hookups between HTTP events and resources. So one user could set up a rule: “When I publish to the project’s blog, mark my `write blog post` task as completed” while someone else sets up a rule that “When anyone edits this wiki page, create an account page message for me.” Throw in a cron-like microapp to schedule HTTP requests and now users can have a new “Write weekly blog post” task created every week and more.

Admittedly I’m doing a tremendous amount of hand-waving here (authorization? authentication? request interception? recursive requests? — implementation details!) and I haven’t come up with very compelling use cases, but I don’t think it’s impossible and I think the real potential of the idea is that *we don’t have to come up with the use cases* — people could come up with really interesting combinations of events far beyond anything we’d want or need to build into our software.

Another big advantage here is that since these microapps would have very well-defined APIs that work over HTTP, we could actually deploy them with minimal user interfaces and gradually build up UI over time. “Power users” would immediately get to experiment with the tools immediately, and we could even use their feedback to determine common usage patterns to inform the friendlier interfaces. So we can turn on Transcluder without any UI, and people could immediately start using it by editing a bit of HTML; then, as a first pass at UI, we could build a switch into the Xinha linking interface to toggle the “rel=’include’” attribute; and eventually we could design a much more sophisticated UI (perhaps leaving the “design-less” interface available underneath in the same way that WYSIWYG editors allow you to drop down to HTML) but without that design ever being a bottleneck to deployment or usage by at least some people. Likewise an event hub could start out by letting people input URLs and HTTP verbs, and eventually build a descriptive interface which lets people connect “things to do” from a list.

I think this would be a really powerful way to add, essentially, arbitrary flexibility to a site without having to hard-code it, and without having to come up with individual use cases, instead offering the tools to users and letting them come up with their own uses. And since we really could treat it as a layer on top of our software, which we don’t actually have to know anything about the use of, it wouldn’t be adding to our administrative burden, since we wouldn’t have to worry about the connections between microapps. I don’t think it replaces the need for some sort of internal connection between features and data — we can inevitably come up with much more tightly integrated services on the backend, and in any event I’m sure there are commonly useful hookups which we ought to just provide directly. But it would be a great way to provide functionality beyond what we decide to offer or what occurs to us to implement.

I thought it would be useful to make note of how a couple things were done recently using Deliverance.

The first is making http://nycstreets.org/help/ point to the main openplans.org help page. This was done ad-hoc, though I’ll show how to do the equivalent in a better way with the new openplans_hooks and build.ini.

The nycsr Zope lives at localhost:12002, while openplans Zope lives at localhost:13002. When a request comes into nycsr it is generally dispatched/forwarded to http://localhost:12002/VirtualHostBase/http/nycstreets.org:80/openplans/VirtualHostRoot/PATH. That path lets Zope know how the request was originally made (so that links don’t point back to localhost:12002).

You can do ad hoc overrides with dvhoster using remote_uri.txt. Specifically we want to forward http://nycstreets.org/help/PATH to http://localhost:13001/VirtualHostBase/http/www.nycsr.com:80/openplans/VirtualHostRoot/help/PATH — note that /help goes after VirtualHostRoot. To do this we change remote_uri.txt to look like:

  [{"path": "/help",
   "remote_uri": "http://localhost:13001/VirtualHostBase/http/www.nycstreets.org:80/openplans/VirtualHostRoot/help"},
   {"path": "", "comments": "main-site"}]
  

Note that paths have to be ordered with longest path first, as the paths are matched in order (and “” matches anything). The “remote_uri” key tells dvhoster where to dispatch the request. The value ultimately goes through openplans_hooks.py which gets another try at rewriting the paths; as implemented it only does this when remote_uri isn’t set.

If you wanted to mount a project this way, you could do it like this:

  [{"path": "/bikesforbob",
     "remote_uri": "http://localhost:14001/VirtualHostBase/http/www.nycstreets.org:80/openplans/projects/VirtualHostRoot/bikesforbob"}]

Another things we did recently is to put static files up at http://dev.nycstreets.org/nymap/. The files were checked out to nymap/src/nymap, and so the remote_uri is file:///path/to/base_path/nymap/src/nymap/. In this case we used build.ini, like:

   nymap uri = file:///usr/local/topp/dev.nycstreets.org/current/nymap/src/nymap/
   nymap path = /maps
   nymap project_local = false

This is similar to the ad hoc technique:

   [{"path": "/maps",
     "remote_uri": "file:///usr/local/topp/dev.nycstreets.org/current/nymap/src/nymap"}]

“nymap project_local = false” means that this is associated with the root of the site. If it was true then /projects/PROJECTNAME/maps would be mapped instead of /maps (in a virtualhost situation /maps would be matched regardless).

This got the maps in place, but they needed access to the artois to do XMLHttpRequests, and those can’t be done across domains. So we needed to proxy artois over to dev.nycstreets.org. This was done with:

   artois_proxy uri = http://artois.openplans.org/geoserver/ows/
   artois_proxy theme = false
   artois_proxy path = /maps/rpc
   artois_proxy project_local = false

Now requests to http://nycstreets.org/maps/rpc get proxied to http://artois.openplans.org/geoserver/ows. We don’t need to theme it because it’s all API calls. Turning off theming in the configuration is slightly more efficient (though only slightly as the XMLHttpRequest calls wouldn’t ultimately have been themed anyway).

Anyway, that’s a couple techniques used recently. This sort of thing can be used to do other manipulations with the site. I’m hoping this description will get people to consider this technique when other kinds of requests come in.

As we consider our architecture, consider rewriting portions of our site, and consider writing small very targeted applications (like an application targeted at block parties) we have to start thinking about what commitment we have to our users and to the continued functioning and maintenance of these applications.

For instance, if we rewrite TaskTracker in a way that doesn’t map well to our current lists, what are we going to do with existing lists? Will anyone care if we just throw them all away? Should we move them over? Can we move them over in a scrappy way, flattening lists and losing metadata, but at least keeping the raw content?

That choice should probably be driven by a sense of how many people use the application, and if we can feasibly collect it, how much those people care about the data in their lists. But given that information we’ve just informed our decision, we haven’t made one.

So what is our commitment to the specific applications we’ve written? What is the cutoff level where we are comfortable throwing stuff away, including data? What is the cutoff level where we can throw the application away and keep the information around for reference (e.g., making a static HTML dump of the data)? For future applications, how can we rapidly make useful applications without burdening ourselves with too much code liability, too many things to maintain and monitor?

One possibility with an application like block party is that we write something with a sunset date. In, say, November 2008 we shut the application down, and we are clear about this up-front. Maybe in September we start marking the site with big “GOING OUT OF BUSINESS” messages. After that we replace it with a static file (and maybe a redirect for the whole tree). If the application was good and we want to deploy it again next year, we can, maybe improved for maybe not.

If we do pick up the pace of new application development, I am a little scared about the code liability we could introduce — either spending too much time keeping the applications up, or creating a sloppy site where applications just disappear and stop responding after a prolonged period of non-maintenance. Or if we do good change control and don’t break applications they just become spam pits, which isn’t any better.

Okay, so maybe that’s a bit of a bold title, considering a) most of you have been using fassembler for a little while already and b) Ian wrote fassembler, not me.

However, it does feel like just today we’ve put the last little bit of polish on it, so that it’s officially ready for prime time for all of our deployment needs. There are still some parts of the process that can be improved, but it’s definitely workable, and it’s orders of magnitude easier to use than anything we’ve had up until now.

How, exactly, should it be used? Well, if you just want to use it to build a local development rig of the OpenPlans software, you can refer to the fassembler how-to. If you want to actually deploy a persistent site, one that lives on either our dev or our live server, you can refer to the OpenPlans deployment instructions.  The deployment instructions go into quite some detail about how our directories are laid out, and how you should manage the configuration and requirements for separate sites, and separate builds of the same site. Please do not try to do any deployment of any site on either flow or theman without reading this first.

Good luck and happy building!

Proposal // question:

Weird, but our architecture is weird.

[ETA: Please let me know if I got any current architecture details wrong.  Sorry, this makes it worth 1,087 words instead.]

This asymmetry really bothers me:

I don’t have it quite right, I know, but that’s sort of the point.  I feel like /wiki and /blog are the ones that are confusing and wrong, like we should offer multiple wikis and blogs per project.  The glossary-of-terms wiki, the public site wiki, the admins’ notepad wiki; the news blog, the development blog …  On the other hand, I don’t want task lists >> tasks in the future, though I do still want them to exist but in a very different form, so I don’t know whether that keeps the symmetry or not.  Depends on how firm a definition “collection” has?

I want to note that I don’t *think* it’s entirely an aesthetic preference; I feel like the asymmetry might cause minor headaches if we try to, ah, “desilo” and integrate all content.  (a wiki page has a task, a blog post has a task, a task has a wiki page, a task has a file, a wiki page has a file … )  Also if we ever try to offer all features equally.  (which we currently don’t — wiki is “more fundamental” than lists, tasks and blogs.)

Yeah, ’cause what goes into those two empty cells in the table are really “PROJECT” and “PROJECT WITH /BLOG APPENDED” … but that’s not right, of course, because “PROJECT” should be above the whole table, uniformly, as the top row.  I feel like I’ve just proven something but I’m kind of at a loss.

Hmm, on the other hand, in the same way that I now envision (thanks, Jeff!) that a task is simultaneously content (task data) and associations to other content (tasklist data), you can also describe a wiki page as content (the text of the document) and associations to other content (wiki links) … that feels shaky, though.