OpenCore Software

Lately I’ve been hearing people talk about a service-based architecture to make our UI easier to work with.  To summarize my understanding of a service based architecture:

• Back end applications have explicit over-the-wire APIs.  That is, you fetch data from them, and render pages based on that data.  The data forms a stable API.  The actual data might be XML or JSON, and applying changes might be RESTful PUTs, or might be RPCish POSTs.
• We develop a single application that is the skin on all our backend applications.  All the UI goes there.  We can use coding conventions to apply UI conventions universally in our site.  Other sites could implement other applications using our backends but with their own frontend UI application.

I believe (strongly) that this is a bad path to go down, and it’s a huge architecture change in response to resolvable development issues.

The fronting UI application is perceived of as simple.  But how simple is it really?  How easy can it be to write an edit screen in this UI application?  If it’s going to give us flexibility in our UI work, that means that often the frontend requests will not be isomorphic to the backend requests — for one save, or even one page render, there could be a large number of backend requests, and complicated ways to aggregate and join the results of those requests.  Does Latest Activity get the last 10 items from everything that could produce items, then choose the real top 10?  How do you configure all the items together?  More XML configuration?  TOPPCML?  If you implement a form that allows editing data from several backends at once, what happens when one of those backend applications gives an exception on save?  When you consider validation the surface area of these individual applications starts to become large.  Given a page, how do we find the code that needs to be changed, if a change can’t be done solely in the UI layer?  How do we we manage the (*very* common) case when both frontend and backend have to be updated in concert?  All these seem like the hard parts of application development.  We don’t want to increase our hard work.

There’s three reasons I see for the interest in a service-based architecture:

1. You can mock out the backends and get a certain amount of stability between frontend and backend, and development can occur in parrallel.  I’m suspicious of whether this is really possible, but it’s a common desire.  Mocking backends is in substantial part a workaround to avoid development deployments, and I hope we don’t need to avoid development deployments anymore.
2. It centralizes the integration of our applications.  Rob Marianski talks about the issue with transclusion and Twirlip.  This is a valid problem.  I’m not sure about the solution, though in some cases we have tools to resolve this (e.g., with Cab).  If our applications have certain bindings to each other, or special configuration or plugins that we have to implement to make them less closely bound, I think this is easier than a service-based architecture.  Basically I’m advocating ad hoc extensibility in our applications rather than a single framework that encompasses all kinds of potential means of extension.
3. It centralizes UI work, because there is a canonical location for templates, CSS, class definitions, etc.

We’ve been working on 1.  2 I admit is hard.  I think 3 is quite resolvable, and that we should resolve it.

To handle UI work it is important to know (a) what files are used to generate the UI, (b) what those files do, and (c) how those files can be changed. 

(a) can be resolved if we included template filenames in all template renderings when some development switch is turned on.  I can tell you how to do this in Pylons applications, and I’m sure someone can figure something out for Zope too, and hopefully even PHP.  Additionally Transcluder and Deliverance would do the same things for their page composition.  Maybe a bookmarklet would reveal the names (since they can mess up rendering if always displayed).

(b) is similar: we include information on the objects used to render the pages (the Zope view, the variables in the pylons.c object, and… all global variables in PHP).

(c) is probably just a matter of implementing (a) and relying on people to edit files in the proper place.  It would also be possible to implement External Editor support.  This is a Zope ZMI product, but its API is actually extremely simple — serve a page with a magic content type that points to the URL where the page source is located (plus an optional cookie auth token), External Editor fetches that and opens an editor, and whenever you save External Editor will PUT the new content to that same location.  I used External Editor a lot when I was doing Zope ZMI development, and it made the experience tolerable.

But that editing only applies to changing the main-line code.  We also want to change UI for local deployments.  To facilitate local modification (e.g., nycstreets.org) we should have a search path for all template loading so that templates can be copied for per-deployment customization.  Of course these copies can also go in svn, and we can use normal merge techniques to keep them working as the original templates are modified.  I also believe strongly that it should be possible to put Real Code into templates; while this isn’t a good smell, it’s an entirely acceptable way to do local modifications.  I remember hearing that even ZPT has the potential to run arbitrary Python with <script language=”python”>, though it is disabled in nearly everyone’s setup.  But we could enable it, if the rumor turns out to be true.  (If it turns out to be false, we could also just look for a neighboring .py file and exec it.)

Service architectures take XML or JSON documents and then transforms and aggregates them into something you render.  As the underlying documents change the transformations have to be updated.  But lacking a clear canonical original transformation (like the original template) this seems harder to maintain.  And ultimately what the transformations are doing is basically equivalent to a template.  Templates transform the input variables into rendered HTML.  The original templates have access to richer objects and methods than what a document transformation can do.  We have the ability to improve our stack to support Rollie and ourselves when doing UI work.  We shouldn’t be shy about making these kinds of infrastructure improvements.