OpenCore Software

It occurs to me I never wrote about what I did for the rest of the sprint after my first couple of posts.  Mostly I spent the second half of the week floating around the big room; I didn’t really do any more long focused work after the first couple of days of pairing with Robert and Martijn. But here’s a rambling overview; sorry it’s so disorganized

KSS 

I looked very briefly at KSS, the “Ajax framework without Javascript” which lets you apply behaviors to a page with something very much like a stylesheet.  I had originally seen KSS a bit over a year ago when Whit pointed me to it, but it seems like it’s come a long way since then.

For one thing, their website is a lot easier to understand. It looks a whole lot more attractive; there’s a lot more up-front explanation of the project; and they picked a consistent name for the project.  (It’s now all just KSS; from what I remember, last year they were talking about KSS, Kukit, and Azax, and I was very confused.)  I guess nice packaging really does help.

Part of me is still wary of it but I think my reason is silly; I just have a gut feeling that it doesn’t make sense to try to bury Javascript and pretend it’s not there, and that KSS is just for people who don’t like Javascript (and I like Javascript, dammit! At least in theory) and instead you may as well leave Ajax for people who don’t mind it.  I’ll say again, though, I think this is a stupid thing to think.  Robert pointed out that it more or less accomplishes the same thing that Nick and I did with Octopus over the summer, so I’d be interested in doing a direct comparison there and seeing what one of our NUI forms would look like with KSS instead of Octopus.

Anyway, Jeff did a lot of work with KSS at the sprint, and Jeff hates Javascript, so I’m also curious to hear his impressions of it.

z3c.autoinclude

I also did some scattered work related to z3c.autoinclude throughout the rest of the week. At Martijn’s suggestion I wrote to the grok dev list about it, and there were a couple of very good suggestions that came out of that like doing better logging and exposing a convenient way to get information about what files to include without actually doing the autoinclusion, so that you could build alternate inclusion mechanisms on top of z3c.autoinclude like a build script that writes out inclusion statements to a static, tweakable site.zcml. I’ve gotta say it was satisfying and pretty exciting to be talking about my code to people I’ve never met, and changing it based on their recommendations, and hearing completely new ideas about how to use it.

Vudo

I spent an afternoon pairing with Malthe Borch on Vudo, a concept for a new content management system built in Grok with easy Jottit-style content creation and draggable, recombinable page elements. (I don’t think they’ve made a website yet but here’s a video interview from the sprint about it.) I worked with him on a custom traversal hook to create a new object at any URL that isn’t already associated with an existing object, object attribute or view. It was interesting to hear how they’re talking about implementing their system, since a lot of their desired UI sounds like things we’ve talked about too. (Jeff, I tried to find your “draggable page building GUI” blog post to link here, but I couldn’t find it, sorry.)

Gibberisch and Bullschit

On the last full day, looking for something totally different to play with, I sat down with Kai Lautaportti and Tarek Ziadé who had decided to write a random text generator. After spending about five hours trying to get started with git (which never ended up working for Tarek — though, despite all our frustrations, in the end I was sold on it at least as a branching and offline supplement for SVN) and talking about Tarek and Kai’s competing one-line generation algorithms, we had “Gibberisch”, a working library and console script that gave us vaguely plausible paragraphs using a customizable bank of sentence parts. Then Tarek started working on an alternate method while Kai and I banged out “Bullschit,” a proper RESTful text-generating web service in Grok powered by Gibberisch. (We also built an awesome demo.)

I really like the way Grok does REST, by the way: it’s ridiculously easy to create a RESTful interface to any object, and unlike Pylons or our Z2/Plone/Z3 system (the only other things I’ve really tried to do any REST with) I didn’t have to fight with it at all to get it to do what I wanted and no more — it autoset really sensible headers, didn’t set any unsensible headers, and didn’t complain at all when I overrode its convenient automation.  A RESTful interface in Grok is implemented as a distinct layer or skin, which I think is a really good idea: so to access /foo/bar RESTfully you use a URL like “++rest++mylayer/foo/bar“ — it lives in its own completely separate namespace so you can have a RESTful service and a user-facing web application sit side by side without interfering with each other.

To make for a smoother deployment next time, I’ve thought of a few things:

• we need to write more tests and we need to run them; when I joined the firefighting of the 0.9.7.7 deployment, the unit and flunc tests were broken in several spots and I suspect they were broken when the site was deployed; also, there were some things like the password request form that had a runtime error and would have been easily caught had there been a test for it
• we need to remember to merge fixes between svn release branches and the trunk; there was at least one bug that was fixed in the trunk but not merged into the release branch and I believe there were some things that were fixed in the branch but not merged to the trunk; this can get complicated because when you fix a bug you need to fix it on both the branch and trunk and be able to run the tests on both to make sure you haven’t broken anything
• when using trac, I found it useful to specify what site the bug was found in and what release of opencore was currently running on the site; after fixing it, I would record both the trunk and branch rev number that I made and then post another entry when it was deployed and verified on the live site and only then would I close it; this last step might only be appropriate during “firefighting” mode
• finally, I think when we are firefighting, having frequent standing meetings, as we did do, is helpful

Lately I’ve been hearing people talk about a service-based architecture to make our UI easier to work with.  To summarize my understanding of a service based architecture:

• Back end applications have explicit over-the-wire APIs.  That is, you fetch data from them, and render pages based on that data.  The data forms a stable API.  The actual data might be XML or JSON, and applying changes might be RESTful PUTs, or might be RPCish POSTs.
• We develop a single application that is the skin on all our backend applications.  All the UI goes there.  We can use coding conventions to apply UI conventions universally in our site.  Other sites could implement other applications using our backends but with their own frontend UI application.

I believe (strongly) that this is a bad path to go down, and it’s a huge architecture change in response to resolvable development issues.

The fronting UI application is perceived of as simple.  But how simple is it really?  How easy can it be to write an edit screen in this UI application?  If it’s going to give us flexibility in our UI work, that means that often the frontend requests will not be isomorphic to the backend requests — for one save, or even one page render, there could be a large number of backend requests, and complicated ways to aggregate and join the results of those requests.  Does Latest Activity get the last 10 items from everything that could produce items, then choose the real top 10?  How do you configure all the items together?  More XML configuration?  TOPPCML?  If you implement a form that allows editing data from several backends at once, what happens when one of those backend applications gives an exception on save?  When you consider validation the surface area of these individual applications starts to become large.  Given a page, how do we find the code that needs to be changed, if a change can’t be done solely in the UI layer?  How do we we manage the (*very* common) case when both frontend and backend have to be updated in concert?  All these seem like the hard parts of application development.  We don’t want to increase our hard work.

There’s three reasons I see for the interest in a service-based architecture:

1. You can mock out the backends and get a certain amount of stability between frontend and backend, and development can occur in parrallel.  I’m suspicious of whether this is really possible, but it’s a common desire.  Mocking backends is in substantial part a workaround to avoid development deployments, and I hope we don’t need to avoid development deployments anymore.
2. It centralizes the integration of our applications.  Rob Marianski talks about the issue with transclusion and Twirlip.  This is a valid problem.  I’m not sure about the solution, though in some cases we have tools to resolve this (e.g., with Cab).  If our applications have certain bindings to each other, or special configuration or plugins that we have to implement to make them less closely bound, I think this is easier than a service-based architecture.  Basically I’m advocating ad hoc extensibility in our applications rather than a single framework that encompasses all kinds of potential means of extension.
3. It centralizes UI work, because there is a canonical location for templates, CSS, class definitions, etc.

We’ve been working on 1.  2 I admit is hard.  I think 3 is quite resolvable, and that we should resolve it.

To handle UI work it is important to know (a) what files are used to generate the UI, (b) what those files do, and (c) how those files can be changed. 

(a) can be resolved if we included template filenames in all template renderings when some development switch is turned on.  I can tell you how to do this in Pylons applications, and I’m sure someone can figure something out for Zope too, and hopefully even PHP.  Additionally Transcluder and Deliverance would do the same things for their page composition.  Maybe a bookmarklet would reveal the names (since they can mess up rendering if always displayed).

(b) is similar: we include information on the objects used to render the pages (the Zope view, the variables in the pylons.c object, and… all global variables in PHP).

(c) is probably just a matter of implementing (a) and relying on people to edit files in the proper place.  It would also be possible to implement External Editor support.  This is a Zope ZMI product, but its API is actually extremely simple — serve a page with a magic content type that points to the URL where the page source is located (plus an optional cookie auth token), External Editor fetches that and opens an editor, and whenever you save External Editor will PUT the new content to that same location.  I used External Editor a lot when I was doing Zope ZMI development, and it made the experience tolerable.

But that editing only applies to changing the main-line code.  We also want to change UI for local deployments.  To facilitate local modification (e.g., nycstreets.org) we should have a search path for all template loading so that templates can be copied for per-deployment customization.  Of course these copies can also go in svn, and we can use normal merge techniques to keep them working as the original templates are modified.  I also believe strongly that it should be possible to put Real Code into templates; while this isn’t a good smell, it’s an entirely acceptable way to do local modifications.  I remember hearing that even ZPT has the potential to run arbitrary Python with <script language=”python”>, though it is disabled in nearly everyone’s setup.  But we could enable it, if the rumor turns out to be true.  (If it turns out to be false, we could also just look for a neighboring .py file and exec it.)

Service architectures take XML or JSON documents and then transforms and aggregates them into something you render.  As the underlying documents change the transformations have to be updated.  But lacking a clear canonical original transformation (like the original template) this seems harder to maintain.  And ultimately what the transformations are doing is basically equivalent to a template.  Templates transform the input variables into rendered HTML.  The original templates have access to richer objects and methods than what a document transformation can do.  We have the ability to improve our stack to support Rollie and ourselves when doing UI work.  We shouldn’t be shy about making these kinds of infrastructure improvements.

A lot of times at the Snow Sprint I was struck by a feeling that I was in the middle of a group of people who were really doing open source right. So on the plane home I started thinking about what that meant and why I haven’t felt that way at TOPP.

Well, a lot of us have said at various times that our attempts at doing open source development have often been frustrating and rarely seem to be worth the effort. Coming out of the sprint, I think that’s because we are missing a very big piece of the “True Spirit of Open Source”. Doing open source properly is not just a matter of putting the GPL on our code—it is not about making a legal contract but about making a social one. It is about committing to reuse and collaboration both internally and externally and it is well worth the effort.

Writing, finding, and using software all take time and care and with someone paying for our time and care we ought to think of these acts as investments. We should make the most of those investments by seeking code reuse and encouraging collaboration. The more we encourage heavy use of software and heavy reuse of code, the better our ability to improve and solidify it. Whether or not we’ve read Dreaming in Code (though everyone here should—as soon as possible—seriously) we all know that software is hard to do well. It’s full of unexpected behavior, prone to breaking when you touch it, and rarely optimized. We can improve it only by testing and exercising it: give it to people to use, poke it in new ways, put it under stress, see what breaks, fix it and don’t let it break in the same way again.

Laziness is a virtue

Because software is expensive to write, it is most responsible of us to write as little as we can. And because software is hard to write, it’s rarely done once it’s written; most software has tremendous room, and often necessity, for improvement. So our goal should be to maximize the impact of every improvement we make. This primarily means seeking to increase the breadth of our code’s effect: convincing other people to use our software, sharing code across multiple products, and reusing products in as many ways as we can across everything we develop.

To give a specific, if vacuous, example, we have five or more different pieces of code deployed on OpenPlans that generate and send email to users: Listen sends mail, Twirlip sends mail, WordPress sends mail, and Opencore sends mail in at least two different ways. Suppose, instead, that we had a single shared way to send mail to users. Aside from the immediate benefit of less overall knowledge to store, the different mail-sending needs of all these applications would force us to develop a stable and reliable method for sending mail, and when heavy usage of one of those dependent applications inevitably reveals an inefficiency or bug in the mail-sending code, multiple otherwise unconnected parts of our site would instantly benefit from our resulting improvements. Put that same mail-sending code on two or more websites and encourage other developers to deploy it in their applications, and we can tremendously increase both the number and variety of stress points and the magnitude of impact of each strengthening, both the opportunity and the payoff for improvements.

So maximizing adoption of every piece of software by both users and developers is a forward-thinking and cost-effective stance. It gives programmers the most opportunities to strengthen their (buggy, brittle) code and ensures that every (hard, pricey) act of improvement has a large and wide-reaching effect.

We’re not alone

It is not only creating software that needs be treated as an investment. In a true open source environment using others’ software, too, is an investment that we should make the most of. When we choose to depend upon existing free technology—in any way: WordPress, Xinha, Zope, Pylons, Trac—we will benefit ourselves and others by doing all that we can to improve it, participate in its communities of users and developers, and encourage the growth of those communities. The core reasons are the same as with our own code: the more eyes and hands that are on a piece of software, the better it can become.

There is, of course, an additional benefit to encouraging outside developers, namely lessening our internal development and maintenance burden by gaining volunteer contributors. When we create our own software we thus have the additional responsibility of building these communities from nothing. But when we are dealing with existing open source software this goal is all the easier to achieve. A community of developers already exists; our only responsibility is to gain admittance. We should treat our lack of commit rights not as an obstacle, but as a challenge.

Alongside the fuzzy feelings that come with earning the respect of an established group, there is, in keeping with the best of the open source mentality, great potential for everybody to win when we are welcomed into an existing community. We stand to reap the benefits of the efforts of the intelligent, experienced and caring people who develop the software that we have decided is worthy of our use, and we gain the potential to influence the path and priorities of their development with our input and interest. They, meanwhile, gain the feedback and work of a new set of smart (and funded) contributors with a real interest in the success of their project.

Finding those unicorns

I’ve framed this primarily as an economic argument to be concrete, but of course there’s more to it than that. The same stance is also about committing to respectful and positive relationships with outside agents; putting our faith in open and collaborative development environments and in the potential for groups of smart, committed people to produce great things even with differing agendas; taking the time and care to make our work as good as it can be and letting the rest of the world be the judge of it. It’s about being responsible consumers who contribute back to the products we consume, and being responsible producers who proudly stand by what we produce. And then there’s our aspirations at TOPP to do good and produce real positive change: all the time and money we can save by reusing and building on existing solutions can be diverted toward new projects to do even more good in the world, with cascading effects.

So let’s build and get involved with communities, submit and fix bugs in others’ code, give thanks and contribute ideas to the people whose software we use.

Let’s release Flunc, Externalator and Deliverance to the world for real, with welcoming websites and helpful documentation, and encourage people to use them and talk to us about them on mailing lists and IRC channels. Along with everything else it’s a good way to put our software to the test: to put a different spin on the dogfood concept, if no one else wants to use our software, why should we? Let’s keep up with Xinha development and submit behavioral patches to its authors instead of keeping our work isolated on our “vendor branch.” Let’s tell people about the recommendations of our smart interaction designers and submit our implementations of that advice; how many open source software communities have an excess of input from web software designers? Let’s release WordPress and Trac plugins to their wider communities and let’s solicit contributions and feedback to Listen and Opencore.

In short, let’s try to practice better open source. After all, we can only bring the open source ethos and philosophy to the broader world after we get it right in our software.

I try my best to do whatever I can to make TOPP successful. I am a programmer. I think my business card says Software Engineer, but a programmer by any other name. I am neither inclined nor qualified to make decisions at an organization-wide level. This includes, to a very large degree, deciding what I’m working on, deciding what would make openplans.org and nycstreets.org an awesome website, or deciding what other people should be working on. I try to offer whatever observations or ideas I have that might be actionable or even just notes to keep in mind for the future. I’ve been told too many times that I shouldn’t worry about TOPP. And largely I’ve decided to agree with this. I will continue to offer observations and ideas for TOPP outside of the little corner that I sit in and spend several months, as long as I feel they will be received in a positive light and are worthwhile to offer. I would love to participate in architecture discussions or any strategic discussion where I have something to offer and are qualified to be a part of. But my job is to take my assignment and to execute it in the manner in which I judge to be best weighted by my understanding of my assignment. My job is to offer feedback on what I have been assigned to do in a constructive manner in order to try to develop an understanding of what is required and desired of me and also within the scope of my understanding of other TOPP goals and plans, both software and beyond. My job is to make decisions directly related to my (typically programming) tasks while soliciting the feedback of others to ensure that it fits in within our plans. My job is to keep people informed of what I am doing. These I will do to the best of my ability. More coordination than that I will not do unless specifically told to do.

[I was told this was just relisting the problems and not really providing the solutions.  Maybe this is true.  I decided to post it as a blog instead of an email response]

 

What we should learn from this experience:                                                                                                                                        

                                                                                                                                                                                  

* we have lots of problems.  It is not okay to ignore them.  We are                                                                                                               

  aware of what our problems are.  We haven’t dealt with them.  We                                                                                                                

  shouldn’t be surprised that something like this would happen.                                                                                                                   

                                                                                                                                                                                  

* we have talked about our tests being in poor condition for some                                                                                                                 

  time.  We haven’t devoted any personnel or time to fix our testing                                                                                                              

  strategy. I think the fact that our site horribly broke demonstrates                                                                                                            

  that tests are user facing features.  Just fixing the tests and writing more                                                                                                    

  tests is not enough.  We need to seriously examine our whole                                                                                                                    

  methodology for testing and QA.  We have experts in this area.  Lets                                                                                                            

  let them do this (assuming they want to) with the rest of us.                                                                                                                   

                                                                                                                                                                                  

* much of our code has known issues.  we have talked about them for                                                                                                               

  some time.  We have avoided working on fixing them in an effort to                                                                                                              

  add new features.  Building on a rickety infrastructure is just                                                                                                                 

  plain stupid.                                                                                                                                                                   

                                                                                                                                                                                  

* we aren’t coordinated.  This is not okay.                                                                                                                                       

                                                                                                                                                                                  

* Bryan has asked for several features and complained about several                                                                                                               

  dire bugs.  We have largely ignored him.                                                                                                                                        

                                                                                                                                                                                  

* not doing a thing because it is hard, boring, time-consuming,                                                                                                                   

  non-glorious or painful is not a good reason                

Robert pointed out that our “microapp” architecture has the potential to become extremely hard to work with and isn’t really giving us any benefits in the process, since we’re not actually decoupling any of the components. I’d like to think more about that, but in the meantime, I’ve been thinking about what I think would be a really compelling use for proper microapps on our site that wouldn’t be increasing our administrative burden in the way Robert talks about.

(Incidentally, just to be clear, when I talk about proper microapps, I mean simple UI-less applications with a very stable API over HTTP that provide a single service intended to be plugged in to other contexts.)

So, we’re trying to use (quasi-)microapps to build our site on the backend, but I think microapps would be most valuable as services we provide *for users* instead. It seems more compatible with the microapp tendency towards runtime configuration over HTTP; while we end up bending the code and our builds to configure these applications to set up our site at “build and compile” time, users could just configure the applications for themselves on an already-running site to gain additional and personalized functionality.

Offered as user facing tools, these microapps would provide extremely flexible functionality beyond what we offer universally — ways for individual users to layer their own custom configurations on top of the site. Transcluder is a simple example; by just turning on a sitewide transcluder, we would allow users to build very flexible componentized webpages. I think an HTTP request hub like Cabochon could be another really powerful service for users: while we handle our own events internally with code and configuration, users could configure a set of individually saved hookups between HTTP events and resources. So one user could set up a rule: “When I publish to the project’s blog, mark my `write blog post` task as completed” while someone else sets up a rule that “When anyone edits this wiki page, create an account page message for me.” Throw in a cron-like microapp to schedule HTTP requests and now users can have a new “Write weekly blog post” task created every week and more.

Admittedly I’m doing a tremendous amount of hand-waving here (authorization? authentication? request interception? recursive requests? — implementation details!) and I haven’t come up with very compelling use cases, but I don’t think it’s impossible and I think the real potential of the idea is that *we don’t have to come up with the use cases* — people could come up with really interesting combinations of events far beyond anything we’d want or need to build into our software.

Another big advantage here is that since these microapps would have very well-defined APIs that work over HTTP, we could actually deploy them with minimal user interfaces and gradually build up UI over time. “Power users” would immediately get to experiment with the tools immediately, and we could even use their feedback to determine common usage patterns to inform the friendlier interfaces. So we can turn on Transcluder without any UI, and people could immediately start using it by editing a bit of HTML; then, as a first pass at UI, we could build a switch into the Xinha linking interface to toggle the “rel=’include’” attribute; and eventually we could design a much more sophisticated UI (perhaps leaving the “design-less” interface available underneath in the same way that WYSIWYG editors allow you to drop down to HTML) but without that design ever being a bottleneck to deployment or usage by at least some people. Likewise an event hub could start out by letting people input URLs and HTTP verbs, and eventually build a descriptive interface which lets people connect “things to do” from a list.

I think this would be a really powerful way to add, essentially, arbitrary flexibility to a site without having to hard-code it, and without having to come up with individual use cases, instead offering the tools to users and letting them come up with their own uses. And since we really could treat it as a layer on top of our software, which we don’t actually have to know anything about the use of, it wouldn’t be adding to our administrative burden, since we wouldn’t have to worry about the connections between microapps. I don’t think it replaces the need for some sort of internal connection between features and data — we can inevitably come up with much more tightly integrated services on the backend, and in any event I’m sure there are commonly useful hookups which we ought to just provide directly. But it would be a great way to provide functionality beyond what we decide to offer or what occurs to us to implement.

I was originally blogging on my harddisk using emacs.  As whit pointed out, this isn’t really ‘blogging, since its not on the web.  So I wrote an app called bitsyblog (which I’ll talk more about + use later).  Still haven’t deployed it yet and really don’t want to take more sprint time to do so, though its essentially done.  So I’ll do some old fashioned xinha based blogging on our site.  Before posting several days of content I’d like to point out why I like the idea of bitsyblog (which basically blogs with a POST) versus TTW blogging.  To write this blog, I have to navigate to openplans and this blog page, which is, what a minute (so there’s no point in writing a blog if its just a quick observation, unless its really important).  Its a context switch — if I’m on the command line or in emacs, I have to quit what I’m doing to write a one-line blog.  Also, I can’t use my editor of choice.  Anyway, enough complaining, more blogging.

———————

And some more stuff.  It kinda degenerated into test posts somewhere.  I’ll probably for the time being use bitsyblog to blog locally and then do copy + paste to here.  Other than that, things are great!  Now I’m back to working on the KSS commenting app.  KSS is cool and so far I’ve learned more about JS from KSS than I ever knew before (which isn’t saying much).   But progress is slow for not knowing it more.  Still, learning, doing some demo stuff.