We did some work this last release with setting up some transclusion. Users can now be notified when tasks they are watching are changed. Implementation-wise, the ui for notifications is being transcluded in.
As far as I know, transcluder was designed to have applications pull in data from other applications in a general fashion. This is a great idea. The problem though, is that the application really does know where it comes from. It needs to know the uri to stick into the include snippet.
Let’s take the twirlip ui as an example. Tasktracker transcludes it on the task page. Opencore transcludes it on the account page. If we wanted to get notified when a blog post changed, we would need to modify wordpress to put in a snippet for twirlip as well. And wordpress would have to know that it’s twirlip doing the transclusion, because it needs to put in the uri for it. If the uri changes for whatever reason, like we decide to put it on another machine, then the configuration for all applications using it would have to change. And if we miss one, then that app will just not pull in the right ui at runtime. It won’t crash or anything since transcluder degrades nicely; we just won’t see it on the page. In some ways that’s worse because there’s no way that we would know about it. It’s not like users are going ton report that they didn’t get the transcluded snippet that they wanted. I guess getting an internal alert when a transclusion fails would get around this, but still.
Now let’s say we want to add a tagging microapp, and have the ability to tag any piece of content. Again, we would have to modify every other app to know specifically about this tagger app to transclude the ui in. And, if we wanted to be notified whenever a new tag was added, the tagger now needs to know about twirlip. And if we wanted to tag some notification types, (I know, unlikely) twirlip would have to be modified to know about the tagging app.
What about a rating app? Or an app that lets you comment on anything? All of these would have to know about each other.
I supposed that we could make some changes in deliverance to make this better. I guess it could know more about whatever it’s pulling in, so the apps themselves wouldn’t have to know about them. Not sure how this would work in practice though. Or maybe transcluder would know these details.
What I would like to do is have a new component dedicated to generating the ui. This app would know about all the other ones, and would essentially pull in everything necessary from the other microapps. The microapps would not generate any ui at all. They would act more like services instead of inter-working applications.
Having the ui generated from one location makes it easier to work with. This isn’t so much of a problem yet, but with many apps generating ui, it wouldn’t be immediately obvious where some html is coming from. This is especially helpful for designers. Having a central file with all the ui just makes it easier to manage. Especially when something goes wrong, or doesn’t look quite right.
Caching should become easier as well. With many apps floating around, each responsible for their own ui, it’s not immediately obvious to me how to do it. The microapps would need to know more about the data that each is generating. Or maybe transcluder would know about each, and it would do the caching. Or we introduce a header that apps can return to signal how long something can be cached.
If we introduce a ui app, then all the other microapps just need to have restful apis, and can return formats that made sense for the data they are working with. This could be json, atom, rss, xml, or even html with/without microformats. If they do generate html, it should be extremely simple, and it should only contain semantic markup. This should help keep the microapps more general and simple, since they wouldn’t have to worry about generating ui in a cross app fashion. Having the ability to use these apps as standalone apps, and integrated with the others at the same time comes up every now and then. The way we are architecting things currently, I don’t see how these apps can be standalone. If you want one, you’ll have to buy most of the others too.
And I don’t think that we shouldn’t use transclusion. Transclusion can just become one method of pulling in html from another app. I don’t think it should be the only way though. I feel the same way about deliverance.
What I’m worried about is that we are making this *the* way that we move data around. It’s more indirect, and it’s not as straightforward, especially when trying to figure out when something goes wrong.
It definitely has it’s uses, and for me it’s most powerful because we can use deliverance and transcluder to pull in data how we like from sites that we don’t own, and make it look like we do own it. It’s a fast way to take a particular set of data from somewhere else and theme it. But for apps that we do own, and plan to customize ourselves heavily, I think using deliverance ends up hurting us in the long run, just because there’s more going on than needs to be. For me, having /help synchronized among nycstreets and openplans was a great example of the power of deliverance. Whether we decide to keep it in the long run or not, I don’t think we’d be able to respond as quickly using a different solution.
In conclusion, I just want to say that we should take a minute to think about how we want to approach adding additional features/ui. It’s still manageable in its current state, but if we add just another app or two that generates ui, things may get very complex very fast.
“What I would like to do is have a new component dedicated to generating the ui. This app would know about all the other ones, and would essentially pull in everything necessary from the other microapps. The microapps would not generate any ui at all. They would act more like services instead of inter-working applications.”
Note that this same design would be equally applicable in a non-microapp-centric architecture. In a single system we could develop pluggable backend libraries (the service microapps) and a frontend layer, either within the same system or in another microapp. I’m with you that this is the best way to go, but I don’t think we need to develop every new feature or service as a microapp — if we write our microapps as proper services and our “single framework” code as proper libraries, we can still keep the UI separate and in one place and choose whether to build a microapp or an in-process library on a case by case basis.
Comment by ejucovy on January 21, 2008 at 7:00 am
I’ll write up a longer response about UI later, as this has come up from several different directions. I think a service architecture is the Hard Way to fix this, and there are some much simpler ways to address this UI issue that are Much Easier (and I’d argue Better).
For some things, like a tagger/twirlip integration, Cabochon is the hub that keeps these pieces from having to know about each other directly.
For some kinds of things we do with transclusion, we can also put the work in Deliverance rules. Right now if you use content_href in Deliverance rules it’s basically equivalent to transclusion but applied site-wide. So it can serve as a kind of integration layer. It’s also possible that some other kind of UI integration layer would be helpful. The top bar hints at something like that, but it would require more thought to make it happen.
Comment by Ian Bicking on January 21, 2008 at 3:31 pm
I think the best solution is in between what we have now, where UI is generated almost entirely by endpoints, and the other extreme of having UI generated entirely by a hub app.
I DO think that some of our templates should be defined near the front of the pipeline, that DVHoster should grow in scope to become more of the central resource. Really, the DVHoster piece should be called “OpenCore”, and not the Plone stuff.
It would also be possible to have DVHoster (or whatever this central service delegation / management piece ends up being called) maintain a transclusion registry of some sort. Then endpoint apps could refer to a DVHoster URI in the transclusion links, isolating them from changes to the actual location and implementation of the transcluded resource.
Comment by ra on February 12, 2008 at 5:26 pm
Well, with SSIs you might do something like set a bunch of “system” variables. I think you can do this as environmental variables or something. Then maybe do (using [] for angle brackets):
[!–#if expr=”$transcluder_item_link” –]
[!–#include path=”$transcluder_item_link” –]
[!–#endif –]
And then define $transcluder_item_link appropriately. Of course SSIs have a data model problem where you can’t put in more than a single item in a variable, and you can’t loop or have any kind of recursive substitution. So there’s a limit to how much you can do. But the condition at least makes it degrade well if you don’t have that variable.
We could do variables with a kind of URI template. E.g., [a href=”{notifications}” rel=”include], and then drop the element if any of the variables used aren’t defined. But the dynamic path doesn’t seem that useful either. Maybe just having external integration rules, like what Deliverance is already doing, as suggested in my other comment.
We could have something like a rules.d/ directory, where any rules in that directory are concatenated to make the final rules. Then Twirlip could drop something in there for TaskTracker, but TaskTracker wouldn’t necessarily have to exist (the rule just wouldn’t do anything then).
Comment by ianb on February 12, 2008 at 10:24 pm