Rob Miller has been thinking about zc.buildout lately, and providing a buildout-based bootstrap for our stack (even if it only calls out immediately to fassembler). This got me to thinking, and I finished a reflection post about fassembler.
At the end I almost talked myself into refactoring stuff to make the good parts of fassembler usable in zc.buildout, so that we could move over to that in the future if we want to. Then… right at the end… pow! Files! All files, only files!
What if our build process just built files? Relocatable files. Not-system-dependent files. You could tar the whole thing up, copy it to another location or server, unpack it, and get the exact same site. That tarball would basically be equivalent to the build itself.
It’s not that App Engine seems particularly awesome. More that it seems like it might be Just The Right Kind Of Dumb. It’s that this makes it so easy to think about the build, about what it does, about stuff like the workflow for testing and deployment and rolling back. Easy to explain to other people, easy to audit, easy to debug.
The problem of course is that it requires pushback into other products. Lots of other products. But we can do that.
Thoughts?
Heh… interestingly, this “we’re just pushing files around” philosophy is what buildit, which we tried at one point, claims to be about.
My first reaction to this is cautiously negative, but then I’ve gotten comfortable with our current build process, so I’m a little reluctant to mix things up too much, especially so soon after things have settled. And, of course, we do depend on some C code, which needs to be recompiled for each platform.
I’m interested in hearing more, though. How do you imagine handling the binary executable stuff? And I’m not sure what you mean by “pushback”… can you elaborate?
Comment by ra on June 20, 2008 at 12:45 pm
By “pushback” I mean making changes in the code to facilitate the build, instead of making the build always facilitate the code as it exists. We’ve already done some of that.
As for BuildIt, well… it’s also what filemaker does. Different implementations.
The change could be as incremental as a change to zc.buildout. That is, we don’t have to make our build process instantly relocatable. Some things like virtualenv would have to be changed, and .pth files. For simple projects this is pretty much enough. For other cases we might need to use an environmental variable to point to the base directory (which we could make sure is set in the startup scripts) and then make sure configuration and such can use things like %(BASE_DIR)s. Another option is to have a kind of fix-up script(s), which would write any files that needed to have paths rewritten.
For platform needs one option is things like platform-dependent eggs, where we ship binaries for everything. This is what Macs use for a lot of platform issues, I think. Unfortunately there’s small issues that can cause problems, like if Python was compiled for UCS2 or UCS4 internal unicode. One option for cases like this is a kind of lazy build system, where on startup we make checks and recompile or otherwise manipulate the environment based on things we detect.
All of which seems like a lot of trouble, but of course only applies to small subsets of the software we are using — most of it is easily relocatable. I’m not *sure* the work gives enough benefit. But there’s something intriguing about the concrete nature of the build in that case. For example, I think it’d make it much easier to figure out our workflow when setting up staging and production builds. Or you could do diffs and get an accurate idea of what’s changed. Or easily archive exact versions of the complete build.
Comment by ianb on June 20, 2008 at 2:03 pm