-
I really don't get our build system. I don't think anyone does. What's up with buildit namespaces? What's up with commands vs. tasks? What are the represent and check methods on commands? Who knows...? (I suppose Chris M. does.) And I can't help but feel that the problem we are trying to solve really isn't that hard. It's important, but just not *hard*. Shell scripts would almost be more clear. So I'm considering, maybe this weekend, tossing together something that I feel more comfortable with. Most of the tools to build this are available, so I think it can be just a some glue, and it overlaps with paster create and some other stuff I've done so I feel like I have an intuition for what it should do. Debuggability and transparency are the most important goals; hopefully everything else can flow from there over time and by demand. Obviously some builds have to be developed alongside the build system, as a kind of basic test of does-this-do-anything-that-matters. I'm not sure what builds to do; obviously opencore is one of the more tricky ones, though stuff like doing database setup in TaskTracker has its own complexities. What follows are some rough notes about goals/design of the system: hese commands are informed by configuration. These are the basic commands: * Copy a tree of files * Run a generic shell command * Create a virtualenv * Install packages there * Do an svn checkout The last three are kind of subsets of a generic shell command. Shell commands will be run *carefully*, just like ``paster`` already does: non-zero exit codes will not be ignored, output will not just spew across the screen. There are no dependencies; all tasks are run in order. It is up to the task to save work if it can do so (though it's far more important that it get everything correct than attempt to save work). An example of saving work might be using ``svn up`` when a checkout exists, instead of wiping the checkout and doing ``svn co`` (that's actually a safety concern as well, as there may be updates in the svn checkout). Obviously there's just the option to write Python code too. Commands have these properties: * They can be run/executed * They can (sometimes) detect differences between the current build and what they would do; this might not be accurate (e.g., you can't really tell what an arbitrary shell command *would have* done), but can be used for auditing a build, or a change in a build Configuration works like: * The configuration file can indicate that it should bring in values from other locations, similar to how zc.buildout works * The command setup can do the same thing, overlapping files (typically based on command-line options, or global config file locations) * The build itself can modify the configuration in-place (e.g., to setup default calculated values) * The resulting configuration can be displayed and inspected Logging works like: * Very explicit logs are written to disk * Very abbreviated log messages are written to the console (except with -v, of course) * Logging messages will be grouped, so in case of a failure all messages from the group leading up to the failure can be displayed * I may look into some sys.trace and sys._getframe hacks to show the flow of the code in the logs too; I'm not sure yet The command-line scripts will generally take these options: * A config file that provides a master set of overrides. This is where the global config goes too * Explicit setting overrides * A --simulate option * A --display-config option (giving the user an opportunity to confirm the config and abort) * Any extra options that the build adds; all options get resolved in the config fixup stage Checking out from svn branches will be handled with options, like ``--opencore-branch=geolocation``; this will write to the ``opencore_branch`` setting, and the default value for this will be ``trunk``. And there will be some routine available in the templating, so you can do ``https://svn.openplans.org/svn/opencore/{{opencore_branch|svnlocation}}`` in a substitution. I'm thinking of using Tempita for all string substitution (except on-disk filenames, which will be like paster create). I.e., even using it for little inline strings. Mostly for the availability of expressions like this example.- Thread Outline:
-
-
Re: Our build system
by
k0s
- Re: Our build system by whit <d.w.morriss@...>
- Re: Our build system by slinkp
-
Re: Our build system
by
k0s
-
I support replacing the build system for the following reasons: * no one understands it, possibly including me * everyone complains to me about it; i'd rather them complain to someone else * everyone has a different idea of how it should work; instead of trying to drive this train, i'd rather have someone else do it * for reasons listed above, it is evident that my architecturing of the system is poor; maybe i'm just a poor designer For further ranting, see also: http://www.openplans.org/projects/opencore/jeff-s-blog-post-october-24-2007 Okay, so these responses are somewhat tongue-in-cheek. But seriously, if someone else wants to work on building and deploying, I'm very glad to get back to (or perhaps 'to', since I feel I've hardly done any of it) web programming and let others be the drivers for building and deploying. Jeff On Thu, Oct 25, 2007 at 04:06:07PM -0500, Ian Bicking wrote: > I really don't get our build system. I don't think anyone does. What's up > with buildit namespaces? What's up with commands vs. tasks? What are the > represent and check methods on commands? Who knows...? (I suppose Chris > M. does.) > > And I can't help but feel that the problem we are trying to solve really > isn't that hard. It's important, but just not *hard*. Shell scripts would > almost be more clear. > > So I'm considering, maybe this weekend, tossing together something that I > feel more comfortable with. Most of the tools to build this are available, > so I think it can be just a some glue, and it overlaps with paster create > and some other stuff I've done so I feel like I have an intuition for what > it should do. Debuggability and transparency are the most important goals; > hopefully everything else can flow from there over time and by demand. > > Obviously some builds have to be developed alongside the build system, as a > kind of basic test of does-this-do-anything-that-matters. I'm not sure > what builds to do; obviously opencore is one of the more tricky ones, > though stuff like doing database setup in TaskTracker has its own > complexities. > > What follows are some rough notes about goals/design of the system: > > > > hese commands are informed by configuration. > > These are the basic commands: > > * Copy a tree of files > * Run a generic shell command > * Create a virtualenv > * Install packages there > * Do an svn checkout > > The last three are kind of subsets of a generic shell command. Shell > commands will be run *carefully*, just like ``paster`` already does: > non-zero exit codes will not be ignored, output will not just spew across > the screen. > > There are no dependencies; all tasks are run in order. It is up to the > task to save work if it can do so (though it's far more important that it > get everything correct than attempt to save work). An example of saving > work might be using ``svn up`` when a checkout exists, instead of wiping > the checkout and doing ``svn co`` (that's actually a safety concern as > well, as there may be updates in the svn checkout). > > Obviously there's just the option to write Python code too. > > > Commands have these properties: > > * They can be run/executed > * They can (sometimes) detect differences between the current build and > what they would do; this might not be accurate (e.g., you can't really tell > what an arbitrary shell command *would have* done), but can be used for > auditing a build, or a change in a build > > > Configuration works like: > > * The configuration file can indicate that it should bring in values from > other locations, similar to how zc.buildout works > * The command setup can do the same thing, overlapping files (typically > based on command-line options, or global config file locations) > * The build itself can modify the configuration in-place (e.g., to setup > default calculated values) > * The resulting configuration can be displayed and inspected > > > Logging works like: > > * Very explicit logs are written to disk > * Very abbreviated log messages are written to the console (except with -v, > of course) > * Logging messages will be grouped, so in case of a failure all messages > from the group leading up to the failure can be displayed > * I may look into some sys.trace and sys._getframe hacks to show the flow > of the code in the logs too; I'm not sure yet > > > The command-line scripts will generally take these options: > > * A config file that provides a master set of overrides. This is where the > global config goes too > * Explicit setting overrides > * A --simulate option > * A --display-config option (giving the user an opportunity to confirm the > config and abort) > * Any extra options that the build adds; all options get resolved in the > config fixup stage > > Checking out from svn branches will be handled with options, like > ``--opencore-branch=geolocation``; this will write to the > ``opencore_branch`` setting, and the default value for this will be > ``trunk``. And there will be some routine available in the templating, so > you can do > ``https://svn.openplans.org/svn/opencore/{{opencore_branch|svnlocation}}`` > in a substitution. I'm thinking of using Tempita for all string > substitution (except on-disk filenames, which will be like paster create). > I.e., even using it for little inline strings. Mostly for the availability > of expressions like this example. > > > > > > > !DSPAM:4014,472106ae315941431913854! >-
On 10/25/07, Jeff Hammel <jhammel@...> wrote: > > I support replacing the build system for the following reasons: > > * no one understands it, possibly including me now why is this? can we make it simpler? * everyone complains to me about it; i'd rather them complain to someone > else this is generally a sign of software success. it's called feedback. ;) if you can figure out why they are complaining and make solutions for it and you keep users and their complaints change, you are moving in the right direction. * everyone has a different idea of how it should work; instead of trying to > drive this train, i'd rather have someone else do it In an area as impacted as build(everyone needs a make file pretty much to start with), the system needs to be as powerful as possible and the bar to entry as low as possible. Maybe what we should aim for is making the build software easy to extend rather than all encompassing. In some ways I see this as an input problem. Before the build, rob and I were the build script, dealing with everyones build headaches. This is alot better now, but it's still more than you should be handling. You should be able to point at clear and simple tools and documentation and say DIY. I think the other problem is build software cause folk's working knowledge of integration become atrophied. sometimes they just need to suck it up and figure out how things work. * for reasons listed above, it is evident that my architecturing of the > system is poor; maybe i'm just a poor designer most designs take a number of iteration to get right. We've already thrown a few build approaches away before this: it's a tough problem. In the short wanderings through our build code I've seen some really good stuff that made me to "ah!" and some stuff that confused the shit out of me. I think we should take a hard look at simplifying things and you deserve some real code review and feedback rather than the heaps of customer feedback you've gotten. and we need to lower that bus number and folk's slackjawed dependence. I think it's fine and good if you want to do more web programming, but we need you to shepherd whoever follows in your wake. -w -- | david "whit" morriss | | contact :: http://public.xdi.org/=whit "If you don't know where you are, you don't know anything at all" Dr. Edgar Spencer, Ph.D., 1995 "I like to write code like other ppl like to tune their cars or 10kW hifi equipment..." Christian Heimes, 2004
-
On Fri, October 26, 2007 08:12, whit wrote: > On 10/25/07, Jeff Hammel <jhammel@...> wrote: > >> >> I support replacing the build system for the following reasons: >> >> >> * no one understands it, possibly including me >> > > > now why is this? can we make it simpler? I was being somewhat sarcastic, but honestly I don't understand setuptools. I can read the webpages until I'm blue in the face, but there are some bugs (or "features") that confound me. > * everyone complains to me about it; i'd rather them complain to someone > >> else > > > this is generally a sign of software success. it's called feedback. ;) > if you can figure out why they are complaining and make solutions for it > and you keep users and their complaints change, you are moving in the > right direction. > > * everyone has a different idea of how it should work; instead of trying > to >> drive this train, i'd rather have someone else do it > > > In an area as impacted as build(everyone needs a make file pretty much to > start with), the system needs to be as powerful as possible and the bar > to entry as low as possible. Maybe what we should aim for is making the > build software easy to extend rather than all encompassing. > > In some ways I see this as an input problem. Before the build, rob and I > were the build script, dealing with everyones build headaches. This is > alot better now, but it's still more than you should be handling. You > should be able to point at clear and simple tools and documentation and > say DIY. > > I think the other problem is build software cause folk's working > knowledge of integration become atrophied. sometimes they just need to > suck it up and figure out how things work. > > * for reasons listed above, it is evident that my architecturing of the > >> system is poor; maybe i'm just a poor designer > > > most designs take a number of iteration to get right. We've already > thrown a few build approaches away before this: it's a tough problem. In > the short wanderings through our build code I've seen some really good > stuff that made me to "ah!" and some stuff that confused the shit out of > me. I think we should take a hard look at simplifying things and you > deserve some real code review and feedback rather than the heaps of > customer feedback you've gotten. > > and we need to lower that bus number and folk's slackjawed dependence. I > think it's fine and good if you want to do more web programming, but we > need you to shepherd whoever follows in your wake. > > -w > > > > > -- > > > | david "whit" morriss > | > | contact :: http://public.xdi.org/=whit > > > "If you don't know where you are, > you don't know anything at all" > > Dr. Edgar Spencer, Ph.D., 1995 > > > > "I like to write code like > other ppl like to tune their cars or 10kW hifi equipment..." > > Christian Heimes, 2004 > > > > !DSPAM:4014,4721da6d273878362916074! > >
-
-
-
On Thu, Oct 25, 2007 at 04:06:07PM -0500, Ian Bicking wrote: > I really don't get our build system. I don't think anyone does. What's up > with buildit namespaces? Not sure what you're asking really. Pymake had a much simpler design where you just initialized a task with a dictionary of names that it would use for string interpolation. This was pretty clunky to work with. Buildit replaces the dict of names with .ini files containing name = value pairs. This is pretty well spelled out in the Namespaces section of the README (about halfway down, after "Task Hints"). > ... commands vs. tasks? Tasks are more or less equivalent to Make rules. A task does its work via a sequence of commands. That's all. > What are the represent and check methods on commands? represent() is a string to tell the user what the command is doing. For a shell command, that's just the shell command itself. As far as I can tell, check() is for doing an early check of the string interpolation stuff so if there's an exception, it gets raised before the whole build starts. Chris M seems to have a habit of doing this: check = represent > (I suppose Chris M. does.) I'll nudge him about updating the readme, which leaves out a lot of this stuff, and AFAICT lies about using Python callables as commands. I think that's a leftover from the Pymake docs. > And I can't help but feel that the problem we are trying to solve really > isn't that hard. It's important, but just not *hard*. Shell scripts would > almost be more clear. Well, maybe at first :) > Debuggability and transparency are the most important goals; +1. I became "the build guy" at my last job; while I didn't create the build system there (based on pymake, the precursor to buildit), I was almost the only one who spent enough time with it to be able to fix problems that arose. And it was pretty brittle. So most people were afraid to touch it other than to bump version numbers. My number one complaint with pymake was, no way to do a dry run and see what the build was going to do. So, +1 for transparency. It was also pretty hard to get anything useful out of a pdb session; don't know if buildit has improved that. I'm always skeptical of the need to start over from scratch, but I'm not gonna stop you either :-) -- Paul Winkler http://www.openplans.org/people/slinkp/profile yahoo: slinkp23 AIM: slinkp1970-
Paul Winkler wrote: > On Thu, Oct 25, 2007 at 04:06:07PM -0500, Ian Bicking wrote: >> I really don't get our build system. I don't think anyone does. What's up >> with buildit namespaces? > > Not sure what you're asking really. Pymake had a much simpler design > where you just initialized a task with a dictionary of names that it > would use for string interpolation. This was pretty clunky to work > with. Buildit replaces the dict of names with .ini files containing > name = value pairs. This is pretty well spelled out in the Namespaces > section of the README (about halfway down, after "Task Hints"). The questions were mostly rhetorical questions made from frustration. After reading the docs more closely I feel somewhat clearer, but not entirely; and my spelunking in the code and our own usage patterns and the readme docs don't all quite match up. Which isn't a problem of accuracy necessary. > >> ... commands vs. tasks? > > Tasks are more or less equivalent to Make rules. A task does its work > via a sequence of commands. That's all. Maybe if I was familiar with Make it would make more sense; having read the docs a bit more, it seems like a peculiar abstraction, as commands and tasks are really the same thing. There's some technical details; maybe a task can be an abstraction over a command for the purpose of variable substitution. But there's no essential details that I see. >> What are the represent and check methods on commands? > > represent() is a string to tell the user what the command is doing. > For a shell command, that's just the shell command itself. > > As far as I can tell, check() is for doing an early check of the > string interpolation stuff so if there's an exception, it gets raised > before the whole build starts. Chris M seems to have a habit of doing > this: > > check = represent This might be part of the confusion, that different ideas are folded together, and when there's an exception it's unclear what's really intended to happen. E.g., when the name, variable check, and actual invocation of a command are all the same thing. >> (I suppose Chris M. does.) > > I'll nudge him about updating the readme, which leaves out a lot of > this stuff, and AFAICT lies about using Python callables as commands. > I think that's a leftover from the Pymake docs. > >> And I can't help but feel that the problem we are trying to solve really >> isn't that hard. It's important, but just not *hard*. Shell scripts would >> almost be more clear. > > Well, maybe at first :) Yes, clearly not for long. Though it's things like the frequent use of os.system that make me feel like buildit is closer to shell than Python in areas. The docs *say* it raises an exception on a non-zero exit code, but I haven't seen that happen consistently. >> Debuggability and transparency are the most important goals; > > +1. I became "the build guy" at my last job; while I didn't create > the build system there (based on pymake, the precursor to buildit), I > was almost the only one who spent enough time with it to be able to > fix problems that arose. And it was pretty brittle. So most people > were afraid to touch it other than to bump version numbers. > > My number one complaint with pymake was, no way to do a dry run and > see what the build was going to do. So, +1 for transparency. > It was also pretty hard to get anything useful out of a pdb session; > don't know if buildit has improved that. > > I'm always skeptical of the need to start over from scratch, but I'm > not gonna stop you either :-) I'm skeptical too. Another option is zc.buildout, which I believe is much easier to debug and work with than it used to be (I used it very early on, and got frustrated for a lot of the same reasons as I'm frustrated with buildit). Both have aspects that I don't think apply to what we want to do; but being too scared of extra inapplicable features is a bad reason for NIH. If I do this, another feature I'd want is that it runs fast. If it runs fast then I won't have a problem tweaking it. Slow systems get tested less and improved less, I think. Ironically, the whole make idea of BuildIt is intended to make builds faster to repeat, but doesn't really do anything for us. Our builds aren't really file-oriented, because we're working with things like easy_install and poach-eggs that work more declaratively. Optimizing those involves things like caching available eggs and suppressing pypi lookups. I think zc.buildout actually has some of these optimizations. I'd be more inclined to add it to poach-eggs -- like "poach-eggs --collect --cache=some_dir (arguments)" would collect all the eggs, and "poach-eggs --cache=some_dir --no-remote (arguments)" would grab the eggs from the cached location. Maybe something similar for svn checkouts, to reuse existing checkouts. Which of course leads back to zc.buildout some as well. It does a bunch of these things, and has a substantial user base. In contrast there's the mix of tools we have, which do some similar things but factored quite a bit differently; we're not starting from scratch, but the approach is quite a bit different. So I'm not sure if we should adapt ourselves, or just fill out the toolset we have independently. -- Ian Bicking : ianb@...
-
Ian Bicking wrote: > I'm skeptical too. Another option is zc.buildout, which I believe is > much easier to debug and work with than it used to be (I used it very > early on, and got frustrated for a lot of the same reasons as I'm > frustrated with buildit). Both have aspects that I don't think apply to > what we want to do; but being too scared of extra inapplicable features > is a bad reason for NIH. i've played a bit w/ zc.buildout since it's become the standard way of doing things in the Plone world. i hope we don't have to use it. it does the job, but it just feels wrong. this obviously isn't a good enough reason to block using it, but i'd be much happier if our end result was a virtualenv-based installation instead of a buildout. > If I do this, another feature I'd want is that it runs fast. If it runs > fast then I won't have a problem tweaking it. Slow systems get tested > less and improved less, I think. > > Ironically, the whole make idea of BuildIt is intended to make builds > faster to repeat, but doesn't really do anything for us. this isn't exactly true. the most time consuming parts of our build right now are the zope and openplans bundle checkouts, and the zope tree build. i've been pretty stoked to find out that aborted builds pick up where they left off, w.r.t. these steps. > Our builds > aren't really file-oriented, because we're working with things like > easy_install and poach-eggs that work more declaratively. Optimizing > those involves things like caching available eggs and suppressing pypi > lookups. I think zc.buildout actually has some of these optimizations. > I'd be more inclined to add it to poach-eggs -- like "poach-eggs > --collect --cache=some_dir (arguments)" would collect all the eggs, and > "poach-eggs --cache=some_dir --no-remote (arguments)" would grab the > eggs from the cached location. Maybe something similar for svn > checkouts, to reuse existing checkouts. build-opencore does have some flags for using existing svn checkouts, actually. > Which of course leads back to zc.buildout some as well. It does a bunch > of these things, and has a substantial user base. In contrast there's > the mix of tools we have, which do some similar things but factored > quite a bit differently; we're not starting from scratch, but the > approach is quite a bit different. So I'm not sure if we should adapt > ourselves, or just fill out the toolset we have independently. i don't feel strongly about starting over; i think our current setup can be fixed, but i'd also be happy to step back and do a virtualenv-based thing that doesn't use buildit at all. i have an aesthetic objection to zc.buildout; i have kind of a 'bad smell' feeling about it, not about the code but just about the way it's all put together. if it makes the most sense, though, i'll just get a noseplug or something. -r
-
Rob Miller wrote: > Ian Bicking wrote: >> I'm skeptical too. Another option is zc.buildout, which I believe is >> much easier to debug and work with than it used to be (I used it very >> early on, and got frustrated for a lot of the same reasons as I'm >> frustrated with buildit). Both have aspects that I don't think apply >> to what we want to do; but being too scared of extra inapplicable >> features is a bad reason for NIH. > > i've played a bit w/ zc.buildout since it's become the standard way of > doing things in the Plone world. i hope we don't have to use it. it > does the job, but it just feels wrong. this obviously isn't a good > enough reason to block using it, but i'd be much happier if our end > result was a virtualenv-based installation instead of a buildout. > >> If I do this, another feature I'd want is that it runs fast. If it >> runs fast then I won't have a problem tweaking it. Slow systems get >> tested less and improved less, I think. >> >> Ironically, the whole make idea of BuildIt is intended to make builds >> faster to repeat, but doesn't really do anything for us. > > this isn't exactly true. the most time consuming parts of our build > right now are the zope and openplans bundle checkouts, and the zope tree > build. i've been pretty stoked to find out that aborted builds pick up > where they left off, w.r.t. these steps. I have some ideas of how to optimize this as well, somewhat more generally. Perhaps. I dunno. >> Our builds aren't really file-oriented, because we're working with >> things like easy_install and poach-eggs that work more declaratively. >> Optimizing those involves things like caching available eggs and >> suppressing pypi lookups. I think zc.buildout actually has some of >> these optimizations. I'd be more inclined to add it to poach-eggs -- >> like "poach-eggs --collect --cache=some_dir (arguments)" would collect >> all the eggs, and "poach-eggs --cache=some_dir --no-remote >> (arguments)" would grab the eggs from the cached location. Maybe >> something similar for svn checkouts, to reuse existing checkouts. > > build-opencore does have some flags for using existing svn checkouts, > actually. Yes, but they are more ad hoc than I'd like. They only apply to a couple packages, and only in the one build. I'm thinking about adding it as some kind of option to poach-eggs, along with an egg cache. My thought is that you'd just tell poach-eggs one or several directories, and it can figure out what requirements those can satisfy using pkg_resources. Or... something. I'm not sure exactly how it should work. But there should be an easy way to use checkouts and branches with builds, and I think poach-eggs can help there. Also, it lets me kind of work on this while deferring a rewrite for a little longer ;) >> Which of course leads back to zc.buildout some as well. It does a >> bunch of these things, and has a substantial user base. In contrast >> there's the mix of tools we have, which do some similar things but >> factored quite a bit differently; we're not starting from scratch, but >> the approach is quite a bit different. So I'm not sure if we should >> adapt ourselves, or just fill out the toolset we have independently. > > i don't feel strongly about starting over; i think our current setup can > be fixed, but i'd also be happy to step back and do a virtualenv-based > thing that doesn't use buildit at all. i have an aesthetic objection to > zc.buildout; i have kind of a 'bad smell' feeling about it, not about > the code but just about the way it's all put together. if it makes the > most sense, though, i'll just get a noseplug or something. Well, no one really *wants* to use zc.buildout. I don't. So I'm guessing it isn't going to go anywhere. I still feel slightly guilty about not using it, just out of a generally NIH fear. But eh. From you description of repoze, that sounds very similar to what I was thinking of; really something more like a simple script with useful helpers (e.g., maybe paste.script.copydir instead of the Skel stuff in buildit). -- Ian Bicking : ianb@... : Team TOPP Unite! Form of: Willow!
-
On Oct 27, 2007, at 4:56 PM, Ian Bicking wrote: > From you description of repoze, that sounds very similar to what I > was thinking of; really something more like a simple script with > useful helpers (e.g., maybe paste.script.copydir instead of the > Skel stuff in buildit). all repoze.project does is set up the virtualenv, install the initial egg, and trigger the entry point. after that it's all up to us, paste.script.copydir seems like a fine way to push the config files around. also, it might make sense to not use setup.py dependencies much (or at all), but to instead use separate requirements files and to depend on poach-eggs as much as possible to handle configuring the environment. then we could pass in the locations of specific requirements files on the command line to get different installations; a release branch would have it's own requirements file, you could switch versions in an existing environment just by re- running poach eggs against a different file. that's all not terribly different from how zc.buildout works, actually... ;-). with buildout, though, i think you _have_ to modify the declarative requirement files to change the rig; you don't have the ability to quickly install something into the environment and keep going like you do w/ virtualenv. -r
-
-
+1 to everything Ra said. I've also played a little with zc.buildout in the experimental stages of the build. While I didn't find it buggy, it was clunky. Compare to buildit which has ten files that are easy to read (most of the time). But as Ra said, if its not me behind the keyboard, I don't care what internals we use as long as they don't make the users' (i.e. us) lives difficult. Jeff On Fri, October 26, 2007 16:19, Rob Miller wrote: > Ian Bicking wrote: > >> I'm skeptical too. Another option is zc.buildout, which I believe is >> much easier to debug and work with than it used to be (I used it very >> early on, and got frustrated for a lot of the same reasons as I'm >> frustrated with buildit). Both have aspects that I don't think apply >> to what we want to do; but being too scared of extra inapplicable >> features is a bad reason for NIH. > > i've played a bit w/ zc.buildout since it's become the standard way of > doing things in the Plone world. i hope we don't have to use it. it does > the job, but it just feels wrong. this obviously isn't a good enough > reason to block using it, but i'd be much happier if our end result was a > virtualenv-based installation instead of a buildout. > >> If I do this, another feature I'd want is that it runs fast. If it >> runs fast then I won't have a problem tweaking it. Slow systems get >> tested less and improved less, I think. >> >> Ironically, the whole make idea of BuildIt is intended to make builds >> faster to repeat, but doesn't really do anything for us. > > this isn't exactly true. the most time consuming parts of our build > right now are the zope and openplans bundle checkouts, and the zope tree > build. i've been pretty stoked to find out that aborted builds pick up > where they left off, w.r.t. these steps. > >> Our builds >> aren't really file-oriented, because we're working with things like >> easy_install and poach-eggs that work more declaratively. Optimizing >> those involves things like caching available eggs and suppressing pypi >> lookups. I think zc.buildout actually has some of these optimizations. >> I'd be more inclined to add it to poach-eggs -- like "poach-eggs >> --collect --cache=some_dir (arguments)" would collect all the eggs, and >> "poach-eggs --cache=some_dir --no-remote (arguments)" would grab the >> eggs from the cached location. Maybe something similar for svn >> checkouts, to reuse existing checkouts. > > build-opencore does have some flags for using existing svn checkouts, > actually. > >> Which of course leads back to zc.buildout some as well. It does a >> bunch of these things, and has a substantial user base. In contrast >> there's the mix of tools we have, which do some similar things but >> factored quite a bit differently; we're not starting from scratch, but >> the approach is quite a bit different. So I'm not sure if we should >> adapt ourselves, or just fill out the toolset we have independently. > > i don't feel strongly about starting over; i think our current setup can > be fixed, but i'd also be happy to step back and do a virtualenv-based > thing that doesn't use buildit at all. i have an aesthetic objection to > zc.buildout; i have kind of a 'bad smell' feeling about it, not about the > code but just about the way it's all put together. if it makes the most > sense, though, i'll just get a noseplug or something. > > -r > > > > -- > Archive: > http://www.openplans.org/projects/opencore/lists/opencore-dev/archive/200 > 7/10/1193429982636 > To unsubscribe send an email with subject unsubscribe to > opencore-dev@.... Please contact > opencore-dev-manager@... for questions. > > > !DSPAM:4014,47224cb0186261137850744! > > >
-
-
On Fri, October 26, 2007 15:01, Ian Bicking wrote: > Paul Winkler wrote: > >> On Thu, Oct 25, 2007 at 04:06:07PM -0500, Ian Bicking wrote: >> >>> I really don't get our build system. I don't think anyone does. >>> What's up >>> with buildit namespaces? >> >> Not sure what you're asking really. Pymake had a much simpler design >> where you just initialized a task with a dictionary of names that it >> would use for string interpolation. This was pretty clunky to work >> with. Buildit replaces the dict of names with .ini files containing >> name = value pairs. This is pretty well spelled out in the Namespaces >> section of the README (about halfway down, after "Task Hints"). > > The questions were mostly rhetorical questions made from frustration. > After reading the docs more closely I feel somewhat clearer, but not > entirely; and my spelunking in the code and our own usage patterns and the > readme docs don't all quite match up. Which isn't a problem of accuracy > necessary. > >> >>> ... commands vs. tasks? >>> >> >> Tasks are more or less equivalent to Make rules. A task does its work >> via a sequence of commands. That's all. > > Maybe if I was familiar with Make it would make more sense; having read > the docs a bit more, it seems like a peculiar abstraction, as commands and > tasks are really the same thing. There's some technical details; maybe a > task can be an abstraction over a command for the purpose of variable > substitution. But there's no essential details that I see. > >>> What are the represent and check methods on commands? >>> >> >> represent() is a string to tell the user what the command is doing. For >> a shell command, that's just the shell command itself. >> >> As far as I can tell, check() is for doing an early check of the >> string interpolation stuff so if there's an exception, it gets raised >> before the whole build starts. Chris M seems to have a habit of doing >> this: >> >> >> check = represent > > This might be part of the confusion, that different ideas are folded > together, and when there's an exception it's unclear what's really intended > to happen. E.g., when the name, variable check, and actual invocation of > a command are all the same thing. > >>> (I suppose Chris M. does.) >>> >> >> I'll nudge him about updating the readme, which leaves out a lot of >> this stuff, and AFAICT lies about using Python callables as commands. I >> think that's a leftover from the Pymake docs. >> >>> And I can't help but feel that the problem we are trying to solve >>> really isn't that hard. It's important, but just not *hard*. Shell >>> scripts would almost be more clear. >> >> Well, maybe at first :) >> > > Yes, clearly not for long. Though it's things like the frequent use of > os.system that make me feel like buildit is closer to shell than Python in > areas. The docs *say* it raises an exception on a non-zero exit code, but > I haven't seen that happen consistently. I haven't had an issue with this. Though buildit is supposed to be more shellish than python. It is make for python. Having familiarity with make, I like this sort of thing (about the only things I hated for make was the damn horrid file format and the inability to do any sort of scripting. neither of these are peculiar to buildit). topp.build.lib is again intended to be a wrapper around buildit for our purposes. I suppose it hasn't done its job there, or we wouldn't be having this conversation. >>> Debuggability and transparency are the most important goals; >>> >> >> +1. I became "the build guy" at my last job; while I didn't create >> the build system there (based on pymake, the precursor to buildit), I was >> almost the only one who spent enough time with it to be able to fix >> problems that arose. And it was pretty brittle. So most people were >> afraid to touch it other than to bump version numbers. >> >> My number one complaint with pymake was, no way to do a dry run and >> see what the build was going to do. So, +1 for transparency. It was also >> pretty hard to get anything useful out of a pdb session; don't know if >> buildit has improved that. >> >> I'm always skeptical of the need to start over from scratch, but I'm >> not gonna stop you either :-) > > I'm skeptical too. Another option is zc.buildout, which I believe is > much easier to debug and work with than it used to be (I used it very early > on, and got frustrated for a lot of the same reasons as I'm frustrated > with buildit). Both have aspects that I don't think apply to what we want > to do; but being too scared of extra inapplicable features is a bad reason > for NIH. > > If I do this, another feature I'd want is that it runs fast. If it runs > fast then I won't have a problem tweaking it. Slow systems get tested less > and improved less, I think. > > Ironically, the whole make idea of BuildIt is intended to make builds > faster to repeat, but doesn't really do anything for us. Our builds aren't > really file-oriented, because we're working with things like easy_install > and poach-eggs that work more declaratively. This the main deficiency of buildit to me -- targets are files and cannot be abstracted. Compare to commands which are usually shell commands, but can be python. > Optimizing those involves > things like caching available eggs and suppressing pypi lookups. I think > zc.buildout actually has some of these optimizations. I'd be more inclined > to add it to poach-eggs -- like "poach-eggs --collect --cache=some_dir > (arguments)" would collect all the eggs, and > "poach-eggs --cache=some_dir --no-remote (arguments)" would grab the > eggs from the cached location. Maybe something similar for svn checkouts, > to reuse existing checkouts. > > Which of course leads back to zc.buildout some as well. It does a bunch > of these things, and has a substantial user base. In contrast there's the > mix of tools we have, which do some similar things but factored quite a > bit differently; we're not starting from scratch, but the approach is > quite a bit different. So I'm not sure if we should adapt ourselves, or > just fill out the toolset we have independently. > > > -- > Ian Bicking : ianb@... > > > > -- > Archive: > http://www.openplans.org/projects/opencore/lists/opencore-dev/archive/200 > 7/10/1193425300844 > To unsubscribe send an email with subject unsubscribe to > opencore-dev@.... Please contact > opencore-dev-manager@... for questions. > > > !DSPAM:4014,47223adc147341439371379! > > >
-
-
-
OpenCore Dev
text.html (text/html) 3.9 kB