-
Today I was wishing that I could force fassembler to use a specific revision number for things it checks out from svn.openplans.org. This is because when buildbot builds fail, it's often tricky to determine exactly what revision was built, since it takes several minutes between when the build starts and when various things get checked out. Often, more commits are made during that time, and these don't make it to the blamelist, and I have to guess what was actually built and tested. And by the time I hear about the failure, the environment has likely been wiped out for the next build. So I hacked up this branch: https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-20080508 ... which adds an option to fassembler:topp like so: ./bin/fassembler fassembler:topp openplans_svn_revision=15000 ... If specified, this is used for the requirements checkout (if it's on svn.openplans.org), and specifies the same revision for any editable checkouts in *-req.txt, regardless of any revision information in those (again, only for things on svn.openplans.org). This works AFAICT, within some IMO reasonable limitations: * other fassembler: projects don't recognize the option on the command line, so you'd have to edit build.ini if you change your mind later - or re-run fassembler:topp. * re-running fassembler:opencore with a changed revision in build.ini can fail if things have moved around in the repo; first updating requirements/ to that revision should help * things could fail at runtime if there's some incompatibility between the revision and an updated third-party component; hopefully this should be rare, unless maybe you're building really old revisions for some reason. Does anybody object to anything here? If not I'll merge it to the fassembler trunk, and then update buildbot to take advantage of it. -- Paul Winkler http://www.openplans.org/people/slinkp/profile yahoo: slinkp23 AIM: slinkp1970
-
In a requirement file you can add @rev-number to any svn URL. Would that satisfy your need? Maybe poach-eggs --freeze would be useful to save information about the build after it's been built? It adds in those @rev things to svn urls. On Thu, May 8, 2008 2:27 pm, Paul Winkler wrote: > Today I was wishing that I could force fassembler to use a specific > revision number for things it checks out from svn.openplans.org. > > This is because when buildbot builds fail, it's often tricky to > determine exactly what revision was built, since it takes several minutes > between when the build starts and when various things get checked out. > Often, more commits are made during that time, and these > don't make it to the blamelist, and I have to guess what was actually built > and tested. And by the time I hear about the failure, the environment has > likely been wiped out for the next build. > > So I hacked up this branch: > https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-2 > 0080508 > > > ... which adds an option to fassembler:topp like so: > > > ./bin/fassembler fassembler:topp openplans_svn_revision=15000 ... > > > If specified, this is used for the requirements checkout (if it's on > svn.openplans.org), and specifies the same revision for any editable > checkouts in *-req.txt, regardless of any revision information in those > (again, only for things on svn.openplans.org). > > > This works AFAICT, within some IMO reasonable limitations: > > > * other fassembler: projects don't recognize the option on the command > line, so you'd have to edit build.ini if you change your mind later - or > re-run fassembler:topp. > > * re-running fassembler:opencore with a changed revision in build.ini > can fail if things have moved around in the repo; first updating > requirements/ to that revision should help > > * things could fail at runtime if there's some incompatibility between > the revision and an updated third-party component; hopefully this should be > rare, unless maybe you're building really old revisions for some reason. > > Does anybody object to anything here? If not I'll merge it to the > fassembler trunk, and then update buildbot to take advantage of it. > > > -- > > > Paul Winkler > http://www.openplans.org/people/slinkp/profile > yahoo: slinkp23 > AIM: slinkp1970 > > > > -- > Archive: > http://www.openplans.org/projects/fassembler/lists/fassembler-discussion/ > archive/2008/05/1210271247947 To unsubscribe send an email with subject > unsubscribe to fassembler-discussion@.... Please contact > fassembler-discussion-manager@... for questions. > > > !DSPAM:4003,4823464e250165210051143! > > >
-
On Fri, May 09, 2008 at 12:45:19AM -0400, ianb@... wrote: > In a requirement file you can add @rev-number to any svn URL. Would that > satisfy your need? I guess I could have the buildbot parse the checked out req files and modify them. That seems like more work and likely more brittle than what I've got. > Maybe poach-eggs --freeze would be useful to save information about the > build after it's been built? It adds in those @rev things to svn urls. Ooh, actually yes that is useful. It doesn't help with the wrong-blamelist problem, but maybe it's good enough. I'll give it a shot. - PW -- Paul Winkler http://www.openplans.org/people/slinkp/profile yahoo: slinkp23 AIM: slinkp1970
-
> On Fri, May 09, 2008 at 12:45:19AM -0400, ianb@... wrote: > > Maybe poach-eggs --freeze would be useful to save information about the > > build after it's been built? It adds in those @rev things to svn urls. Yep, this should be useful: http://pw.openplans.org:6020/builders/opencore-minimal/builds/647/steps/shell_7/logs/stdio -- Paul Winkler http://www.openplans.org/people/slinkp/profile yahoo: slinkp23 AIM: slinkp1970
-
-
Okay, so independently of getting the info from poach-eggs --freeze, would you object to me merging my fassembler branch? or do you think it's the just the wrong thing to do? https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-20080508 - PW On Fri, May 09, 2008 at 12:45:19AM -0400, ianb@... wrote: > In a requirement file you can add @rev-number to any svn URL. Would that > satisfy your need? > > Maybe poach-eggs --freeze would be useful to save information about the > build after it's been built? It adds in those @rev things to svn urls. > > > > On Thu, May 8, 2008 2:27 pm, Paul Winkler wrote: > > Today I was wishing that I could force fassembler to use a specific > > revision number for things it checks out from svn.openplans.org. > > > > This is because when buildbot builds fail, it's often tricky to > > determine exactly what revision was built, since it takes several minutes > > between when the build starts and when various things get checked out. > > Often, more commits are made during that time, and these > > don't make it to the blamelist, and I have to guess what was actually built > > and tested. And by the time I hear about the failure, the environment has > > likely been wiped out for the next build. > > > > So I hacked up this branch: > > https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-2 > > 0080508 > > > > > > ... which adds an option to fassembler:topp like so: > > > > > > ./bin/fassembler fassembler:topp openplans_svn_revision=15000 ... > > > > > > If specified, this is used for the requirements checkout (if it's on > > svn.openplans.org), and specifies the same revision for any editable > > checkouts in *-req.txt, regardless of any revision information in those > > (again, only for things on svn.openplans.org). > > > > > > This works AFAICT, within some IMO reasonable limitations: > > > > > > * other fassembler: projects don't recognize the option on the command > > line, so you'd have to edit build.ini if you change your mind later - or > > re-run fassembler:topp. > > > > * re-running fassembler:opencore with a changed revision in build.ini > > can fail if things have moved around in the repo; first updating > > requirements/ to that revision should help > > > > * things could fail at runtime if there's some incompatibility between > > the revision and an updated third-party component; hopefully this should be > > rare, unless maybe you're building really old revisions for some reason. > > > > Does anybody object to anything here? If not I'll merge it to the > > fassembler trunk, and then update buildbot to take advantage of it. > > > > > > -- > > > > > > Paul Winkler > > http://www.openplans.org/people/slinkp/profile > > yahoo: slinkp23 > > AIM: slinkp1970 > > > > > > > > -- > > Archive: > > http://www.openplans.org/projects/fassembler/lists/fassembler-discussion/ > > archive/2008/05/1210271247947 To unsubscribe send an email with subject > > unsubscribe to fassembler-discussion@.... Please contact > > fassembler-discussion-manager@... for questions. > > > > > > > > > > > > > > > > -- > Archive: http://www.openplans.org/projects/fassembler/lists/fassembler-discussion/archive/2008/05/1210308325604 > To unsubscribe send an email with subject unsubscribe to fassembler-discussion@.... Please contact fassembler-discussion-manager@... for questions. > > > !DSPAM:4043,4823d773177981849620573! > -- Paul Winkler http://www.openplans.org/people/slinkp/profile yahoo: slinkp23 AIM: slinkp1970
-
Paul Winkler wrote: > Okay, so independently of getting the info from poach-eggs --freeze, > would you object to me merging my fassembler branch? > or do you think it's the just the wrong thing to do? > https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-20080508 i don't know what ian's thoughts are, but it feels a bit off to me. maybe that's b/c i think of fassembler as more of a general tool, sort of like make or buildit. of course, the fassembler source code is littered w/ TOPPisms... all of our custom projects are defined in there. my druthers would be for the fassembler core to be its own project, with all of the topp:* projects defined in a separate package. this is also part of my motivation for putting the code that manages our particular deployment specifics (directory structure, etc.) into a separate package, rather than rolling it into a part of fassembler's core. of course, we may not have making fassembler work as a more general tool on our list of cares; it's certainly not a high organizational priority. even then, though, i'd still lean against adding arbitrary variable options to the countless we already have, if you've got a solid workaround that makes it unnecessary. 2-cents-which-won't-get-you-much-these-days'ly yours, -r > > - PW > > On Fri, May 09, 2008 at 12:45:19AM -0400, ianb@... wrote: >> In a requirement file you can add @rev-number to any svn URL. Would that >> satisfy your need? >> >> Maybe poach-eggs --freeze would be useful to save information about the >> build after it's been built? It adds in those @rev things to svn urls. >> >> >> >> On Thu, May 8, 2008 2:27 pm, Paul Winkler wrote: >>> Today I was wishing that I could force fassembler to use a specific >>> revision number for things it checks out from svn.openplans.org. >>> >>> This is because when buildbot builds fail, it's often tricky to >>> determine exactly what revision was built, since it takes several minutes >>> between when the build starts and when various things get checked out. >>> Often, more commits are made during that time, and these >>> don't make it to the blamelist, and I have to guess what was actually built >>> and tested. And by the time I hear about the failure, the environment has >>> likely been wiped out for the next build. >>> >>> So I hacked up this branch: >>> https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-2 >>> 0080508 >>> >>> >>> ... which adds an option to fassembler:topp like so: >>> >>> >>> ./bin/fassembler fassembler:topp openplans_svn_revision=15000 ... >>> >>> >>> If specified, this is used for the requirements checkout (if it's on >>> svn.openplans.org), and specifies the same revision for any editable >>> checkouts in *-req.txt, regardless of any revision information in those >>> (again, only for things on svn.openplans.org). >>> >>> >>> This works AFAICT, within some IMO reasonable limitations: >>> >>> >>> * other fassembler: projects don't recognize the option on the command >>> line, so you'd have to edit build.ini if you change your mind later - or >>> re-run fassembler:topp. >>> >>> * re-running fassembler:opencore with a changed revision in build.ini >>> can fail if things have moved around in the repo; first updating >>> requirements/ to that revision should help >>> >>> * things could fail at runtime if there's some incompatibility between >>> the revision and an updated third-party component; hopefully this should be >>> rare, unless maybe you're building really old revisions for some reason. >>> >>> Does anybody object to anything here? If not I'll merge it to the >>> fassembler trunk, and then update buildbot to take advantage of it.
-
One very minor objection to the branch which would need to get fixed before it's merged -- https://svn.openplans.org doesn't actually point to a single repository; there's at least ./svn and ./melk (and I believe ./config is also separate?) independently tracking revisions, so where in the code you say ``if full_repo.count('svn.openplans.org')`` you probably mean 'svn.openplans.org/svn'. On Fri, May 9, 2008 at 7:40 PM, Rob Miller <robm@...> wrote: > i don't know what ian's thoughts are, but it feels a bit off to me. maybe > that's b/c i think of fassembler as more of a general tool, sort of like > make or buildit. Perhaps it could be generalized to a new [repository-revisions] section in build.ini with configuration options of the form "SVN_REPO_BASE_URL = REVISION_TO_PEG"? But I'm not sure how that would work for the convenient command-line option. > of course, we may not have making fassembler work as a more general tool on > our list of cares; it's certainly not a high organizational priority. even > then, though, i'd still lean against adding arbitrary variable options to > the countless we already have, if you've got a solid workaround that makes > it unnecessary. I don't think there is a solid workaround, though? As Paul said, using --freeze-eggs to generate requirements files pegged to particular code revisions is sort of the other end of the problem -- this switch is for those times when you *don't* have a "revision tag" pre-generated. Anyway, personally I'm very pro-merging; this is something I've been wanting very badly for a long time, actually. egj-
On Sun, May 11, 2008 at 07:00:55PM -0400, Ethan Jucovy wrote: > One very minor objection to the branch which would need to get fixed > before it's merged -- https://svn.openplans.org doesn't actually point > to a single repository; there's at least ./svn and ./melk (and I > believe ./config is also separate?) independently tracking revisions, > so where in the code you say ``if > full_repo.count('svn.openplans.org')`` you probably mean > 'svn.openplans.org/svn'. Whoops, you're correct, fixed in two places. > On Fri, May 9, 2008 at 7:40 PM, Rob Miller <robm@...> wrote: > > i don't know what ian's thoughts are, but it feels a bit off to me. maybe > > that's b/c i think of fassembler as more of a general tool, sort of like > > make or buildit. In theory I agree with this, but in practice it will take a lot of detangling before that's true. And pragmatically, I don't mind building a little bike shed that violates a principle when the entire neighborhood already violates that principle. Seems like the wrong time to get strict. > Perhaps it could be generalized to a new [repository-revisions] > section in build.ini with configuration options of the form > "SVN_REPO_BASE_URL = REVISION_TO_PEG"? But I'm not sure how that > would work for the convenient command-line option. I'm not sure I want to save this in build.ini at all. I think if you want to make it persistent, the place to do that would be in requirements files. > > of course, we may not have making fassembler work as a more general tool on > > our list of cares; it's certainly not a high organizational priority. even > > then, though, i'd still lean against adding arbitrary variable options to > > the countless we already have, if you've got a solid workaround that makes > > it unnecessary. > > I don't think there is a solid workaround, though? As Paul said, > using --freeze-eggs to generate requirements files pegged to > particular code revisions is sort of the other end of the problem -- > this switch is for those times when you *don't* have a "revision tag" > pre-generated. Yeah, I think there's several different cases: - finding out what got built, a posteriori (poach-eggs trunk is good for this) - persisting a frozen set of requirements for later reuse (poach-eggs is good for this) - building an ad-hoc frozen snapshot for which no frozen requirements exist in svn. This is the use case the patch was intended to serve. Aside from helping with the buildbot blamelist, this might be useful for tasks like figuring out which changeset introduced a bug. -- Paul Winkler http://www.openplans.org/people/slinkp/profile yahoo: slinkp23 AIM: slinkp1970
-
-
-
Paul Winkler wrote: > Okay, so independently of getting the info from poach-eggs --freeze, > would you object to me merging my fassembler branch? > or do you think it's the just the wrong thing to do? > https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-20080508 I'd rather this take the form of something like requirements/revision-pin.txt, which would contain: https://svn.openplans.org/svn/ {YYYYMMDD} (Any revision specifier that svn took would work, like an integer or a date) I think this would be both more general and more useful. Or, maybe instead of an ad hoc format in a file like this, something like requirements/control.ini: [repository https://svn.openplans.org/svn/] pin_revision = {YYYYMMDD} Then we could add more options later. E.g., if we make a local mirror, we could do: [repository http://svn.zope.org] mirror = https://svn.openplans.org/mirrors/svn.zope.org Sometimes a more general option is desired, like: [repository *] pin_revision = {YYYYMMDD} which would be a nice catch-all (jhbuild has an option like this, for example). Ian-
I discovered something recently that's relevant to this old discussion. Maybe you guys knew this, but I was surprised. It turns out that, contrary to my assumptions, you CAN'T easy_install something from an svn repo and peg it to a revision *except* by install it as --editable. This is because easy_install only does an svn checkout when given --editable, and svn urls of the form URL@REVISION are actually implemented by the svn *client*, not the server. (Yes, really. That was a nasty surprise.) The only workarounds I know of so far are: * install as editable (if the package provides setup.py develop) * put a pegged svn external in the bundle * make a tag (if it's ours) or vendor import (if it's not) in our svn repo, and install from that URL instead * or, build a source egg, throw it on svn.openplans.org/eggs, and explicitly install that. So the whole premise for this thread was partly spurious.
-
Paul Winkler wrote: > I discovered something recently that's relevant to this old discussion. > Maybe you guys knew this, but I was surprised. > > It turns out that, contrary to my assumptions, you CAN'T easy_install > something from an svn repo and peg it to a revision *except* by install it as > --editable. > This is because easy_install only does an svn checkout when given --editable, > and svn urls of the form URL@REVISION are actually implemented by the svn > *client*, not the server. (Yes, really. That was a nasty surprise.) > > The only workarounds I know of so far are: > > * install as editable (if the package provides setup.py develop) > * put a pegged svn external in the bundle > * make a tag (if it's ours) or vendor import (if it's not) in our svn repo, > and install from that URL instead > * or, build a source egg, throw it on svn.openplans.org/eggs, and explicitly > install that. > > So the whole premise for this thread was partly spurious. Well, yes, but we're not installing via easy_install directly, we're installing via fassembler. And fassembler can do whatever we want. easy_install can't really do --editable anyway, at least not in a useful way -- fassembler actually checks out each of those and runs setup.py develop to install. (It could check them out to a temporary directory and run setup.py install, but that never seemed particularly advantageous over putting the source in src/). Ian
-
-
-
-