• fassembler discussion

  • Command line option for revision?

    from slinkp on May 08, 2008 02:27 PM
    Today I was wishing that I could force fassembler to use a specific
    revision number for things it checks out from svn.openplans.org.
    
    This is because when buildbot builds fail, it's often tricky to
    determine exactly what revision was built, since it takes several
    minutes between when the build starts and when various things get
    checked out. Often, more commits are made during that time, and these
    don't make it to the blamelist, and I have to guess what was actually
    built and tested. And by the time I hear about the failure, the
    environment has likely been wiped out for the next build.
    
    So I hacked up this branch:
    https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-20080508
    
    ... which adds an option to fassembler:topp like so:
    
    ./bin/fassembler fassembler:topp openplans_svn_revision=15000 ...
    
    If specified, this is used for the requirements checkout (if it's on
    svn.openplans.org), and specifies the same revision for any editable
    checkouts in *-req.txt, regardless of any revision information in
    those (again, only for things on svn.openplans.org).
    
    This works AFAICT, within some IMO reasonable limitations:
    
    * other fassembler: projects don't recognize the option on the command
    line, so you'd have to edit build.ini if you change your mind later -
    or re-run fassembler:topp.
    
    * re-running fassembler:opencore with a changed revision in build.ini
    can fail if things have moved around in the repo; first updating
    requirements/ to that revision should help
    
    * things could fail at runtime if there's some incompatibility between
    the revision and an updated third-party component; hopefully this
    should be rare, unless maybe you're building really old revisions for
    some reason.
    
    Does anybody object to anything here? If not I'll merge it to the
    fassembler trunk, and then update buildbot to take advantage of it.
    
    
    -- 
    
    Paul Winkler
    http://www.openplans.org/people/slinkp/profile
    yahoo: slinkp23
    AIM:   slinkp1970
    
    Thread Outline:
  • Re: Command line option for revision?

    from ianb on May 09, 2008 12:45 AM
    In a requirement file you can add @rev-number to any svn URL.  Would that
    satisfy your need?
    
    Maybe poach-eggs --freeze would be useful to save information about the
    build after it's been built?  It adds in those @rev things to svn urls.
    
    
    
    On Thu, May 8, 2008 2:27 pm, Paul Winkler wrote:
    > Today I was wishing that I could force fassembler to use a specific
    > revision number for things it checks out from svn.openplans.org.
    >
    > This is because when buildbot builds fail, it's often tricky to
    > determine exactly what revision was built, since it takes several minutes
    > between when the build starts and when various things get checked out.
    > Often, more commits are made during that time, and these
    > don't make it to the blamelist, and I have to guess what was actually built
    > and tested. And by the time I hear about the failure, the environment has
    > likely been wiped out for the next build.
    >
    > So I hacked up this branch:
    > https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-2
    > 0080508
    >
    >
    > ... which adds an option to fassembler:topp like so:
    >
    >
    > ./bin/fassembler fassembler:topp openplans_svn_revision=15000 ...
    >
    >
    > If specified, this is used for the requirements checkout (if it's on
    > svn.openplans.org), and specifies the same revision for any editable
    > checkouts in *-req.txt, regardless of any revision information in those
    > (again, only for things on svn.openplans.org).
    >
    >
    > This works AFAICT, within some IMO reasonable limitations:
    >
    >
    > * other fassembler: projects don't recognize the option on the command
    > line, so you'd have to edit build.ini if you change your mind later - or
    > re-run fassembler:topp.
    >
    > * re-running fassembler:opencore with a changed revision in build.ini
    > can fail if things have moved around in the repo; first updating
    > requirements/ to that revision should help
    >
    > * things could fail at runtime if there's some incompatibility between
    > the revision and an updated third-party component; hopefully this should be
    > rare, unless maybe you're building really old revisions for some reason.
    >
    > Does anybody object to anything here? If not I'll merge it to the
    > fassembler trunk, and then update buildbot to take advantage of it.
    >
    >
    > --
    >
    >
    > Paul Winkler
    > http://www.openplans.org/people/slinkp/profile
    > yahoo: slinkp23
    > AIM:   slinkp1970
    >
    >
    >
    > --
    > Archive:
    > http://www.openplans.org/projects/fassembler/lists/fassembler-discussion/
    > archive/2008/05/1210271247947 To unsubscribe send an email with subject
    > unsubscribe to fassembler-discussion@....  Please contact
    > fassembler-discussion-manager@... for questions.
    >
    >
    > !DSPAM:4003,4823464e250165210051143!
    >
    >
    >
    
    
    • Re: Command line option for revision?

      from slinkp on May 09, 2008 11:38 AM
      On Fri, May 09, 2008 at 12:45:19AM -0400, ianb@... wrote:
      > In a requirement file you can add @rev-number to any svn URL.  Would that
      > satisfy your need?
      
      I guess I could have the buildbot parse the checked out req files and
      modify them. That seems like more work and likely more brittle than
      what I've got.
      
      > Maybe poach-eggs --freeze would be useful to save information about the
      > build after it's been built?  It adds in those @rev things to svn urls.
      
      Ooh, actually yes that is useful.  It doesn't help with the
      wrong-blamelist problem, but maybe it's good enough. I'll give it a
      shot.
      
      
      - PW
      
      
      -- 
      
      Paul Winkler
      http://www.openplans.org/people/slinkp/profile
      yahoo: slinkp23
      AIM:   slinkp1970
      
      • Re: Command line option for revision?

        from slinkp on May 09, 2008 05:03 PM
        > On Fri, May 09, 2008 at 12:45:19AM -0400, ianb@... wrote:
        > > Maybe poach-eggs --freeze would be useful to save information about the
        > > build after it's been built?  It adds in those @rev things to svn urls.
        
        Yep, this should be useful:
        
        http://pw.openplans.org:6020/builders/opencore-minimal/builds/647/steps/shell_7/logs/stdio
        
        
        -- 
        
        Paul Winkler
        http://www.openplans.org/people/slinkp/profile
        yahoo: slinkp23
        AIM:   slinkp1970
        
    • Re: Command line option for revision?

      from slinkp on May 09, 2008 07:13 PM
      Okay, so independently of getting the info from poach-eggs --freeze,
      would you object to me merging my fassembler branch?
      or do you think it's the just the wrong thing to do?
      https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-20080508
      
      - PW
      
      On Fri, May 09, 2008 at 12:45:19AM -0400, ianb@... wrote:
      > In a requirement file you can add @rev-number to any svn URL.  Would that
      > satisfy your need?
      > 
      > Maybe poach-eggs --freeze would be useful to save information about the
      > build after it's been built?  It adds in those @rev things to svn urls.
      > 
      > 
      > 
      > On Thu, May 8, 2008 2:27 pm, Paul Winkler wrote:
      > > Today I was wishing that I could force fassembler to use a specific
      > > revision number for things it checks out from svn.openplans.org.
      > >
      > > This is because when buildbot builds fail, it's often tricky to
      > > determine exactly what revision was built, since it takes several minutes
      > > between when the build starts and when various things get checked out.
      > > Often, more commits are made during that time, and these
      > > don't make it to the blamelist, and I have to guess what was actually built
      > > and tested. And by the time I hear about the failure, the environment has
      > > likely been wiped out for the next build.
      > >
      > > So I hacked up this branch:
      > > https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-2
      > > 0080508
      > >
      > >
      > > ... which adds an option to fassembler:topp like so:
      > >
      > >
      > > ./bin/fassembler fassembler:topp openplans_svn_revision=15000 ...
      > >
      > >
      > > If specified, this is used for the requirements checkout (if it's on
      > > svn.openplans.org), and specifies the same revision for any editable
      > > checkouts in *-req.txt, regardless of any revision information in those
      > > (again, only for things on svn.openplans.org).
      > >
      > >
      > > This works AFAICT, within some IMO reasonable limitations:
      > >
      > >
      > > * other fassembler: projects don't recognize the option on the command
      > > line, so you'd have to edit build.ini if you change your mind later - or
      > > re-run fassembler:topp.
      > >
      > > * re-running fassembler:opencore with a changed revision in build.ini
      > > can fail if things have moved around in the repo; first updating
      > > requirements/ to that revision should help
      > >
      > > * things could fail at runtime if there's some incompatibility between
      > > the revision and an updated third-party component; hopefully this should be
      > > rare, unless maybe you're building really old revisions for some reason.
      > >
      > > Does anybody object to anything here? If not I'll merge it to the
      > > fassembler trunk, and then update buildbot to take advantage of it.
      > >
      > >
      > > --
      > >
      > >
      > > Paul Winkler
      > > http://www.openplans.org/people/slinkp/profile
      > > yahoo: slinkp23
      > > AIM:   slinkp1970
      > >
      > >
      > >
      > > --
      > > Archive:
      > > http://www.openplans.org/projects/fassembler/lists/fassembler-discussion/
      > > archive/2008/05/1210271247947 To unsubscribe send an email with subject
      > > unsubscribe to fassembler-discussion@....  Please contact
      > > fassembler-discussion-manager@... for questions.
      > >
      > >
      > > 
      > >
      > >
      > >
      > 
      > 
      > 
      > --
      > Archive: http://www.openplans.org/projects/fassembler/lists/fassembler-discussion/archive/2008/05/1210308325604
      > To unsubscribe send an email with subject unsubscribe to fassembler-discussion@....  Please contact fassembler-discussion-manager@... for questions.
      > 
      > 
      > !DSPAM:4043,4823d773177981849620573!
      > 
      
      -- 
      
      Paul Winkler
      http://www.openplans.org/people/slinkp/profile
      yahoo: slinkp23
      AIM:   slinkp1970
      
      • Re: Command line option for revision?

        from ra on May 09, 2008 07:38 PM
        Paul Winkler wrote:
        > Okay, so independently of getting the info from poach-eggs --freeze,
        > would you object to me merging my fassembler branch?
        > or do you think it's the just the wrong thing to do?
        > https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-20080508
        
        i don't know what ian's thoughts are, but it feels a bit off to me.  maybe 
        that's b/c i think of fassembler as more of a general tool, sort of like make 
        or buildit.
        
        of course, the fassembler source code is littered w/ TOPPisms... all of our 
        custom projects are defined in there.  my druthers would be for the fassembler 
        core to be its own project, with all of the topp:* projects defined in a 
        separate package.
        
        this is also part of my motivation for putting the code that manages our 
        particular deployment specifics (directory structure, etc.) into a separate 
        package, rather than rolling it into a part of fassembler's core.
        
        of course, we may not have making fassembler work as a more general tool on 
        our list of cares; it's certainly not a high organizational priority.  even 
        then, though, i'd still lean against adding arbitrary variable options to the 
        countless we already have, if you've got a solid workaround that makes it 
        unnecessary.
        
        2-cents-which-won't-get-you-much-these-days'ly yours,
        
        -r
        
        
        > 
        > - PW
        > 
        > On Fri, May 09, 2008 at 12:45:19AM -0400, ianb@... wrote:
        >> In a requirement file you can add @rev-number to any svn URL.  Would that
        >> satisfy your need?
        >>
        >> Maybe poach-eggs --freeze would be useful to save information about the
        >> build after it's been built?  It adds in those @rev things to svn urls.
        >>
        >>
        >>
        >> On Thu, May 8, 2008 2:27 pm, Paul Winkler wrote:
        >>> Today I was wishing that I could force fassembler to use a specific
        >>> revision number for things it checks out from svn.openplans.org.
        >>>
        >>> This is because when buildbot builds fail, it's often tricky to
        >>> determine exactly what revision was built, since it takes several minutes
        >>> between when the build starts and when various things get checked out.
        >>> Often, more commits are made during that time, and these
        >>> don't make it to the blamelist, and I have to guess what was actually built
        >>> and tested. And by the time I hear about the failure, the environment has
        >>> likely been wiped out for the next build.
        >>>
        >>> So I hacked up this branch:
        >>> https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-2
        >>> 0080508
        >>>
        >>>
        >>> ... which adds an option to fassembler:topp like so:
        >>>
        >>>
        >>> ./bin/fassembler fassembler:topp openplans_svn_revision=15000 ...
        >>>
        >>>
        >>> If specified, this is used for the requirements checkout (if it's on
        >>> svn.openplans.org), and specifies the same revision for any editable
        >>> checkouts in *-req.txt, regardless of any revision information in those
        >>> (again, only for things on svn.openplans.org).
        >>>
        >>>
        >>> This works AFAICT, within some IMO reasonable limitations:
        >>>
        >>>
        >>> * other fassembler: projects don't recognize the option on the command
        >>> line, so you'd have to edit build.ini if you change your mind later - or
        >>> re-run fassembler:topp.
        >>>
        >>> * re-running fassembler:opencore with a changed revision in build.ini
        >>> can fail if things have moved around in the repo; first updating
        >>> requirements/ to that revision should help
        >>>
        >>> * things could fail at runtime if there's some incompatibility between
        >>> the revision and an updated third-party component; hopefully this should be
        >>> rare, unless maybe you're building really old revisions for some reason.
        >>>
        >>> Does anybody object to anything here? If not I'll merge it to the
        >>> fassembler trunk, and then update buildbot to take advantage of it.
        
        
        • Re: Command line option for revision?

          from darcy on May 11, 2008 07:01 PM
          One very minor objection to the branch which would need to get fixed
          before it's merged -- https://svn.openplans.org doesn't actually point
          to a single repository; there's at least ./svn and ./melk (and I
          believe ./config is also separate?) independently tracking revisions,
          so where in the code you say ``if
          full_repo.count('svn.openplans.org')`` you probably mean
          'svn.openplans.org/svn'.
          
          On Fri, May 9, 2008 at 7:40 PM, Rob Miller <robm@...> wrote:
          >  i don't know what ian's thoughts are, but it feels a bit off to me.  maybe
          > that's b/c i think of fassembler as more of a general tool, sort of like
          > make or buildit.
          
          Perhaps it could be generalized to a new [repository-revisions]
          section in build.ini with configuration options of the form
          "SVN_REPO_BASE_URL = REVISION_TO_PEG"?  But I'm not sure how that
          would work for the convenient command-line option.
          
          > of course, we may not have making fassembler work as a more general tool on
          > our list of cares; it's certainly not a high organizational priority.  even
          > then, though, i'd still lean against adding arbitrary variable options to
          > the countless we already have, if you've got a solid workaround that makes
          > it unnecessary.
          
          I don't think there is a solid workaround, though?  As Paul said,
          using --freeze-eggs to generate requirements files pegged to
          particular code revisions is sort of the other end of the problem --
          this switch is for those times when you *don't* have a "revision tag"
          pre-generated.
          
          Anyway, personally I'm very pro-merging; this is something I've been
          wanting very badly for a long time, actually.
          
          egj
          
          • Re: Command line option for revision?

            from slinkp on May 12, 2008 01:35 PM
            On Sun, May 11, 2008 at 07:00:55PM -0400, Ethan Jucovy wrote:
            > One very minor objection to the branch which would need to get fixed
            > before it's merged -- https://svn.openplans.org doesn't actually point
            > to a single repository; there's at least ./svn and ./melk (and I
            > believe ./config is also separate?) independently tracking revisions,
            > so where in the code you say ``if
            > full_repo.count('svn.openplans.org')`` you probably mean
            > 'svn.openplans.org/svn'.
            
            Whoops, you're correct, fixed in two places.
            
            > On Fri, May 9, 2008 at 7:40 PM, Rob Miller <robm@...> wrote:
            > >  i don't know what ian's thoughts are, but it feels a bit off to me.  maybe
            > > that's b/c i think of fassembler as more of a general tool, sort of like
            > > make or buildit.
            
            In theory I agree with this, but in practice it will take a lot of
            detangling before that's true.
            
            And pragmatically, I don't mind building a little bike shed that
            violates a principle when the entire neighborhood already violates
            that principle. Seems like the wrong time to get strict.
             
            > Perhaps it could be generalized to a new [repository-revisions]
            > section in build.ini with configuration options of the form
            > "SVN_REPO_BASE_URL = REVISION_TO_PEG"?  But I'm not sure how that
            > would work for the convenient command-line option.
            
            I'm not sure I want to save this in build.ini at all. I think if you
            want to make it persistent, the place to do that would be in
            requirements files.
            
            > > of course, we may not have making fassembler work as a more general tool on
            > > our list of cares; it's certainly not a high organizational priority.  even
            > > then, though, i'd still lean against adding arbitrary variable options to
            > > the countless we already have, if you've got a solid workaround that makes
            > > it unnecessary.
            > 
            > I don't think there is a solid workaround, though?  As Paul said,
            > using --freeze-eggs to generate requirements files pegged to
            > particular code revisions is sort of the other end of the problem --
            > this switch is for those times when you *don't* have a "revision tag"
            > pre-generated.
            
            Yeah, I think there's several different cases:
            
            - finding out what got built, a posteriori (poach-eggs trunk is good
              for this)
             
            - persisting a frozen set of requirements for later reuse
              (poach-eggs is good for this)
            
            - building an ad-hoc frozen snapshot for which no frozen requirements
              exist in svn. This is the use case the patch was intended to serve.
              Aside from helping with the buildbot blamelist, this might be useful
              for tasks like figuring out which changeset introduced a bug.
            
            -- 
            
            Paul Winkler
            http://www.openplans.org/people/slinkp/profile
            yahoo: slinkp23
            AIM:   slinkp1970
            
      • Re: Command line option for revision?

        from ianb on May 12, 2008 11:18 PM
        Paul Winkler wrote:
        > Okay, so independently of getting the info from poach-eggs --freeze,
        > would you object to me merging my fassembler branch?
        > or do you think it's the just the wrong thing to do?
        > https://svn.openplans.org/svn/fassembler/branches/slinkp-revision-option-20080508
        
        I'd rather this take the form of something like 
        requirements/revision-pin.txt, which would contain:
        
           https://svn.openplans.org/svn/ {YYYYMMDD}
        
        (Any revision specifier that svn took would work, like an integer or a date)
        
        I think this would be both more general and more useful.  Or, maybe 
        instead of an ad hoc format in a file like this, something like 
        requirements/control.ini:
        
           [repository https://svn.openplans.org/svn/]
           pin_revision = {YYYYMMDD}
        
        Then we could add more options later.  E.g., if we make a local mirror, 
        we could do:
        
           [repository http://svn.zope.org]
           mirror = https://svn.openplans.org/mirrors/svn.zope.org
        
        Sometimes a more general option is desired, like:
        
           [repository *]
           pin_revision = {YYYYMMDD}
        
        which would be a nice catch-all (jhbuild has an option like this, for 
        example).
        
           Ian
        
        • Re: Command line option for revision?

          from slinkp on Jul 03, 2008 07:17 PM
          I discovered something recently that's relevant to this old discussion.
          Maybe you guys knew this, but I was surprised.
          
          It turns out that, contrary to my assumptions, you CAN'T easy_install
          something from an svn repo and peg it to a revision *except* by install it as
          --editable.
          This is because easy_install only does an svn checkout when given --editable,
          and svn urls of the form URL@REVISION are actually implemented by the svn
          *client*, not the server. (Yes, really. That was a nasty surprise.)
          
          The only workarounds I know of so far are:
          
          * install as editable (if the package provides setup.py develop)
          * put a pegged svn external in the bundle
          * make a tag (if it's ours) or vendor import (if it's not) in our svn repo,
          and install from that URL instead
            * or, build a source egg, throw it on svn.openplans.org/eggs, and explicitly
          install that.
          
          So the whole premise for this thread was partly spurious.
          
          • Re: Re: Command line option for revision?

            from ianb on Jul 03, 2008 08:28 PM
            Paul Winkler wrote:
            > I discovered something recently that's relevant to this old discussion.
            > Maybe you guys knew this, but I was surprised.
            > 
            > It turns out that, contrary to my assumptions, you CAN'T easy_install
            > something from an svn repo and peg it to a revision *except* by install it as
            > --editable.
            > This is because easy_install only does an svn checkout when given --editable,
            > and svn urls of the form URL@REVISION are actually implemented by the svn
            > *client*, not the server. (Yes, really. That was a nasty surprise.)
            > 
            > The only workarounds I know of so far are:
            > 
            > * install as editable (if the package provides setup.py develop)
            > * put a pegged svn external in the bundle
            > * make a tag (if it's ours) or vendor import (if it's not) in our svn repo,
            > and install from that URL instead
            >   * or, build a source egg, throw it on svn.openplans.org/eggs, and explicitly
            > install that.
            > 
            > So the whole premise for this thread was partly spurious.
            
            Well, yes, but we're not installing via easy_install directly, we're 
            installing via fassembler.  And fassembler can do whatever we want. 
            easy_install can't really do --editable anyway, at least not in a useful 
            way -- fassembler actually checks out each of those and runs setup.py 
            develop to install.  (It could check them out to a temporary directory 
            and run setup.py install, but that never seemed particularly 
            advantageous over putting the source in src/).
            
               Ian