On Mon, May 12, 2008 at 12:50:15PM -0400, Ethan Jucovy wrote:
> On Mon, May 12, 2008 at 12:44 PM, Jeff Hammel <jhammel@...> wrote:
> >  I would like to see things be kept as easy as possible for non-TOPP people to build our software.  Even if we're not interested in outside contributions or use at this point (which we're not, as I understand things), I think writing fassembler with this in mind is probably easier to do up front than to reengineer later.  I can see the other point of view here too (that is, why waste our time keeping fassembler working outside of our core developers at this point of time?), but I'm speaking purely from my perspective on how I like to develop software.
> 
> +1.  Just to be clear, let me disambiguate:
> 
>  * Building fassembler in a way that allows non-TOPPers to build our
> software: +1.
>  * Building fassembler in a way that allows fassembler to build any
> software: -1.
> 
> In other words, I think Fassembler's purpose should be to build The
> OpenPlans Software Stack only, but to build it for anyone, not just
> TOPP.

Yes, this is what I meant :)

jeff

Ethan Jucovy wrote:
>> I've thought about this some with relation to zc.buildout.  There are
>> several features that fassembler has that zc.buildout doesn't (and should),
>> and of course zc.buildout is a more generally used tool.
> 
> Which features do you have in mind?

Mostly filemaker, which serves as the basis for all the tasks, where 
zc.buildout recipes tend to use ad hoc code that isn't very safe.

I also think the configuration in fassembler allows for less verbose 
build plans.  The line between configuration and code in zc.buildout 
is... well, maybe not more vague, but it's placed differently.

I also think some parts of our installation story are more sensible, 
especially how eggs are handled.  But then, I think a lot of people 
(probably a majority?) would agree with me, so it would not be very 
controversial to make such a change (I was going to say "to suggest such 
a change", but it's been suggested several times, and it's really a 
question of implementation not suggestion).

   Ian

On May 23, 2008, at 1:44 PM, Ethan Jucovy wrote:
>> I've thought about this some with relation to zc.buildout.  There are
>> several features that fassembler has that zc.buildout doesn't (and  
>> should),
>> and of course zc.buildout is a more generally used tool.
>
> Which features do you have in mind?

having finally dug in a bit more w/ zc.buildout, i've got some  
thoughts on how they compare.

zc.buildout does a better job of encapsulating all of the information  
needed for a deployment in a single place... there's a single  
buildout.cfg file, and (in theory, anyway) this and buildout's  
bootstrap.py script is all you need to go from zero to fully  
deployed.  the buildout.cfg file is a single file that serves the same  
purpose as fassembler's projects.txt and build.ini files, in what i  
think is a more easily understandable format.  this may seem like a  
small thing, but i think it has a big psychological impact.  when you  
want to sit down and get started with a new buildout, it's clear where  
you need to start; with fassembler, this is a bit more fuzzy.

a buildout is made up of "parts", analogous to fassembler's projects.   
each part has a "recipe", which is a python egg that contains the code  
that performs the actual work.  the buildout.cfg file defines the  
parts for a buildout and contains any configuration information  
required by the various recipes.

buildout does not provide any tools that actually make the process of  
building a recipe any easier, however.  fassembler provides the  
"filemaker" (a library that provides all sorts of goodies for pushing  
files around, handling diffs and sanity checks and such) and a nice  
task- and  project-based object model.

another difference is that zc.buildout does some weird stuff w/ the  
sys.path to isolate your buildout's eggs... fassembler doesn't try to  
solve this problem, instead using virtualenv to handle this.  i think  
this is a better approach, because it's more like what python folks  
are already doing to manage this problem, and because it allows for  
more interactive prototyping.  with virtualenv you can activate an  
environment and start installing stuff, eventually using "poach-eggs -- 
freeze" to capture the state of the environment for later  
reproduction.  w/ buildout, you have to edit the buildout.cfg to  
specify what eggs you want to use, then re-run the buildout binary to  
set up the environment.  again, not a huge thing, but it feels more  
awkward to me.  also, since buildout uses some strange magic to set up  
the python environment, it's not clear to me how to introspect the  
python environment to debug the setup.

luckily, buildout and virtualenv are not mutually exclusive.  it  
wouldn't be hard to create a buildout that simply sets up a bunch of  
virtualenv's and calls poach-eggs for each one.

>> That would be nice for zc.buildout, but I don't know if it would  
>> help us.
>> zc.buildout is not quite the same scope as fassembler, and in  
>> practice
>> fassembler *does* have bindings to some of our conventions --  
>> conventions
>> that are not necessarily product-specific, but
>> how-we-glue-our-products-together-specific.  And many of fassemblers
>> features really are at the recipe level.  So, we could change our  
>> work to be
>> zc.buildout recipes, maybe keep a lot of the features (so long as  
>> we don't
>> use anyone else's recipes), and maybe even get as polished an  
>> experience.
>> But we couldn't use other people's recipes, and I'm not sure other  
>> people
>> could easily use our recipes (though maybe so).

i'm curious why you think our recipes wouldn't interoperate w/ other  
recipes.  they may work differently, but are they really mutually  
exclusive?  couldn't they coexist peacefully?

in any event, if we built a compelling tool set for recipe building  
(which i think we already have), i suspect others would start to use  
it.  maybe filemaker just needs to be its own package...

>> If zc.buildout had cool features that *we* want, then maybe this  
>> would be
>> useful.  I don't know... does it have features we want?
>
> It depends on who "we" are.  The OpenPlans software stack doesn't use
> buildout, and -- for the same pragmatic reasons that I'm against
> opensourcing fassembler -- I don't think there's any reason to start
> using buildout.  But TOPP does use buildout; Whit's used it for a few
> projects, and Ra just wrote a buildout for vanilla-Listen.  And plenty
> of people outside of TOPP have written buildout recipes centered
> around Deliverance.  So we can contribute useful features from
> fassembler into buildout and still benefit directly.  And of course
> even if we didn't use buildout at all, and if no one else was using
> buildout with our products, we would presumably benefit indirectly --
> a better build tool for the Zope and Plone communities presumably
> translates into more productive and successful projects in those
> communities, some percentage of which it's not crazy to assume TOPP
> will one day want to reuse.
>
>> If we want people to build our website, or some subset of our  
>> website, then
>> sure.  If we want people to mix new pieces into a website, when the  
>> website
>> is structured similar to how ours is, then we need to build other  
>> kinds of
>> software with fassembler.
>>
>> Also, fassembler is one piece of what makes a stack like ours  
>> feasible.  If
>> we want people to use stacks like ours, there needs to be a good  
>> build
>> story.  fassembler doesn't *have* to be that story, but it's the  
>> only story
>> we have.  If we want to make something like dvhoster open source,  
>> we need
>> this story -- without a diverse and interesting set of backend  
>> processes
>> dvhoster just isn't interesting or manageable.
>>
>> From this perspective, maybe adopting and improving a tool like  
>> zc.buildout
>> makes more sense.  Of course, internally it doesn't make sense,  
>> because it
>> has all the problems of making fassembler open source, except times  
>> 100, as
>> we squeeze our specific needs into this general tool.
>>
>> For the person thinking about how to make a website and how to  
>> build their
>> tools, making fassembler open source does just add to the mix and  
>> make the
>> decision harder and perhaps more partisan.  So from a strategic  
>> point of
>> view it's still not an easy or clear decision.
>>
>> Also, I think the work we've done building up conventions on  
>> building and
>> deploying a site is important, and itself worth sharing.  How that  
>> fits into
>> fassembler (or how it should fit into fassembler), honestly I don't  
>> know.
>
> This, I think, also feels premature to me.  I think it's clear that
> "fassembler is one piece of what makes a stack like ours feasible" but
> I also think that our stack isn't really there yet; fassembler has
> taken it a big step forward, and I think will continue to do so, but I
> at least don't yet feel confident that our stack is feasible, or that
> our conventions on building and deploying a site are yet bulletproof.
> So while I'd love for us to share these things, I think we've got a
> lot of work left to figure out this story in practice before we can
> actually tell it to anyone else.

i disagree.  we are actively dealing with, and making progress  
against, deployment challenges that are nearly universal to people  
building web tools.  it's not perfect yet.  it may never be perfect.   
but we've been through a number of iterations, it's gotten better each  
time, and i think we already have valuable information, based on real- 
world pain, encapsulated in our tools and our process.  i don't think  
we need to wait until we reach some ultimate state of perfection  
before what we've done is valuable to others.

-r