Ethan Jucovy wrote:
> On Fri, May 2, 2008 at 12:09 PM, Robert Marianski
> <rmarianski@...> wrote:
>> So then the zeo stuff would rougly look like:
>> 1. if something is running on the zeo port, don't try to start zeo
>>   (remember if you started it or not)
> 
> If we're going to do this (presumably fassembler is in that case
> assuming that, since it already asked you about unavailable ports and
> you didn't abort the build, you know what's on zeo's port and are cool
> with it) I'd prefer that we move, or repeat, the "unavailable ports"
> question in each component's build, and maybe make it a little more
> explicit.  An up-front "the following ports are unavailable;
> continue?" is somewhat easier to ignore unthinkingly than "port 10003,
> which I'm about to build zeo on, is already in use; continue?"
> 
> Other than this, I'm alright with the "implicit check & safe scripts"
> approach, though I do feel like it's more work than it's worth
> compared to the alternative.

What's the alternative?

   Ian

Ian Bicking wrote:
> Robert Marianski wrote:
>>>> This works great assuming we always add things to zeo that are
>>>> idempotent. And when we are developing, we are usually more interested
>>>> in just getting something to work. This relies on us to keep this in 
>>>> the
>>>> back of our minds, and we are in a big surprise come deployment if we
>>>> slip. This is why this makes me nervous.
>>> That's why I thought it would be good to have to explicitly mark a 
>>> script as safe, and so those scripts that get run every build and we 
>>> know to be safe (like do_nothing) don't make us think unnecessarily 
>>> about the state of the ZODB.  Everything added after that would get 
>>> marked unsafe by default, unless a developer specifically puts the 
>>> thought into making it safe.
>>>
>>> (We'd have to document exactly what we mean by safe/unsafe of course)
>>
>> I'd be fine with us only running safe scripts.
>>
>> So then the zeo stuff would rougly look like:
>> 1. if something is running on the zeo port, don't try to start zeo
>>    (remember if you started it or not)
>> 2. run all safe scripts
>> 3. if zeo was started through fassembler, then stop it
>>
>> Correct?
> 
> I realized recently in a development scenario I had this problem, except 
> the ZEO instance already open was pointed at a different ZODB.  And I 
> really did need to run those scripts against the right ZODB.

is this during a fassembler (re)build?  or a migration?  we seem to be 
confusing the two slightly in this thread, i'd like to make sure we're on the 
same page.

> I can just figure this out with the port warnings, but is it also 
> possible to query ZEO about what ZODB/Data.fs it is using?  If it's not 
> the right ZODB, then we need to simply abort with an error.

this doesn't seem possible to me, at first glance.  not without embedding some 
sort of information into the app about what the current state of the database 
should be... and even then it's imperfect.

consider that i have two development rigs, tracking the same branches, set up 
(for some reason) on the same ports.  on one i run some migrations, but forget 
to shut down zeo when it's all done.  so then i trigger the 'zopectl run' 
migration script on the other install.  it can examine the database, but the 
best it's going to do is determine that the migration isn't necessary.  how is 
it going to know that there's another, unmigrated database sitting around 
somewhere, for which it was originally intended?  by checking the filesystem 
location of the current Data.fs?  seems questionable.

i remember in our original fassembler discussions, i made sure that having 
fassembler work in a case where zeo is running outside of a given build would 
be supported, since that's the ultimate goal for our live deployments.  having 
the zeo built anew each time is great for bootstrapping a dev instance, but i 
think it's a bit of an anti-pattern for production.  rather than trying to 
make the zeo-swapping bulletproof, i think our energy is better spent getting 
our production environments set up so that zeo is run outside of any 
particular build, handled as a separate service a'la mysql.

-r

Rob Miller wrote:
> Ian Bicking wrote:
>> Rob Miller wrote:
>>> Robert Marianski wrote:
>>>> On Fri, May 02, 2008 at 12:50:03AM -0500, Ian Bicking wrote:
>>>>> Rob Miller wrote:
>>>>>> Ethan Jucovy wrote:
>>>>>>> Hey,
>>>>>>>
>>>>>>> As rmarianski and I discovered today, the zeo installer in 
>>>>>>> fassembler
>>>>>>> really doesn't work well if zeo's port is already in use when
>>>>>>> fassembler builds zeo.  That's because fassembler tries to start 
>>>>>>> zeo,
>>>>>>> run a few scripts to add an openplans site, and then shut it down
>>>>>>> again before the build finishes, but it doesn't first check if the
>>>>>>> port is in use and ends up hanging there forever on the start_zeo()
>>>>>>> command.
>>>>>>>
>>>>>>> I suppose we could just fix this by having the script first try 
>>>>>>> to hit
>>>>>>> zeo's port and back off, maybe with an interactive prompt, if the
>>>>>>> port's already in use.  I kind of think this is the wrong approach,
>>>>>>> though -- it seems a bit too implicit and isn't really able to 
>>>>>>> get at
>>>>>>> the *real* question, which isn't "can I start zeo?" but rather "do I
>>>>>>> want to run these scripts at all?".  I'd rather have a flag of some
>>>>>>> sort like --parallel-deployment or --not-fresh-zodb that tells
>>>>>>> fassembler not to even try to run those scripts, or, alternatively,
>>>>>>> put the scripts in a separate fassembler command like
>>>>>>> fassembler:finish_new_databases.
>>>>>>>
>>>>>>> Does anyone have objections or alternate preferences here?  I feel
>>>>>>> very strongly that explicit is better than ... magical in this case,
>>>>>>> but I don't have any particular preference for *how* to go down the
>>>>>>> explicit path.  It is something we should address ASAP in any case,
>>>>>>> because it's pretty important for upgrading a site with minimal
>>>>>>> downtime.
>>>>>> i'd rather not depend on having to call fassembler with explicit 
>>>>>> flags, personally.  in fact, since the zopectl scripts are all 
>>>>>> idempotent, i'd be happy just with making the StartZeo task 
>>>>>> smarter; if it finds zeo is already running on the port, then it 
>>>>>> just lets it be, setting a flag so that the StopZeo step knows 
>>>>>> that it shouldn't actually try to shut down the server.
>>>>
>>>> This works great assuming we always add things to zeo that are
>>>> idempotent. And when we are developing, we are usually more interested
>>>> in just getting something to work. This relies on us to keep this in 
>>>> the
>>>> back of our minds, and we are in a big surprise come deployment if we
>>>> slip. This is why this makes me nervous.
>>>
>>> i can understand this.  it still doesn't worry me a bunch, though.  
>>> we're talking about initial database setup, not migrations.  we don't 
>>> use fassembler to run migrations.  we don't add steps to the initial 
>>> database setup very often.  when we do, it's typically stuff that 
>>> can't be done twice... it's much more likely that a poorly written 
>>> initialization step (e.g. one that tries to create an 'openplans' 
>>> site w/o checking to see if it's already there) is going to fail 
>>> rather than cause problems to an existing site.
>>
>> Maybe an adaptation of the let's-just-not-do-this solution, that Ethan 
>> seems to prefer, would simply be:
>>
>> * Combine StartZeo/RunZopectlScript/StopZeo
>> * Before starting ZEO, try to connect to it, and fail if we don't get 
>> connection refused.  If you fail you can continue, so long as all 
>> these pieces are one task.  By checking the port, the build fails 
>> explicitly (and recoverably) instead of just stalling as it currently 
>> does.
>>
>> I think this actually handles all the cases we currently care about.
> 
> so the build fails unless it can start and stop the zeo?  doesn't this 
> fail to meet our current need, which is to do a new build of the 
> software without actually having to bring down the production instance 
> that's already running?

The task gives an exception.  If you know it's okay, you can continue. 
I think that satisfies the needs for a production deployment.

We can also have it query about it, instead of an exception.  The effect 
would be the same, but the lack of an exception would make it look less 
scary.

   Ian

On Fri, May 2, 2008 at 1:59 PM, Rob Miller <robm@...> wrote:
>> Maybe an adaptation of the let's-just-not-do-this solution, that Ethan
>> seems to prefer, would simply be:
>>
>> * Combine StartZeo/RunZopectlScript/StopZeo
>> * Before starting ZEO, try to connect to it, and fail if we don't get
>> connection refused.  If you fail you can continue, so long as all these
>> pieces are one task.  By checking the port, the build fails explicitly (and
>> recoverably) instead of just stalling as it currently does.
>>
>> I think this actually handles all the cases we currently care about.
>
> so the build fails unless it can start and stop the zeo?  doesn't this fail
> to meet our current need, which is to do a new build of the software without
> actually having to bring down the production instance that's already
> running?

Yes, the build fails, but it does meet our current need, because the
build fails recoverably since the failure is on a single "command" --
so you just hit "c" and the build moves on to the next step.  You just
skip the run-zeo-scripts step, which is exactly what we want to do in
the current situation.

This sounds pretty good me, though I think it's a bit confusing since
it *looks* like a failure rather than *asking* you if it's a failure.

egj