* Now talking on #topp-talks * Topic for #topp-talks is: TOPP Talks! Today's Talk: Performance Testing à la WOPR (thanks funky_c!) | Transcripts in patented Really Real Time ™ | http://openplans.org/projects/topp-talks * Topic for #topp-talks set by dwins at Thu May 15 09:51:50 2008 * #topp-talks :[freenode-info] channel flooding and no channel staff around to help? please check with freenode support: http://freenode.net/faq.shtml#gettinghelp mchua: you made it! sorry remote folks, Tim hasn't gotten approval from all of his sources so the presentation isn't online yet will attach to wiki as soon as possible okay, talk starts Okay, so a couple of weeks ago I was out for a week at the WOPR conference at Google's NY office What is WOPR? You've heard of the sandwich, but that's not it It's a performance testing workshop LAWST-inspired (Los Altos Workshop on Software testing) this WOPR's focus: performance testing an innovation So, we'll start with an intro to testing in general Testing: an investigative process meant to provide stakeholders with quality-related information Performance Testing: s/quality/ performance quality/ The role of the tester is not to ensure that the tested system is good, but to determine how good it is (or isn't) So, what is quality? According to Jude McQuaid, Intuit: * Does it float in water? (ie, work) * Can it stay floating in water? (ie, 'not break') * Even under heavy loads? * How about in a disaster? So, OpenPlans is probably at the 'stay floating' stage of development right now Quality according to Jerry Weinberg: "Quality is value to some person." for example, I just got a new camera, so my old one has very little value to me * hands to ethan Hey, this doesn't turn on! So yeah, anyone see any value there? *4 hands go up* Okay, so can we predit how our software will work *under load* on a live system? We don't want to do performance testing on the live site, so we make predictions based on a staging/development environment All right, so WOPR was divided into PRE-WOPR and POST-WOPR During pre-wopr Gustavo Vasquez showed us these youtube video *s * shows athletes competing in a 100-meter dash Questions: Who won? How do we know? feel free to chime in, remote folks Which factors should we measure to determine who won? food he had for dinner, sleep he had last night, shoes, amount of clothing * Shows horses racing One horse pulls 15 lengths ahead the horses enter the back-stretch that first horse starts to lose ground and ends last So we had maybe a functional problem, maybe a requirements misunderstanding here but the horse that seemed to be doing really well at the beginning performed quite poorly in the long run So what factors do we want to measure in horse racing? How do we explain this failure? Tizzicoop (the losing horse) tuned performance for the wrong scenario other factors include shoes, sleep, food, the jockey's weight, etc. all these things factor into what makes one horse better than the others anyway, one more video * Shows a stock car race cars navigating a wide range of curves and tight turns hey that guy only won because he's got a ton of money backing his car cars, track, engine, aerodynamics, etc. so, one more example. What about OpenCore? the wind the track! what factors do we have to take into account? we have databases, zope, things talking over HTTP lots of factors to deal with Gratuitous tools list! jmeter, grinder, opensta All of these measure and give data about software performance * Shows a performance chart for a piece of software that starts at a particular level, dips, then levels off, and finally starts to increase without apparent bound that point where the performance starts to increase is called the 'knee', and basically it indicates that you're not too far from catastrophic failure btw, the chart shows response time vs. # concurrent users for a particular site (see http://www.informit.com/articles/article.aspx?p=391645 ) I haven't done much performance testing myself, so I can't show you any statistics I've produced Current troubles in the industry testing is overlooked experts don't agree there's no set of 'basic skills' management doesn't understand the tester's role communication problems within organizations teaching/training problems http://www.io.com/~wazmo/papers/four_schools.pdf www.testingeducation.org/a/TheOngoingRevolution.pdf *raises hand for question* - what is the relationship between testers and developers? between testing and development, if testers *are* developers? And under what conditions/metrics is it optimal to have testers be developers vs having separate test/dev groups? good question, I don't have a ready-made statement I can't say that testers have to be separate developers (or the other way) but I have to say that you do what works for your situation ie, it's good to have testers who aren't developers, but if financial/other constraints mean you can't hire them then you do what you have to mchua: any further questions on that? 'basic skills' for performance testers * about 20 topics introspective * how do we define quality for OpenCore? * how do we ensure that quality? * what is the role of the tester within TOPP? * can we answer the Big Question? (how can we predict the performance of how our system performs under load?) * what is 'good enough' performance for OpenCore? okay, floor is open for questions from the audience It seems like you just run the thing, and testing just looks at what you've got. That is, how do you determine the appropriate environment in which testing should be done? Well, that's tough and I don't have enough training to give a great answer. I do want to assert that there is no 'completion' of testing for a piece of software. We measure a few well-known characteristics like 'throughput' (how many requests can we handle per unit time) and 'response time' You want to create graphs showing the data that you measure and communicate those characteristics to those who will be making decisions. How do you feel about optimizing before performance testing? What I've heard is that you shouldn't work on performance issues until you have performance issues. Hopefully those will show up early in development though. Those tools you listed, are they just profiling tools? I haven't used them, I think they help to create 'high-load' situations I wanted to know what facilities they provided for viewing the measured data, etc. what can developers do to make life easier for testing and vice versa? (yeah, I'm stuck on the dev/tester relationship atm ;) So one thing that was said at WOPR was "forget all the statistic analysis tools out there, Excel is the best there is" I don't know in some organizations testers and developers fight (i don't know why, aren't they working toward the same goal?) I understand some testers don't communicate effectively and then developers get mad (or the other way around) and I think what dev's need to understand is that testers are just pointing out facts about the system (not pointing fingers) mchua? dwins: so testers should try to communicate as neutrally as possible with non-finger-pointing disclaimers and not "I think the problem is caused by X" guesses? one thing I've heard is that developers should think of bug reports as ads for things that are all competing for work bug advocacy ftw! I don't know if finger-pointing helps at all * explains that there are psychological/emotional issues at work here amusing mental image: testers are like tv producers coming up with shows and things to put up on your screen, bugmaster is like the network who decides what to put in what slot, what to hang posters promoting, etc. and developers view the channels and watch the shows and then act on them it seems like some of those people problems can be solved with technological tools like automatically blaming based on subversion in geoserver we have stuff delegated to particular people as well, that also helps depersonalize blame pt.2: So if we have tickets as pitching tasks to compete for developers, we raise the questions of who should make that call in openplans we have a release manager, (ethan) who gathers all the tickets in trac and decides which priorities and developers to assign for bugs ethan has work concerns, business concerns (gathered by discussing with management) and ultimately both of those influences should affect which tickets are fixed thanks for coming remote folks, transcript will be up soonish (i don't have a cleanup script for this irc client's log format so it won't be immediate) thanks for the Really Real Time transcription, dwins! any time, mel well, except for imaginary times !DSPAM:4040,482c76c734518362916074!