I’m really digging it so far and I want to give a big thanks to our awesome design team for pitching in on the project.

With the release of version 0.4, we’re also launching a slew of fixes, improvements to workflows and new features that have been patiently waiting in the wings including openid login, starred item view, tracking folks you know, and a much improved feed se­arch. Gory details can be found in the Changelog or the Trac Milestone. Now, back to the eyecandy…

­

 ­ ­­

­

SELECT * FROM sometable ORDER BY RAND() LIMIT ­samplesize

In MySQL, this appears to do just what it says: generate a random number for every row then sort them and return the lowest few — which is prohibitively slow for any reasonably big table.

If you know the identifiers of rows in the table and they form a gapless sequence, the story is somewhat better;  you can generate samplesize random numbers from the range and select those identifiers.  Of course, this is generally not the case (and almost begs the question).  The best approach along these lines that I’ve seen requires maintaining a table that maps from a gapless sequence of integers to the identifiers of the table that you want to sample from using a bunch of triggers — http://jan.kneschke.de/projects/mysql/order-by-rand .

I’m also unsure of the right way to tackle this in databases like couchdb or the appengine datastore.  Maybe a similar tactic with views/query indexes? Anyone implemented anything similar?

­­ ­ ­

Thankfully, there exists a very nice python-openid package. The documentation assumes some knowledge of OpenID specifications, so I had to do some digging there to understand how I was supposed to extract identity information from the response.

Here’s a nice screenshot of me logging in using OpenID:

The only tricky part to get working was Simple Registration. The OpenID specifications include a way to request Simple Registration information from the provider. In our case, we need a nickname and an e-mail address. Nicknames are used internally by Melkjug to find user’s jugs. For example, my full reading list exists at http://melkjug.org/tilgovi/full.

in some cases, the OpenID provider does not honor the Simple Registration request. In other cases, the user fails to supply the proper information. In these cases, Melkjug will ask for the additional required information before allowing you to sign up with your OpenID (see screenshot). If everything looks good, you’re in!

 Look out for OpenID logins in the next release of Melkjug.

The 0.3 release focuses on lowering barriers to entry for new users based on the work Sonali and Bryan have been putting in.  Most importantly, you can now check out the functionality of the site without creating an account and save what you make if you choose to sign up.  We’ve also added a more informative “about” page, made the text more consistent overall and put better blank slates in place.  There’s still a long way to go on this to get it where we want it, but I like how this first pass turned out.

For existing users, we’ve implemented some of the most requested features/fixes like sharing links via email / facebook etc, an option to keep you logged in after closing the browser, improvements to speed,  and elimination of certain annoying duplicate articles.  We have also moved all management of feeds back into the normal reading window and moved the detailed configuration of filters out of the sidebar and into lightbox dialogs.

Of particular note is Josh’s work to allow the tag filter to use multiple tags at once resulting from Bryan’s work trying to put together a decent demo.   This makes tag filtering… actually kind of useful.  Using this, you can group together related tags to represent a concept, or create clusters of synonymous tags to correct for differences between blogs.  Enjoy!

try it out at melkjug.org — download it at melkjug.openplans.org

As always, feedback is appreciated

On the cusp of the 0.3 release, coinciding roughly with my first month of full-time work on Melkjug, I thought it’d be a good time to blog a bit about what I’ve been up to. While wetting my whistle with cookies, controllers, and captchas, I’ve also been getting my first real taste of some not-so-small potatoes like Pylons, jQuery, and CouchDB. Somehow, somewhere along the way I almost started to feel like a web programmer. And that’s not all: Recently we stumbled upon the precipice of none other than Greenspun’s Tenth Rule, basked momentarily in its profound and infinite wisdom, and then plowed heedlessly ahead. My past few weeks have been as exciting as they have formative. And some people think programming is dull!

Now to talk about some of the more interesting things I’ve been working on…

The 0.3 release will include some work toward #173: directly email/share articles from melkjug to popular services. After a survey of some available options, Luke noted:

ShareThis has some blood surrounding logo takedown notices so I’m inclined to skip this one altogether on principle.

I’ve seen AddThis around and it seem to work and is extremely comprehensive. It’s probably recognizable and everyone who does this sort of thing would be comfortable with it. It’s also ugly and divulges information to a centralized server owned by some strange LLC.

I’ve never actually seen iBeginShare used, but it appears to be open source (MIT) decentralized and uses the alternative open sharing logo. It seems to cover most of the huge services.

So I went with iBeginShare and set to work writing a small integration layer to see how it felt. The code turned out to be a bit hard to work with, and the UI was also not as lightweight as we were looking for, so I decided to scrap it and roll my own

Here’s what I’m calling the “iBeginWhere?!” widget (working title):

clicking the green “open share” icon reveals the various services to its left

Still a work in progress UI-wise, but it’s a decent approximation.

Implementing this was educational in a number of different ways. For one, I got to play with jQuery (which I was excited to do after attending rmarianski’s excellent TOPP Talk). But hooking up the email button was particularly interesting. First, a screen shot:

email link form (logged-in)

Getting the above to happen was pretty easy using the javascript Luke had already written. But then we realized it would be nice if, when you filled out the form incorrectly, it would reappear with the errors indicated, like this:

This was interesting because it wasn’t the basic case of synchronous form validation. Instead, the user clicks the email button, and some javascript makes an Ajax request to fetch the form and then display it in that nice modal dialog. If the user submits an invalid form, Pylons has to respond with an error code as well as a snippet containing the form with the indicated errors, which the javascript then pops up again. This was easily the most Ajax I’ve ever done (as well as my first time using FormEncode), and it was pretty cool getting to see how it works.

Another thing that made this interesting was the non-logged-in case: Unauthenticated users should not be able to send emails without first solving a captcha. Looks a little something like this:

­

email link form (not logged in)

As you can see, we’re using recaptcha, which has a nice ajax api to generate the captcha. This was interesting not only because it was more ajax, but also because it complicated validation of the form: An incorrectly-solved captcha invalidates the form when the user is not logged in, but when the user is logged in, the captcha is not even present. How do you represent that with a FormEncode schema? After pairing with novalis on this (shout out), we found a nice solution: use the compound validator “Any” with two custom validators, one which validates iff the user is logged in, and another which actually makes the request to recaptcha.net to see if the captcha was solved correctly, and validates accordingly. The implementation of this is here.

I should also note that along the way Luke found and fixed a pretty critical bug in the recaptcha python plugin and submitted a patch to the recaptcha mailing list, but unfortunately we haven’t heard back yet. Here’s to good open source karma, anyway.

Working on this ticket was interesting for one last reason, because I got to work with MIMEMultiPart messages (which is what gets sent when a user actually fills out the form correctly). Here’s a screenshot of what the resulting email looks like:

“share link” email

The interesting thing about implementing this was figuring out how to correctly attach an html message and a plaintext message to the MimeMultiPart. It turns out that you have to attach the html message first, or else mail clients will show the plaintext message instead. Learned that one the hard way.

Also in the next release will be work toward #187: promote articles based on a set of tags. The motivation for this was a use case Bryan thought of while working on a demo of Melkjug for new users: Suppose Blog A uses one tag, but Blog B uses a different tag to refer to the same concept (not to name names). It would be great to have a single filter that controls the presence of articles with any of the several tags you’re interested in. Another use case for the same functionality would be if you wanted to group tags referring to different concepts that are still somehow related. For instance, baseball, golf, and competitive eating could all be grouped under “sports”. Enter the MultiTagFilter.

To effect this feature, we discussed the possibility of transfiguring the simple filters of the mixing spec into meta-filters, entities no longer limited to containing a single atomic filter, but rather allowed to contain an arbitrary function of other filters. The MultiTagFilter then just becomes some SingleTagFilters or’ed together. Implementing this was apparently tantamount to defining the lambda calculus (hence the reference to Greenspun’s Tenth), but for better or worse, we decided to do it anyway. Keep it under raps though, I hear he sicks his enormous Siberian dog on serious offenders. The upside, at least, now that filters can be arbitrarily composed, is that if we ever want to offer a MultiAuthorFilter, for example, it will be trivial.

The last thing I’ll mention is work on #176: logins should be able to last longer. We accomplished this as follows:

• set beaker session cookies to never expire
• add a random string to the session, the ‘authenticator’
• if when the user logs in they select ‘keep me logged in’, store an additional ‘melkauth’ cookie which contains the authenticator and never expires
• if the user restarts the browser and then hits melkjug, check if the authenticator in the melkauth cookie matches the authenticator in the session. if so, they will be logged in.

I’m blogging about this one because it was my first foray into cookie auth stuff and I found it interesting, but also to solicit feedback. For instance, I suspect this is overkill:

def _make_authenticator(self):
    return ''.join(random.choice(ascii_letters) for i in xrange(64)) # 52^64 distinct strings

If anyone with more experience and intuition about this wants to chime in, I’d love to hear it. :)

Thanks for reading,

Josh

App Engine is open to everyone now, but still in preview release.  They rolled out access to memcache, an image manipulation api and announced an expected pricing structure for resources beyond the free limits (details).  In the “fireside chat” they suggested an alternate pricing model for non-profits and education, but with no specifics.

They mentioned a timeline of “by the end of the year” for rolling out billing.  It’s worth noting it’s basically the only thing they would attach a timeline to, although their big priorities also seem to be offline processing and alternate language support.

The question whether you will be charged for the size of the indices that are created to service your queries appears to be a strangely touchy issue.  The word so far is they would “prefer not to” charge, but it appears this may be a matter of some internal debate?

There don’t seem to be any plans around offering versioned objects. Bigtable can do this, they don’t have the appengine’s tables configured this way. Versions up to the last committed are garbage collected.

The biggest goal is to eliminate (or significantly lower) the barrier to seeing what is special about the site when you arrive. An anonymous user should be able to show up, see some feeds being displayed and immediately get the experience that takes about 4 long boring and meaningless steps to get to currently (sign up, create a jug, add some feeds, add some filters). There should be something on the front page when you show up that is perhaps interesting and that you can immediately start refining without signing up or logging in. We allow you to save the customized thing what you’ve done if you sign up.

Meanwhile, in the non-user-facing realm, I’ve been a bit (okay… maybe more than a bit) distracted from the main line of development trying to absorb what Google App Engine is all about, how much it has to offer to melkjug and how much would need to be done to take advantage of it in a significant way. I’m really excited about it, and I think that certain portions of the application map very well onto what Google is offering. Big props to Ian for appengine-monkey.

In a lot of ways, the changes I put in place to accommodate CouchDb have also payed off in fiddling around with the app engine data api — in terms of managing data, and constraints on finding it, I find them to be deeply similar. (p.s. looks like opencore isn’t alone in hand drawn diagram dept) Put together, CouchDB and App Engine are really starting to shape my concept of what melkjug needs to look like on the inside if it’s going to handle more than a handful of users.

They’re very strange and restrictive environments in some respects, but I think at least some of these constraints are valuable to have right in front of you to prevent you from doing things that just really aren’t going to scale. I’m no expert on doing things in these ways, and I’m certain I’m doing things stupidly, but I think I’m starting to see the light… I’ll post more on this along with some of the prototypes I’m been playing around with at some point soon — poke me if you want them sooner :)

Give it a try at http://melkjug.com and don’t forget to give feedback!

• Feature #10 export feed set as OPML
• Feature #91 change title of a jug
• Feature #94 Use melkjug with Internet Explorer
• Feature #125 auto-generate or auto-fill jug ‘url’ field
• Feature #131 Choice algorithm should produce a balanced set of articles
• Feature #132 display component feeds on sidebar of feed view
• Feature #147 Display indicator of article loading
• Feature #159 pre-create one jug on signup
• Feature #160 see filter settings on a jug you don’t own
• Fix #96 No error message when adding a feed fails
• Fix #107 dismissed items sometimes blip back into existence momentarily
• Fix #111 Using back button in firefox brings back dismissed articles
• Fix #124 ’shadow jug’ page icon has wrong proportions
• Fix #127 Allow password reset
• Fix #129 Dismiss link at bottom of article
• Fix #149 There should be a url accessable subscription url
• Fix #152 usernames are case sensitive
• Fix #157 title bar displays wrong name
• Fix #158 dynamic page updates are not reflected when back button is hit
• Fix #163 forms are invisible in IE
• Fix #165 javascript does not reload when javascript changes
• Fix #168 Cannot import OPML
• Fix #169 Filtering on age is funky
• Work to address #171 Many jugs are too slow