• Web Services: Making Plone talk with other sites

  last modified October 15, 2007 by skleinfeldt

Idea

Web services and mashups continue to grow in importance and prominence. This sprint topic covers the following needs:

  • We need a way to web-service-enable Plone sites so that people can easily  create microapps out of data extracted from Plone events, news, blog posts, mp3s, videos, and other (custom and standard) content types. 
  • We would like the same approach to work for exporting and importing content.
  • We also need an easy way to create Plone microapps that talk with various Internet web services (like google, flickr, twitter, facebook), many of which already have Python APIs.


Existing Products

We plan to add to and/or merge products that are in this space.  The products include:

  • ATRest - Created by Kapil, Sally K., and David S. at the Boston Plone4Artists sprint. Adds a new Zope 3 view giving every piece of AT content (folders and non-folders) a read-only RESTful web service interface.  It relies on the Marshall product to do almost all of the work.  For non-folders the interface includes all AT fields; for folders and Smart Folders this includes the title and some basic metadata about the contained objects.
  • GSXML - Created by Ramon Bartl and Stefan Eletzhofer (InQuant GmbH). Uses GenericSetup to import and export ATCT based Content as pure XML data.  Similar to ATRest, it piggybacks on existing, well tested Products, like Marshall and GenericSetup, combining and extending them through Zope 3's Component Architecture.


Sprinters

  • Stefan Eletzhofer
  • Francesco Merlot
  • David Siedband
  • Sally Kleinfeldt
With help from
  • Ramon Navarro

Initial Work

  • Test if ATRest does work on a Plone 3 buildout and have a peek at the code
  • Do a writeup on the code of ATRest
  • Check out GSXML
  • Have a look at the GSXML code
  • Think about how to marry/extend/improve ATRest with GSXML, or if there's need to do that or not.

ATRest Installation

http://svn.plone.org/svn/collective/ATRest/trunk

  • Apparently the ATMarshaller of Plone 3 (trunk) is broken as it writes binary data to xml, i.e. NOT base64 encoded
  • Basically the ATRest Product does work (see below)

ATRest Code Overview

  • ATRest is still an old-style "Product"
  • It just uses browser views to hook up with ATXML Marshaller
  • It defines a view for folder listing and topic which are manually generated XML from folderContents() and queryCatalog()
  • It defines a base view for all IBaseObject which hooks in the ATXML marshaller

GSXML installation

http://svn.plone.org/svn/collective/gsxml/trunk/

  • Installs painlessly on Plone 3.0 trunk
  • Import/export works

GSXML Code Overview

  • GSXML uses GenericSetup to Import and Export ATCT based Content as pure XML data
  • Makes heavy use of Five to avoid starting from scratch with another Import/Export Product
  • Uses benefits of already existing, well tested Products, like Marshall and GenericSetup, with the new Zope 3 Component Architecture approach, to extend the existing functionality and melt it together to one product.


Buildout

Created a Plone 3 buildout that includes both ATRest and GSXML.
http://svn.plone.org/svn/collective/ATRest/buildout/

Results of Lunch talk

At lunch we discussed several use cases the sprint group has. These are in no particular order:
  • Web 2.0 services
    • Clients from any language/platform can easily make mashup applications from Plone content
  • Export content
    • Your manager is worried about the company's valuable content being buried inside this mysterious ZODB, and wants a filesystem dump of the content.
  • Synchronize content
    • From a standalone Plone site on a laptop that was used in the field to collect data, export the content/data from the laptop Plone site, and import to a central Plone site
  • Migrate content types
    • Export all the ATArticles from a Plone site, and import as ATDocument


Requirements

Provide read-only RESTful web services on Plone content
  • From any Archetypes object on a Plone site, do a HTML GET on an XML representation of that object (for example, http://mysite/path/to/object/xml => XML representation)
  • If the object has image, file, or similar binary fields, the XML representation will not include the binary content.  Instead the binary content will be represented by an XML "xlink" to the URI for the binary object.
  • If the object is folderish, the XML representation will provide the attributes of the folder, plus the contents of the folder as xlinks to URIs.  Each child object's XML representation will include, in addtion to xlink, attributes such as title (typically available in the object's brains) sufficient to make a reasonable UI.
  • Extend the RESTful web services to be writable.  This means being able to go to any Archetypes object on a Plone site and do a HTML PUT on an XML representation of that object.  This would edit the object, or if it is a new object, add it.
  • Be able to mark each field in an AT schema, whether or not it should be exposed via XML.  (Permissions will handle permission issues, but this can be used, for example, if you have a large, complex schema and you only want to expose a subset of the fields most useful for mashups.)
Provide read-write RESTful web services on Plone content
  • From any Archetypes object on a Plone site, do a HTML PUT on an XML representation of that object.  This would edit the object, or if it is a new object, add it.


Design

We decided to explore a single solution for both use cases.  We wanted to base our solution on what Marshall does (creates xml for AT content fields), but adding additional xml with proper xlinks for child objects.  Ramon helped us understand that it is this embedding xlinks with URIs that makes the solution completely RESTful.  After discussion, we decided to replace GSXML with this new solution.   Instead of putting everything into an old-style product like ATRest, we created a pure Python package, plone.rest, to do this work.

We also decided it was not wise to change Marshall, and add to it the additional xml (xlinks with URIs) we wanted.  Instead we decided to invent a way to inject our additional xml into the xml that Marshall creates.  Using this approach we can also filter out fields Marshall creates, if they are marked as not being xml-exportable.


Implementation

Reimplementing the ATRest functionality in plone.rest was easy. Creating proper xml snippets containing the child elements and their xlinks/URIs was also easy.  Merging the two using a proper xml library that creates valid xml proved difficult.  Elementtree clobbered the namespaces on all the Marshall-generated xml.  lxml blew up on macs.  We did not find a good solution to this problem.  Our next idea is to try to use libxml2.

You can find our work at
http://svn.plone.org/svn/collective/ATRest/buildout/naples-sprint