I currently host my Vim Javadoc doclet on SourceForge and every time I have to deal with it, it's just a monumental pain. The documentation is insanely long and detailed, the website looks horribly out of date and cruddy, and when compared to stuff like GitHub and Lighthouse, it's almost embarrassing how difficult it is to deal with and how bad the UI is (despite my best efforts, it still insists that the featured download is the vimdoc samples and not the doclet itself. WTF?).

I'm already hosting the code in GitHub and just moved my tickets to Lighthouse. The only thing left is where to host binary downloads and static assets. For RESTUnit, I'm using Google Code, which is pretty easy to deal with (about a zillion times simpler and easier than SourceForge), however it has no facility for hosting arbitrary HTML. Currently, I'm just using my website for static assets.

While I do like the new Web-2.0 way of doing things (one site like GitHub really focusing on source, another like Lighthouse just does ticketing, etc. and they integrate via web services), I'm not sure where the best place is to host downloads and static assets. I would need programmatic access and some liberal download/diskspace quotas for sure. It would also be nice to be able to connect to other services, for example generate a changelog based on commits or tickets closed since the last release.

In my reply to a post on Tim Bray's blog about using RSpec for testing REST services, I briefly described a project I'm working on, based on the work I've been doing at Gliffy, which is a testing framework for REST services called, unsurprisingly, RestUNIT.

For Gliffy's REST-based integration API, I needed a way to test it, and hand-coding test cases using HTTPClient was just not going to cut it. Further, requests to Gliffy's API require signing (similar to how Flickr does it), and our API was going to support multiple ways of specifying the representation type as well as tunneling over POST.

So, it occured to me that there was a lazier way of doing this testing. All I really needed to specify was the relative URL, parameters, headers, method, and expected response. Someone else could do the signing and re-run the tests with the various options (such as specifying the MIME Type via the Accept: header, and then again via a file extension in the URL).

I ended up creating a bunch of text files with this information. I then used a Ruby script to generate two things: an XML file that could be deserialized into a java object useful for testing, and a PHP script to test our PHP client API.

The Ruby script would also do things like calculate the signature (the test text files contained the api and secret keys a Gliffy user would have to use the API) and generate some derivative tests (e.g. one using a DELETE, and another tunneling that over POST). The testing engine could generate some additional derivative tests (e.g. GET requests should respond to conditional gets if the server sent back an ETag or Last-Modified header). All this then runs as a TestNG test.

The whole thing works well, but is pretty hackish. So, RestUNIT was created as a fresh codebase to create a more stable and useful testing engine. My hope is to specify tests as YAML or some other human-readable markup, instead of XML (which is essentially binary for any real-sized data) and to allow for more sophisticated means of comparing results, deriving tests, and running outside a container (all the Gliffy tests require a specific data set and run in-container).

I'm currently working the integration API for Gliffy, which is a REST-based service. The API is fairly stable and we're readying a few ancillary things for release. One of those is the documentation for the API. I found it quite difficult to completely describe the REST services and ultimately ended up creating something that lists out "objects" and "methods", even though the API is not really object-based. For example, the object "Diagram" has a "method" called "list"; to "call" it, you do an HTTP GET to accounts/your account name/diagrams.

The original spec I created to work against (and thus, our initial draft of API documentation) was basically a list of URLs and the HTTP methods they responded to. Not very easy to navigate or understand on a first sitting. Some sort of schema to describe the REST API would have been really helpful (along the lines of an XML Schema). Such a schema could facilitate documentation, testing, code generation.

As an example, consider some features of the Gliffy API: you can list the users in an account, list the diagrams in an account and reference an individual diagram via id. Here's a YAML-esque description of these services:

<b>accounts:</b>

  <b>kind:</b> literal

  <b>desc:</b> <i>"Reference to all accounts"</i>

  <b>POST:</b>

    <b>desc:</b> <i>"Creates a new account"</i>

    <b>parameters:</b>

        - account_name

            <b>required:</b> true

            <b>desc:</b> <i>"Name of the account you want to create"</i>

        - admin_email

            <b>required:</b> true

            <b>desc:</b> <i>"Email address of an administrator for the new account"</i>

  <b>children:</b>

     <b>account_name:</b>

       <b>kind:</b> variable

       <b>desc:</b> <i>"The name of your account"</i>

       <b>GET:</b>

         <b>desc:</b> <i>"Returns meta-data about the account"</i>

         <b>parameters:</b>

            - show_users

              <b>required:</b> false

              <b>desc:</b> <i>"If true, users are included, if false, they are not"</i>

       <b>children:</b>

         <b>diagrams:</b>

           <b>kind:</b> literal

           <b>desc:</b> <i>"All diagrams in the account"</i>

           <b>POST:</b>

             <b>desc:</b> <i>"Creates a new diagram"</i>

             <b>parameters:</b>

               - diagram_name

                 <b>required:</b> true

                 <b>desc:</b> <i>"Desired name for this diagram"</i>

               - template_id

                 <b>required:</b> false

                 <b>type:</b> numeric

                 <b>dsec:</b> <i>"If present, the id of the diagram to copy, instead of using the blank one"</i>

           <b>GET:</b>

             <b>desc:</b> <i>"Gets a list of all diagrams in this account"</i>

           <b>children:</b>

             <b>id:</b>

               <b>kind:</b> variable

               <b>type:</b> numeric

               desc <i>"The id of a particular diagram"</i>

               <b>GET:</b>

                 <b>desc:</b> <i>"Gets the diagram; the requested encoding type will determine the form"</i>

                 <b>parameters:</b>

                   - <b>version:</b> 

                     <b>desc:</b> <i>"The version to get, 1 is the original version.  If omitted, current version is retrieved"</i>

                     <b>required:</b> false

                     <b>type:</b> numeric

                   - <b>size:</b>

                     <b>desc:</b> "For rastered formats, determins the size

                     <b>type:</b> enumeration

                       - L

                       - M

                       - S

               <b>DELETE:</b>

                 <b>desc:</b> <i>"Deletes this image"</i>

          <b>users:</b>

            <b>kind:</b> literal

            <b>desc:</b> <i>"All users in the account"</i>

            <b>GET:</b>

              <b>desc:</b> <i>"gets a list of all users in the account"</i>

A standard format like this could easily be used to generate documentation, expectations, test cases, and even stub code. This format could even be delivered by an OPTIONS call to a resource. I realize there is not much standardization around how to design and implement a REST service, but something like this could at least be a stake in the ground and support a specific method.

Following on from my post on Gliffy's blog...

On more than a few occasions, I've been faced with making significant refactorings to an existing application. These are things where we need to overhaul an architectural component without breaking anything, or changing the application's features. For an applicaiton without any test cases, this is not only scary, but ill-advised.

I believe this is the primary reason that development shops hang on to out-dated technology. I got a job at a web development shop after four years of doing nothing but Swing and J2EE. My last experience with Java web development was Servlets, JSPs and taglibs. This company was still using these as the primary components of their architecture. No Struts, no Spring, no SEAM. Why? One reason was that they had no test infrastructure and therefore not ability to refactor anything.

Doing it anyway

Nevertheless, sometimes the benefits outweigh the costs and you really need to make a change. At Gliffy, I was hired to create an API to integrate editing Gliffy diagrams into the workflow of other applications. After a review of their code and architecture, the principals and I decided that the database layer needed an overhaul. It was using JDBC/SQL and had become difficult to change (especially to the new guy: me). I suggested moving to the Java Persistence Architecture (backed by Hibernate), and they agreed. Only problem was how to make sure I didn't break anything. They didn't have automated tests, and I was totally new to the application environment.

They did have test scripts for testers to follow that would hit various parts of the application. Coming from my previous enviornment, that in and of itself was amazing. Since the application communicates with the server entirely via HTTP POST, and recieves mostly XML back, I figured I could manually execute the tests and record them in a way so they could be played back later as regression tests.

Recording Tests

This is suprisingly easy thanks to the filtering features of the Servlet specification:

<filter>
  <filter-name>recorder</filter-name>
  <filter-class>com.gliffy.test.online.RecordServletFilter</filter-class>
</filter>

<!-- ... -->

<filter-mapping>
  <filter-name>recorder</filter-name>
  <url-pattern>/*</url-pattern>
</filter-mapping>

The request proxy had to read everything from the requests input stream, save it, and send a new stream that would output the same data to the caller. It had to do the same thing with the Reader. I'm sure it's an error to use both in the same request, and Gliffy's code didn't do that, so this worked well.

private class RecordingServletRequest extends javax.servlet.http.HttpServletRequestWrapper
{
    BufferedReader reader = null;
    ServletInputStream inputStream = null;

    String readerContent = null;
    byte inputStreamContent[] = null;

    public RecordingServletRequest(HttpServletRequest r) { super(r); }

    public BufferedReader getReader()
        throws IOException
    {
        if (reader == null)
        {
            StringWriter writer = new StringWriter();
            BufferedReader superReader = super.getReader();
            int ch = superReader.read();
            while (ch != -1)
            {
                writer.write(ch);
                ch = superReader.read();
            }
            readerContent = writer.toString();
            return new BufferedReader(new StringReader(readerContent));
        }
        return reader;
    }

    public ServletInputStream getInputStream()
        throws IOException
    {
        if (inputStream == null)
        {
            ByteArrayOutputStream os = new ByteArrayOutputStream();
            ServletInputStream superInputStream = super.getInputStream();
            int b = superInputStream.read();
            while (b != -1)
            {
                os.write(b);
                b = superInputStream.read();
            }
            inputStreamContent = os.toByteArray();
            inputStream = new ByteArrayServletInputStream(inputStreamContent);
        }
        return inputStream;
    }
}

The response recorder was a bit trickier, because I needed to save things like status codes and content types. This implementation probably wouldn't work for all clients (for example, it ignores any response headers), but since Gliffy is an OpenLaszlo app, and OpenLaszlo has almost no view into HTTP, this worked well for our purposes. Again, I had to wrap the OutputStream/Writer so I could record what was being sent back.

    private class RecordingServletResponse extends HttpServletResponseWrapper
{
    public RecordingServletResponse(HttpServletResponse r)
    {
        super(r);
    }

    int statusCode;
    StringWriter stringWriter = null;
    ByteArrayOutputStream byteOutputStream = null;
    String contentType = null;

    private PrintWriter writer = null;
    private ServletOutputStream outputStream = null;

    public ServletOutputStream getOutputStream()
        throws IOException
    {
        if (outputStream == null)
        {
            byteOutputStream = new ByteArrayOutputStream();
            outputStream = new RecordingServletOutputStream(super.getOutputStream(),new PrintStream(byteOutputStream));
        }
        return outputStream;
    }

    public PrintWriter getWriter()
        throws IOException
    {
        if (writer == null)
        {
            stringWriter = new StringWriter();
            writer = new RecordingPrintWriter(super.getWriter(),new PrintWriter(stringWriter));
        }
        return writer;
    }

    public void sendError(int sc)
        throws IOException
    {
        statusCode = sc;
        super.sendError(sc);
    }

    public void sendError(int sc, String msg)
        throws IOException
    {
        statusCode = sc;
        super.sendError(sc,msg);
    }

    public void setStatus(int sc)
    {
        statusCode = sc;
        super.setStatus(sc);
    }

    public void setContentType(String type)
    {
        contentType = type;
        super.setContentType(type);
    }
}

The filter then needs to use this and inject them into the actual servlet calls:

public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
    throws IOException, ServletException
{
    RecordingServletRequest recordingRequest = 
      new RecordingServletRequest((HttpServletRequest)request);
    RecordingServletResponse recordingResponse = 
      new RecordingServletResponse((HttpServletResponse)response);

    chain.doFilter(recordingRequest,recordingResponse);
    

Running the Tests

There were a few problems with this approach:

• The tests required certain test data to exist
• Each test potentially modifies the database, meaning the tests have to be run in the order they were created.
• The test results had temporal data in them that, while irrelevant to the tests "passing", complicated exact-match comparisions of results

I ultimately decided to group my tests into logical areas, and ensure that: a) tests were run in a predictable order, and b) the first test of a group was run against a known dataset. I created a small, but useful, test dataset and created a TestNG test that would do both (a) and (b). It wasn't pretty, but it worked. This clearly isn't the way a unit test framework should be used, and I would call these sorts of tests functional, rather than unit. But, since our CI system requires JUnit test results as output, and the JUnit format isn't documented, might as well use TestNG to handle it for me.

The last problem was making accurate comparisons of results. I did not want to have to parse the XML returned by the server. I settled on some regular expressions that stripped out temporal and transient data not relevant to the test. Both the expected and received content were run through this regexp filter and those results were compared. Parsing the XML might result in better failure messages (right now I have to do a visual diff, which is a pain), but I wasn't convinced that the existing XML diff tools were that useful.

Results

Overall, it worked out great. I was able to completely overhaul the database layer, and the Gliffy client was none the wiser. We were even able to use these tests to remove our dependence on Struts, simplifying the application's deployment (we weren't using many features of Struts anyway). The final validation of these tests actually came recently, when we realized a join table needed to be exposed to our server-side code. This was a major change in two key data container, and the recorded tests were crucial to finding bugs this introduced.

So, if you don't have the luxury of automated tests, you can always create them. I did a similar thing with EJB3 using the Interceptors concept.

My current project is slowly moving from JDBC-based database interaction to JPA-based. Following good sense, I’m trying to change things as little as possible. One of those things is that we are deploying under Tomcat and not under a full-blown J2EE container. This means that EJB3 is out. After my post regarding this configuration, I quickly realized that my code started to get littered with:

EntityManager em = null;
try
{
  em = EntityManagerUtil.getEntityManager();
  // do stuff with entity manager
}
finally
{
  try {
    if (em != null) em.close();
  } catch (Throwable t) {
    logger.error("While closing an EntityManager",t);
  }
}

Pretty ugly, and seriously annoying to have to add 13 lines of code to any method that needs to interact with the database. The Hibernate docs suggest using ThreadLocal variables to provide access to the EntityManager throughout the life of a request (which wouldn’t really work for a Swing app, but since this is servlet-based, it should work fine). The ThreadLocal javadocs contain possibly the most annoying example ever, and I didn’t follow how to use it.

Anyway, I finally got around to it, and also solved the close problem as well, by using a Servlet Filter. I guess this type of thing would normally be solvable by Spring or Guice, but I didn’t want to drag all of that into the application to refactor this one thing; I would’ve easily spent the rest of the day dealing with XML configuration and deployment.

The solution was quite simple:

/** Provides access to the entity manager.  */
public class EntityManagerUtil 
{
    public static final ThreadLocal<EntityManager> 
        ENTITY_MANAGERS = new ThreadLocal<EntityManager>();

    /** Returns a fresh EntityManager */
    public static EntityManager getEntityManager()
    {
        return ENTITY_MANAGERS.get();
    }
}

public class EntityManagerFilter implements Filter
{
    private Logger itsLogger = Logger.getLogger(getClass().getName());
    private static EntityManagerFactory theEntityManagerFactory = null;

    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
        throws IOException, ServletException
    {
        EntityManager em = null;
        try
        {
            em = theEntityManagerFactory.createEntityManager();
            EntityManagerUtil.ENTITY_MANAGERS.set(em);
            chain.doFilter(request,response);
            EntityManagerUtil.ENTITY_MANAGERS.remove();
        }
        finally
        {
            try 
            { 
                if (em != null) 
                    em.close(); 
            }
            catch (Throwable t) { 
                itsLogger.error("While closing an EntityManager",t); 
            }
        }
    }
    public void init(FilterConfig config)
    {
        destroy();
        theEntityManagerFactory = 
          Persistence.createEntityManagerFactory("gliffy");
    }
    public void destroy()
    {
        if (theEntityManagerFactory != null)
            theEntityManagerFactory.close();
    }
}

So, when the web app gets deployed, the entity manager factory is created (and closed when the web app is removed). Each thread that calls EntityManagerUtil to get an EntityManager gets a fresh one that persists for the duration of the request. When the request is completed, the entity manager is closed automatically.

• Create a persistence.xml file as per standard documentation leaving out the jta-data-source stanza (I could not figure out how to get Hibernate/JPA to find my configured data source)
• Create your hibernate.cfg.xml, being sure to include JDBC conncetion info. This will result in hibernate managing connections for you, which is fine
• Create a persistence jar containing:
  • Hibernate config at root
  • persistence.xml in META-INF
  • All classes with JPA annotations in root (obviously in their java package/directory structure)
• This goes into WEB-INF/lib of the war file (being careful to omit the JPA-annotated classes from WEB-INF/classes

EntityManagerFactory emf = 
    Persistence.createEntityManagerFactory("name used in persistence.xml");
EntityManager em = emf.createEntityManager(); 

Query query = em.createQuery("from Account where name = :name");
query.setParameter("name",itsAccountName);
List results = query.getResultList();

// do stuff with your results


em.close();
emf.close();

EntityManagerFactory emf = 
    Persistence.createEntityManagerFactory("name used in persistence.xml");
EntityManager em = emf.createEntityManager(); 
EntityTransaction tx = em.getTransaction();

tx.begin();
Query query = em.createQuery("from Account where name = :name");
query.setParameter("name",itsAccountName);
List results = query.getResultList();

// modify your results somehow via persist() 

// or merge()


tx.commit();
em.close();
emf.close();

git-checkout -b local-trunk trunk
git branch local-foo FOO

Both Amazon Web Services and the Flickr Services provide REST APIs to their services. I’m currently working on developing such a service, and noticed that both use signatures based on a shared secret to provide security (basically using a Hash Message Authentication Code).

It works as follows:

1. Applications receive a shared secret known only to them and the service provider.
2. A request is constructed (either a URL or a query string)
3. A digest/hash is created using the shared secret, based on the request (for Flickr, the parameter keys and values are assembled in a certain way, so that Flickr can easily generate the same string)
4. The digest is included in the request
5. The service provider, using the shared secret, creates a digest/hash on the request it receives
6. If the service provider’s signature matches the one included in the request, the request is serviced

It’s actually quite simple, and for one-time requests, is effective. The problem, however, is that anyone intercepting the request can make it themselves, without some other state being shared with the client and service provider. Consider a request for an image. The unsigned request might look like:

/api/images?image_id=45&type=jpg

The signed request, would look like so:

/api/images?image_id=45&type=jpg&signature=34729347298473

So, anyone can then take that URL and request the resource. They don’t need to know the shared secret, or the signature algorithm. This is a bit of a problem. One of the advantages of REST is that URLs that request resources are static and can be cached (much as WWW resources are). So, if I wish to protect the given URL, how can I do so?

HTTP Authentication

The usual answer is HTTP Authentication; the service provide protects the resource, and the client must first log in. Login can be done programmatically, and this basically accomplishes sending a second shared secret with the request that cannot be easily intercepted. HTTP Auth has its issues, however, and might not be feasible in every context.

Another way to address this is to provide an additional piece of data that makes each request unique and usable only once. To do so requires state to be saved on the client and the server.

Negotiated One-time Token

Authentication can be avoided by using the shared secret to establish a token, usable for one request of the given resource. It would work like this:

1. Client requests a token for a given resource
2. Service Provider creates a token (via some uuid algorithm ensuring no repeats) and associates it with the resource
3. Client creates a second request, as above, for the resource, including the token in the request
4. Service Provider checks not just for a valid signature, but also that the provided token is associated with the given resource
5. If so, the token is retired, and the resource data is returned

Here, the URL constructed in step 3 can be used only once. Anyone intercepting the request can’t make it again, without constructing a new one, which they would be unable to do without the shared secret. Further, this doesn’t preclude caching. The main issue here is that since two requests are required, simultaneous access to one resource could result in false errors: if Client A acquires a token, and Client B requests one before Client A uses the token, Client A’s token could be squashed, resulting in an error when he makes his request. The service provider can alleviate this by allowing the issuance of multiple active tokens per resource.

Timestamp

A disadvantage to the One-Time Token method is that it requires two requests of the service provider for every actual request (one to get the token and one to request the resource). A way around that is to include a timestamp in the request. This would work as follows:

1. Client creates request, including the current time. This request is signed as per above procedure
2. Service provider validates the request and compares it’s time with the given timestamp.
3. If the difference in the service provider’s time and the client’s provided time is within some tolerance, the request is serviced

This obviously requires the two clocks to be vaguely in sync. It also allows the resource to be requested by anyone within the timespan of the tolerance. But, it does save a second request to the client.

Self-created One-time Token

This is an amalgam of the Timestamp solution and the Negotiated One-time Token solution. Here, the client creates its own token, as a simple integer of increasing value. The server maintains the last requested value and accepts only requests with a higher number:

1. Client creates request, using a global long-lived number
2. Client signs requests and sends it to the service provider
3. Service provider validates the signature and compares the provided numeric token with the one last used (the tokens can be globally scoped, or scoped for a given resource)
4. If the provided numeric token is greater than the previous, the request is serviced
5. The Client increments his numeric token for next time

As with the Timestamp solution, only one request is required. As with the negotiated one-time token solution, the URL can never be used twice. The main issue here is if the client forgets its numeric token. This could be addressed with an additional call to re-establish the token, made only when the Client has determined it no longer knows the last used value.

Unfortunately, this is much more susceptible to race conditions than the Negotiated one-time token. Since the service provider doesn’t know what tokens to expect (only that they should be greater than the last requested one), the client has to ensure that the “create request, submit request, receive response, update local numeric token” cycle is atomic. That is not straightforward.

Update Got another idea from a co-worker

Session Token

When a user access the system that uses the REST API, they get issued a token (via the REST API). This token is just like a session token, with an inactivity timeout and so forth. The token can be manually invalidated via the API, so that when a user logs out or completes some logical task, the token can be invalidated.

This suffers none of the problems of the other solutions, though it isn’t the most secure. However, the security problem it has (using the valid URL before the session times out) is fairly minor, and the tradeoff of getting one request per actual request and no race conditions makes it probably the best way to go.