Both Amazon Web Services and the Flickr Services provide REST APIs to their services. I’m currently working on developing such a service, and noticed that both use signatures based on a shared secret to provide security (basically using a Hash Message Authentication Code).

It works as follows:

1. Applications receive a shared secret known only to them and the service provider.
2. A request is constructed (either a URL or a query string)
3. A digest/hash is created using the shared secret, based on the request (for Flickr, the parameter keys and values are assembled in a certain way, so that Flickr can easily generate the same string)
4. The digest is included in the request
5. The service provider, using the shared secret, creates a digest/hash on the request it receives
6. If the service provider’s signature matches the one included in the request, the request is serviced

It’s actually quite simple, and for one-time requests, is effective. The problem, however, is that anyone intercepting the request can make it themselves, without some other state being shared with the client and service provider. Consider a request for an image. The unsigned request might look like:

/api/images?image_id=45&type=jpg

The signed request, would look like so:

/api/images?image_id=45&type=jpg&signature=34729347298473

So, anyone can then take that URL and request the resource. They don’t need to know the shared secret, or the signature algorithm. This is a bit of a problem. One of the advantages of REST is that URLs that request resources are static and can be cached (much as WWW resources are). So, if I wish to protect the given URL, how can I do so?

HTTP Authentication

The usual answer is HTTP Authentication; the service provide protects the resource, and the client must first log in. Login can be done programmatically, and this basically accomplishes sending a second shared secret with the request that cannot be easily intercepted. HTTP Auth has its issues, however, and might not be feasible in every context.

Another way to address this is to provide an additional piece of data that makes each request unique and usable only once. To do so requires state to be saved on the client and the server.

Negotiated One-time Token

Authentication can be avoided by using the shared secret to establish a token, usable for one request of the given resource. It would work like this:

1. Client requests a token for a given resource
2. Service Provider creates a token (via some uuid algorithm ensuring no repeats) and associates it with the resource
3. Client creates a second request, as above, for the resource, including the token in the request
4. Service Provider checks not just for a valid signature, but also that the provided token is associated with the given resource
5. If so, the token is retired, and the resource data is returned

Here, the URL constructed in step 3 can be used only once. Anyone intercepting the request can’t make it again, without constructing a new one, which they would be unable to do without the shared secret. Further, this doesn’t preclude caching. The main issue here is that since two requests are required, simultaneous access to one resource could result in false errors: if Client A acquires a token, and Client B requests one before Client A uses the token, Client A’s token could be squashed, resulting in an error when he makes his request. The service provider can alleviate this by allowing the issuance of multiple active tokens per resource.

Timestamp

A disadvantage to the One-Time Token method is that it requires two requests of the service provider for every actual request (one to get the token and one to request the resource). A way around that is to include a timestamp in the request. This would work as follows:

1. Client creates request, including the current time. This request is signed as per above procedure
2. Service provider validates the request and compares it’s time with the given timestamp.
3. If the difference in the service provider’s time and the client’s provided time is within some tolerance, the request is serviced

This obviously requires the two clocks to be vaguely in sync. It also allows the resource to be requested by anyone within the timespan of the tolerance. But, it does save a second request to the client.

Self-created One-time Token

This is an amalgam of the Timestamp solution and the Negotiated One-time Token solution. Here, the client creates its own token, as a simple integer of increasing value. The server maintains the last requested value and accepts only requests with a higher number:

1. Client creates request, using a global long-lived number
2. Client signs requests and sends it to the service provider
3. Service provider validates the signature and compares the provided numeric token with the one last used (the tokens can be globally scoped, or scoped for a given resource)
4. If the provided numeric token is greater than the previous, the request is serviced
5. The Client increments his numeric token for next time

As with the Timestamp solution, only one request is required. As with the negotiated one-time token solution, the URL can never be used twice. The main issue here is if the client forgets its numeric token. This could be addressed with an additional call to re-establish the token, made only when the Client has determined it no longer knows the last used value.

Unfortunately, this is much more susceptible to race conditions than the Negotiated one-time token. Since the service provider doesn’t know what tokens to expect (only that they should be greater than the last requested one), the client has to ensure that the “create request, submit request, receive response, update local numeric token” cycle is atomic. That is not straightforward.

Update Got another idea from a co-worker

Session Token

When a user access the system that uses the REST API, they get issued a token (via the REST API). This token is just like a session token, with an inactivity timeout and so forth. The token can be manually invalidated via the API, so that when a user logs out or completes some logical task, the token can be invalidated.

This suffers none of the problems of the other solutions, though it isn’t the most secure. However, the security problem it has (using the valid URL before the session times out) is fairly minor, and the tradeoff of getting one request per actual request and no race conditions makes it probably the best way to go.

• 149 cd - Well, duh
• 104 ./run.sh - Occasionally to run JBoss (I have to have two JBoss servers running locally, and when I do a complete recompile/redeploy (not often), I must restart, as ant fstats the ever-loving shit out thousands of files (multiple times, per, 'natch, cause it so superior to make) at the same time JBoss is hot-deploying (twice) and my machine just dies), but more often to run the application I'm developing.
• 88 tail - To read log files (were I on OS X, Console.app would be the way to go)
• 84 ls - Well, duh
• 73 git - I commit a lot, I diff alot, I stat alot, and I blame alot. You should, too
• 57 sqlshell.pl - Awesomely awesome command-line SQL client written by my co-worker and enhanced on occasion by me. I'll probably steal it when I leave this project, and I wish he'd put it up on sourceforge
• 51 jobs - To find out which JBosses to kill (frequently executed in the wrong shell window)
• 44 kill - To then go kill JBoss
• 43 vi - To get some work done

• Commit very rarely
• Get new changes from the repository only when absolutely needed

1. Assign a team lead to integrate the code - this is a senior developers who can assess code quality, provide mentoring and guidance and can be trusted to put code into the repository destined for QA and production
2. Each developer clones the team lead's repository - This is done to baseline the start of their work
3. Developers commit, branch, merge, and pull as necessary - Since Git makes merging simple, developer's can have full use of all features of version control and can do so in their environment without the possibility of polluting the main line of development. They can also share code amongst themselves, as well as get updates from the team lead's repository of "blessed" code¹
4. Developer's inform the lead of completion
5. Lead pulls from their repository - The lead reviews the developer's changes and applies the patch to his repository. He can then exercise whatever quality control mechanisms he wishes, including automated tests, manual tests, reviews, etc².
6. Lead rejects patches he doesn't agree with - If the patch is wrong, buggy, or just not appropriate in some way, the lead rejects the patch and provides the developer with information on the correct approach
7. Lead accepts patches he does agree with - If the lead agrees with the patch, he applies it to his repository, where it is now cleared for QA

             Large maintenance 
+          aggressive schedule 
+       lots of new developers 
+ minimal system documentation
______________________________
 Need for highly efficient and 
       effective QA procedures

Finding bad code

Diff-based code review

• A concurrent version control system developers trust. Recommed Git or Subversion if you must.
• A simple script to email diffs on every commit. Usually included as an example hook for must version control systems.
• IM clients (Google talk within GMail works in even the most oppressive environment)
• A sane version control policy: committed code must:
  • Compile
  • Not prevent application deployment/startup
  • Not horribly break someone else's code (optional)
  Developers should commit as frequently as they want (and preferably frequently). I typically commit code I feel is "done" but that might not add up to an actual feature. This requires accepting that head is not a real verison. Most real version control systems have the ability to tag, branch, etc. These features are for "real working versions". The head of the trunk is not.
• A sane coding style policy: if you must re-indent or change bracing style, do it in its own commit, outside of actual code changes. Better yet, don't do it at all. Formatting changes can obscure the history of a piece of code and should be made minimally, if at all.

1. Diffs get emailed to the senior developers as soon as they happen
2. Senior Developers read the diffs, using IM to discuss any issues
3. If code does have issues, the diff is forwarded to the developer who committed, with comments on what to change and why (senior developers decide amongst themselves who will send the feedback, or a "lead developer" can, if one is identified)

• But imports help the compiler locate classes! - Why should I help the compiler locate classes? Why put a fully automatable task in the hands of a developer? Are you telling me the compiler can't index the classpath to more efficiently search at compile-time?
• But imports let you know class dependencies - No, they don't. Only if you don't use star imports and only if you import only what you need would this be the case. However, not really, because your class could Class.forName. And, honestly, how much time do you spend looking at the import statements to perform this analysis? An automated tool could provide this information much more correctly and completely
• But how would I know what classes are in a jar and what are in the codebase? - You'd know the same way the compiler knows. And, to be honest, the code should be written for the maintainers, not for the new kids. Anyone new to a codebase can, relatively quickly, figure out what is in the code base and what isn't. This, along with proper tools for locating classes integrated into your IDE would be much better than looking at import statements and grep'ing the output of jar tvf.

package myapp;
// no point in putting the dir-structure as dots, the compiler

// can figure it out.  Instead we indicate that this class, whever

// it is, is part of the conceptual package "myapp"


import commons-lang[2.1,];     // check for version 2.1 or greater

import commons-logging[1.0.*]; // check for version 1.0.* only

import j2ee[5.0,5.3];          // check for any version from 5.0 to 5.3


clarify java.util.Date;

public class Whatever
{
    public static void main(String args[]) 
    {
        Date date = new Date();
        // whatever else

    }
}

So, in an effort to bring my parents into the 21st century, I ordered my Mom a new computer. They, unfortunately, use Windows, so a PC it was. I had previously suggested using Dell or Gateway, but it seems that both companies insist on installing massive amounts of bloatware, and neither provides the actual install disks for re-installing your machine. So, I went to a budget computer builder, had them build it, install Windows XP and ship it to me. I then completed the setup, installed patches and all that. Having never used XP, I was pleasantly surprised that it just doesn't suck as much as Windows 2000, but still utterly baffled at the amount of reading and bizarre decision making required of the average user. Plus, I have no idea why the fuck I have to have a puppy dog talking to me when I do a search of the hard drive.

At any rate, I got the computer installed on their "network" without any problems. Their network consists of a dying Windows 2000 box, with the main printer connected to it, a wireless router and the new computer. It was rather inconvienient to move the main printer to connect directly to the XP box, so I used the magical power of networking to connect the printer. In Windows 2000, it's amazingly simple (though not remotely as simple as on OS X). The previous computer pretty much stayed connected to it all the time, even during the myriad of reboots required to keep the computers up and running. I figured with teh awesomz0rz pwnage of XP, it would be even simpler. Right off the bat, you cannot browse to the computer. It just churns and churns and churns, presumably checking every single port of every single possible IP address and asking "Hey! You there! Yes, you, Mr. Port! Do you have any Microsoft products connected to you?".

So, I just go to the computer directly, via the good ole \\COMPUTER_NAME notation. It asks me to login, which I do (why the fuck do I have to log in?!?!!?), indicating I would like the credentials to be remembered, and then I see a list of shared "stuff". The printer is top of the list, right-click, "Connect" and viola, I've now accomplished (hopefully), in six steps what requires one on OS X. But, whatever.

Of course, a few days later, my Mom calls and can't print. Windows has just dropped the connection to the printer. I instruct her to do as above, and, of course, she used forward slashes instead of backslashes (thus searching google for the other computer's name) and, once I'd corrected her, Windows did not remember any part of the login credentials, despite being told to do so. Of course, the printer connected fine and worked. Every 4-5 days, XP just drops the printer, and my Mom has to call me up and we go through this rigamaroll again.

Now, eventually, the Windows 2000 box the printer is connected to will be put out to pasture, and the printer would have to be plugged into the XP box. So, I figure I can save myself some tech support calls if I just connect the printer to the new box and screw the networking crap that Microsoft still can't seem to get right. The printer is about 12 feet away from the computer, so I figure a 15-foot USB cable should be plenty for connecting without having to rearrange anything in my mom's office.

Best Buy's price for a 12 foot cable (the longest they had): $39.99
CompUSA's price for same (again, the longest): $32.99 For a fucking USB cable.

So, to the Internets. I'd heard good things about NewEgg, so I figured this would be a good way to try them out. 15 foot USB cable retails for a measly $3.99. Brief perusal of other online dealers yielded a similar price. What the fuck are the BigBoxes thinking?. I guess that every consumer is a completely uninformed idiot who likes being cheated. Seriously, I can see paying twice as much for the "instant gratification" thing (even though THAT is ridiculous in and of itself), but 10 times as much?!?!?! Wow.

So, the shipping on a $4 USB cable is about $5, so I figure while I'm buying, I'll throw in a few other things I need. Put in a 7-port USB hub for myself, a spindle of CD-Rs for my Mom (who never buys them and therefore never backs up anything ever), and some DVD-R DLs for me (having recently exhausted the majority of my supply backing up the demo for my band), and some CD-Rs for myself as well. All said and done, I've got a good $150 in my shopping cart.

Now, this is ultimately stuff for my Mom, and since I live in the city (and therefore both UPS and FedEx actively hate me), I figured I'll just have it shipped to my Mom's house in Manassas. I create my NewEgg account, enter the shipping address and then am told that if my shipping address and billing address are different:

Contact your card issuer and have the alternate Ship-to Address added as an authorized shipping location in your account records or in the memo field. If you choose to ship to an address other than your billing address that has not been specified as an alternate Ship-to Address with your card-issuer, your order may be delayed by up to several days as we complete verification.

It seems to me that as of 19 fucking 99, the collective online portion of the human race has solved the difficult problem of shipping to an address other than your billing address. I can't even reacll the last time I was on a website where this feature was not seamless. Yet, these assclowns want me to call my credit card company?!?!?!?! From their own FAQ:

Why must Newegg.com verify my shipping address?
For fraud prevention purposes, if your billing and shipping addresses are different, we must verify your shipping address. Please contact the bank that issued your credit card and have your shipping address listed as an alternate address in that bank's memo field. Please make sure your credit card issuer bank's phone number is correctly listed in your Newegg.com account information.

A quick trip to Directron and the same items are now en-route to my Mom's house (I actually saved $10 on the USB hub, to boot). I usually use them for my online computer junk needs, but figured I'd try someone new. I won't be making that mistake again. Right after, I emailed NewEgg informing them that their stupid policy cost them today's sale, and any future sales. I guess if it prevents $150 worth of fraud it was worth it to them.

Update: Received the following message from them regarding my email that they lost a sale:

Thank you for contacting Newegg.

We humbly apologize if this safety precaution has inconvenienced you in anyway but please understand that our intent is only to ensure your satisfaction. <? xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" />