Spy on external HTTP API requests in Rails

November 16th 2009

A.P. AYE Ever had to use a third party API through a gem or library in your Rails app and wondered where the requests are going to?

Through the magic of Net HTTP Spy (a little gem I wrote) you can spy on the majority of HTTP traffic to external API’s taking place in your Rails application.

Installation

sudo gem install martinbtt-net-http-spy

Real World Example using Google Data API’s

You can now get access to pretty much any Google application/content through their comprehensive set of API’s – very handy indeed.

In this example we are going to use the DocList manager sample application as seen on the GData on Rails tutorial pages.

First grab the source:

svn co http://gdata-samples.googlecode.com/svn/trunk/doclist/DocListManager

Then create a new file initializers/api_logging.rb and add the following:

if ENV.has_key?('API\_LOGGING')
  gem 'martinbtt-net-http-spy'
  require 'net-http-spy'
  Net::HTTP.http_logger = Rails.logger
  Net::HTTP.http\_logger\_options = {:body => true} if ENV['API\_LOGGING\_FULL']
end

Sample Usage and Output:

API_LOGGING=true ./script/server
Processing DoclistController#documents (for 127.0.0.1 at 2009-11-14 20:24:47) [POST]
  Parameters: {"authenticity_token"=>"76e735433a9459f28a2dfb23090dec203b73e65c"}
CONNECT: ["docs.google.com", 80]
GET /feeds/default/private/full/-/document/mine
PARAMS {} 
BODY: Net::HTTPOK
Rendered doclist/_documents_list (7.4ms)
Completed in 1672ms (View: 11 | 200 OK [http://localhost/]

Processing DoclistController#show (for 127.0.0.1 at 2009-11-14 20:24:49) [POST]
  Parameters: {"url"=>"http://docs.google.com/feeds/default/private/full/document:0BUjxlHAOrFICZGh1aGdrZ18xM2ZudGt3M2Zz", "authenticity_token"=>"76e735434a9459f08a2dfb23090dec203b73e65c"}
CONNECT: ["docs.google.com", 80]
GET /feeds/default/private/expandAcl/document:0BUjxlHAOrFICZGh1aGdrZ18xM2ZudGt3M2Zz
PARAMS {} 
BODY: Net::HTTPOK
CONNECT: ["docs.google.com", 80]
GET /feeds/download/documents/Export?docID=0BUjxlHAOrFICZGh1aGdrZ18xM2ZudGt3M2Zz&exportFormat=png
PARAMS {} 
BODY: Net::HTTPOK
Rendered doclist/_show (7.7ms)
Completed in 2515ms (View: 15 | 200 OK [http://localhost/doclist/show]

Note 1: By setting API_LOGGING=full the BODY: portion of the logging output will be replaced with the actual body response from the server. In this case it is the XML used to display the documents.

Note 2: Not all API libs are built the same and many interact with Net::HTTP in ways not easy to capture. Also be warned.. this modifies the Net::HTTP class directly so be careful if you intend using this in production.

Summary

Net HTTP spy provides a great way to peek at what is going on under the hood of your application and find out which http requests are being made to third parties. I’ve used this technique successfully with my team at work to track down API issues and I’ve also had numerous reports of other developers making use of the gem for the same reasons.

I hope you find this post helpful and if you have any further questions or feedback feel free to ask in the comments section.

Quick Reference

HTTP Spy on GitHub: http://github.com/martinbtt/net-http-spy
GData on Rails tutorials: http://code.google.com/apis/gdata/articles/gdata_on_rails.html
Google API Directory: http://code.google.com/apis/gdata/docs/directory.html