So you have a great website idea and you want to build and bring that first version online as fast as you can. You figured that node.js is the way to go. You kick-off the development and after a couple of hours of hacking you realize that although you’re progressing at breakneck speed you’re missing a few important bits:

• - How do I better structure my project?
• - I want to test this thing. I want unit tests, UI (headless browser) tests and public API tests (I want that API offering out too of course)
• - I want proper CSS and html templating
• - Looks like I need non-trivial request routing, I need more than the default provided

Oh, and after you have all of this, you want to be able to deploy it to a node-ready cloud environment like Heroku without hassle.

Enter bootstrap.js.

It bootstraps the application structure, integrates technologies for templating, routing, etc. and as importantly, comes with the option of pushing the application to the cloud.

Project structure
The project structure, influenced by this suggestion is:

controllers/
views/                <-- view templates
public/               <-- public files
public/stylesheets    <-- stylus css files
spec/                 <-- BDD tests and web service tests
test/                 <-- unit tests

The bootstrapping also creates a git repo for your convenience.

Technologies
Bootstrap.js comes integrated with:

• - Express framework for routing
• - Jade template engine
• - Stylus CSS generator
• - JQuery 1.5.2

A word about testing
Nodejs applications are particularly interesting from a testing perspective given the asynchronous nature of the language itself but I found that the tests are especially valuable when trying-out different technologies since testing is the best way to validate them. Bootstrap.js’s tests are structured to provide the more important aspects of application testing: there are basic unit, BDD and public API tests that are shipped along, giving the developer a starting point to add more tests in those areas as the development progresses. Look in the test/ and spec/ directories for them.

Cloud deployment
Although the bootstrapping can be done locally, the more exciting option is to drop your web application in a hosted environment (and show it off your friends afterwards). There are currently two cloud environments supported: Heroku and Joyent’s no.de. Both have native node support and integrate with your code via git and both making a live service upgrade as easy as “git push”.

Do contribute
I welcome your suggestions/improvements. Get and fork the code for others to enjoy!

		Location Based Services are all the rage these days. The space is still being defined and the players are trying to differentiate their service offerings in order to attract the critical mass of developers. In this post I’ll draw a side-by-side comparison of the main features provided by the major Places API providers today. While I have no hard numbers to back-up the “major provider” claim, I’ll simply go for the web
companies I would look for when building an application around Location services.

Here are my candidates ordered by their first API release date:

The features of all these APIs are designed primarily to support (and promote) the business use cases of each respective competitor. One notable exception is Yahoo’s GeoPlanet API which advertises itself as being a general purpose API for referencing places.

I won’t try to identify any “best” API in the end. This post is meant to allow the reader to make an informed decision on which API(s) to use.

Since there can be only one true collection of physical places (we’re not dealing with quantum realities fortunately) and lacking a common industry effort towards harmonization, the default behavior is to compete to win or dominate the space. One notable mention, Foursquare is now trying to aggregate the various place identifiers referring to the same physical place through Venues Project. Others will follow suit.

A comparison of features available in each API:

Discovering places is an important feature and all competitors offer it:

What kind of data is returned when fetching a place?

*Core Place data is represented differently by each provider, see individual documentation for more details.

What are the access restrictions when using the APIs?

This concludes the comparison galore.

The only offering that felt “general purpose” was Yahoo’s. The API is designed to provide generic access to the place information while not tying to any specific Yahoo data. All of the other offerings are built around specific features provided by the vendor (although the core place information is neutral in each case).

An interesting comparison would be showing the POI coverage for each API provider but this data is unavailable.

I would have liked to add a few more comparisons like number of apps using each API, daily usage. This data is also not public (or at least I couldn’t find it).

Flickr image courtesy mil8

A short while back I read a challenging article titled t’s Time For An Open Database Of Places. There, Erich Schonfeld notes: A long list of companies including Twitter, Google, Foursquare, Gowalla, SimpleGeo, Loopt, and Citysearch are far along in creating separate databases of places mapped to their geo-coordinates.

 These efforts at creating an underlying database of places are duplicative, and any
competitive advantage any single company gets from being more comprehensive than
the rest will be short-lived at best. It is time for an open database of places which all
companies and developers can both contribute to and borrow from.

I agree that there is duplication of effort but this is what happens with many competitive technologies (look at now many online maps are available today). Each company tries to add a competitive advantage to its offering while providing the same core functionality as the competition.
Update: I started this post back in April and a lot of developments recently only enforce: this point. (Check Facebook Places and Google Places for more info).

I like the idea of an open database of places. Any company could build value-added services on top of it and sell them while they are not concerned about issues that come with building and maintaining such database like geo-location/address accuracy and duplicate place resolution to name just a few. Techcrunch’s Schonfeld adds another issue: who can a place and who should be in control of it, suggesting that anybody can update the database and “the best data should prevail”. This is hard and suggests a wiki-like approach for better or worse.
I’m not a fan of centralizing such database. Since there are such great market forces at play, it may become a playground for fights (my data is better than yours), a committee will attempt to regulate it just to push it into oblivion while everybody will get their toys and go build their own database.

I have a different idea (and it’s not new either).

Businesses have a great deal of interest in such database. It puts them on the map. They don’t particularly care who is using their place as long as the data about their business is correct and their customers easily reach their venue. The experience with using a mobile routing software to get to a place in real world is the equivalent of not waiting more than four seconds for a webpage to load. It just has to route the customer precisely to a location.

Why not letting the business to own their own geo data? All it takes is for them to have a website and add a bit of information to it to allow for auto-discovery; it’s called geotagging. It’s the same idea that Matt Griffith had back in 2002 that allows RSS feed autodiscovery applied to geo. The real win is for small businesses that adopt geotagging. All they need to do is add a small bit of metadata on their homepage and let web indexers do the job of collecting this data. Oh, and it’s free.
This brings a double win: companies in the mapping business access accurate geo information about businesses. The business themselves are happy that their customers can precisely find their physical location by means of address and/or geo-coordinates. Moreover, the accuracy of the data is maintained by the businesses since they want their customers to find them even when they move. A Places database that aggregates this type of data can mark these places as “verified” since they come directly from merchants. It even provides more accurate means of building forward and reverse geocoding tools.
Going forward with this model, the competition will shift their efforts from building a database of places to adding value to a (more or less) common Places database like local promotions and building great mapping products to allow us, the customers to find them.

The hard part is promoting this model. If say, half of the brick and mortar small businesses with a web presence embed geo metadata on their website, then the big players take notice. How to get there is the real challenge.

Image via Flickr/bryankennedy

Handling errors in a REST way is seemingly simple enough: upon requesting a resource, when an error occurs, a proper status code and a body that contains a parseable message and using the content-type of the request should be returned.
The default error pages in Tomcat are ugly. Not only they expose too much of the server internals, they are only HTML formatted and making them a poor choice if a RESTful web service is deployed in that Tomcat container. Substituting them to simple static pages is still no enough since I want a dynamic response containing error information.

Here’s how to do it in 3 simple steps:

Step 1
In web.xml

   <servlet-mapping>
      <servlet-name>jsp</servlet-name>
      <url-pattern>/error.jsp<url-pattern>
   </servlet-mapping>
   <!-- other servlet mappings --&&gt&

  <error-page>
    <error-code>404</error-code>
    <location>/error.jsp</location>
  </error-page>
  <error-page>
    <error-code>400</error-code>
    <location>/error.jsp</location>
  </error-page>
...
  <error-page>
    <exception-type>java.lang.Throwable</exception-type>
    <location>/error.jsp</location>
  </error-page>

The servlet called jsp comes out-of the box with the Tomcat container and is the one responsible for processing JSPs . Make sure this mapping comes first, especially if you have a catch-all url-pattern that should map all other URLs to your webapp.

Step 2
Create a file called errors.jsp in the root of your web application:

<%@ page contentType="application/json" pageEncoding="UTF-8"%>
{
   status:<%=request.getAttribute("javax.servlet.error.status_code") %>,
   reason:<%=request.getAttribute("javax.servlet.error.message") %>
}

Step 3
Hook the exception handling using @ExceptionHandler:

@Controller
public class SomeController {
...

@ExceptionHandler(BusinessException.class, ValidationException.class, ...)
public void handleApplicationExceptions(Throwable exception, HttpServletResponse response) {
   String formattedErrorForFrontEnd = formatErrMessage(exception);
   if(exception instanceof BusinessException) {
      response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
formattedErrorForFrontEnd);
   } else if (exception instanceof ValidationException) {
      response.sendError(HttpServletResponse.SC_BAD_REQUEST, formattedErrorForFrontEnd);
   }
   ...
}

//other @ExceptionHandlers skipped

   private String formatErrMessage(Throwable exception) {
      //skipped
   }
}

I used @ExceptionHandler to map between application exceptions and error codes. You can apply message formatting as well if the default messages are not sufficient. Also, consider having an @ExceptionHandler(Throwable.class) for the “all other exceptions” case.

As you can see, there is no need to write another Spring Controller that handles exceptional paths. The error-page mappings in web.xml, a 2 line JSP and the standard @ExceptionHandler combination was sufficient.

Fetching, aggregating and transforming data for delivery is a seemingly complex task. Imagine a service that serves aggregated search results from Twitter, Google and Bing where the response has to be tailored for mobile and web. One has to fetch data from different sources, parse and compose the results then transform them into the right markup for delivery to a specific client platform.
To cook this I’ll need:
- a web server
- a nice way to aggregate web service responses (pipelining would be nice)
- a component to transform the raw aggregated representation into a tailored client response.

I could take a stab at it and use Apache/Tomcat, Java (using Apache HttpClient 4.0), a servlet dispatcher (Spring WebMVC) and Velocity templating but it sounds too complex.

Enter Node.js. It’s an event-based web server built on Google’s V8 engine. It’s fast and it’s scalable and you develop on it using the familiar Javascript.
While Nodejs is still new, the community has built a rich ecosystem of extensions (modules) that greatly ease the pain of using it. If you’re unfamiliar with the technology, check-out the Hello World example, it should get you started.
Back to the task at hand, here are the modules I’ll need:
- Restler to get me data.
- async to allow parallelizing requests for effective data fetching.
- Haml-js for view generation

To restate the requirement, we want to aggregate results from Google and Twitter using any arbitrary search predicate.
First we need node.js itself:

git clone http://github.com/ry/node/tree/master
./configure
make
sudo make install

Let’s bring the necessary modules. Create a project directory then fetch them in it:

git clone  http://github.com/caolan/async.git
git clone http://github.com/danwrong/restler.git
git clone http://github.com/creationix/haml-js.git

Most of the action take place in one file that fires the webserver and contains all the event logic. My small little server is called servus.js so I’ll call the file just that.

Update Jun/28: Servus now lives in github, grab it here

First thing, import all the modules, assuming that all of them are in the root of the project directory and create the basic port listener (all code in servus.js):

var sys = require('sys'),
  http = require('http'),
  rest = require('./restler/lib/restler'),
  async = require('./async/lib/async'),
  haml = require('./haml-js/lib/haml');
  url = require('url'),
  fs = require('fs');

http.createServer(function (req, res) {
    process.addListener('uncaughtException', function (err) {
      sys.puts('Caught exception: ' + err);
      res.writeHead(500, 'text/plain');
      res.end('error!');
    });

   res.writeHead(200, {'Content-Type': 'text/plain'});
   res.end('Servus\n');
}).listen(8124, "127.0.0.1");
sys.puts('Server running at http://127.0.0.1:8124/');

Listening for ‘uncaughtException’ events ensures that the server keeps running when an exception occurs. Run this with node servus.js and you should see the server responding with ‘Servus’ when browsing to http://localhost.

Let’s add some real code. First, we define the request URLs for Twitter and Google. Next, parse the URL and extract the query predicate. Servus will respond to /aggregate?q= requests and querying both search services using async. Replace res.writeHead and res.end lines with:

    var serviceUrls = {};
    serviceUrls.twit = 'http://search.twitter.com/search.json?q=';
    serviceUrls.goo = 'http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=';
    var urlParsed = url.parse(req.url, true);
    var path = urlParsed.pathname;

    switch(path) {
      case '/aggregate':
        //extract search predicate
        var query = urlParsed.query;
        if(query != undefined) {
          predicate = query.q;
        } else {
          res.writeHead(400);
          res.end('Query expected. Use q=... in the URL');
          return;
        }
        async.parallel([
          function(callback) {
            callRestService(serviceUrls['goo'] + predicate, 'goo', callback);
          },
          function(callback) {
            callRestService(serviceUrls['twit'] + predicate, 'twit', callback);
          }
        ],
        //callback
        function (err, results) {
             //use the results argument and a haml template to form the view.
        }
      );
      break;

Through the async module, both GET requests are fetched in parallel. The callback function is called after all functions in the first argument async.parallel have been completed; the ‘results’ argument in the callback function accumulates the search results in an array. The function ‘callRestService’ gets the JSON results and parses them in hashes of format {url: …, text: …}. Let’s implement it:

    function callRestService(url, serviceName, callback) {
        request = rest.get(url);
        request.addListener('success', function(data) {
            searchResults = [];
            if(serviceName == 'goo') {
              dataJson = JSON.parse(data).responseData.results;
              for(sr in dataJson) {
                searchResult = {}
                searchResult.url = dataJson[sr].url;
                searchResult.text = dataJson[sr].title;
                searchResults.push(searchResult);
              }
            } else if(serviceName == 'twit') {
              dataJson = data.results;
              for(sr in dataJson) {
                searchResult = {}
                searchResult.url = 'http://twitter.com/' + dataJson[sr].from_user +
 '/status/' + dataJson[sr].id;
                searchResult.text = dataJson[sr].text;
                searchResults.push(searchResult);
              }
            }
          callback(null, searchResults);
        });
        request.addListener('error', function(data) {
          sys.puts('Error fetching [' + url + ']. Body:\n' + data);
          callback(null, ' ');
        });
    }

The callback function in async.parallel takes the search results and will a use haml template to produce the view for web serving. Here’s the template (search-res.haml):

%h1 Search results
  :if items.length === 0
    %p There are no search results
  :if items.length > 0
    %table
      :each sr in items
        %tr
          %td %a{href: sr.url}&= sr.text

We’ll load this template and use it subsequently in the callback function:

    //load template; this goes before http.createServer
    var searchResHamlTemplate;
    fs.readFile('./search-res.haml', function(e, c) {
        searchResHamlTemplate = c.toString();
      });

The callback function above is implemented like this:

        //callback
        function (err, results) {
          //results accumulated two arrays of search results,
          //for google and twitter respectively.

          res.writeHead(200, {'Content-Type': 'text/html'});
          var searchItems = [];
          for(i=0;i<results.length;i++) {
            for(sr in results[i]) {
              searchItems.push(results[i][sr]);
            }
          }
          res.end(haml.render(searchResHamlTemplate, {locals: {items: searchItems}}));
        }

Done. You can grab the complete code here: servus.js. The final version contains a few more bits to be able to properly respond to GET queries. Querying the server using http://localhost:8124/aggregate?q=worldcup will render a few google results on top followed by all twitter search results.

Adding a Service Level Agreement guarantee
What if we want to add SLA (service level agreement) to our little server? Some web service clients require that servers respond within a guaranteed time. Turns out it’s not hard at all. We only have to add one more function in the parallel.async call that is called after the preset timeout using setTimeout:

    var timeoutReached = false;
...
        async.parallel([
          function(callback) {
            callRestService(serviceUrls['goo'] + predicate, 'goo', callback);
          },
          function(callback) {
            callRestService(serviceUrls['twit'] + predicate, 'twit', callback);
          },
          function(callback) {
            setTimeout(function() {
              if(!timeoutReached) {
                //serve search results if any
                serveContent(partialResults);
                sys.puts("Timeout reached");
              }
            }, 200);
          }
        ]);

partialResults is an array where search results are accumulated as they come in. If this array size becomes 2 then we know that the fetch/parse has completed and the result can be served to the view:

    function callRestService(url, serviceName, callback) {
        request = rest.get(url);
        request.addListener('success', function(data) {

...same as before...

           partialResults.push(searchResults);
           if(partialResults.length ==  2 && !timeoutReached) {
             sys.puts("Timeout not reached");
             serveContent(partialResults);
           }
        });

    function serveContent(results) {

...same as before...

      timeoutReached = true;  //prevent further search results to be processed
    }

Lastly, timeoutReached flag is turned off in serveContent to prevent further results to be processed.

I’ve attached the final version of servus.js with sla: servus-sla.js. The code is available now in a github project called servus. The sla timeout is specified as an argument when starting nodejs. If none is specified, the default is 2 seconds.

If you’re behind a corporate proxy, use proxychains. You can find more details about this issue here

Update Mar.04 Thanks to @ewolff some of the points described below are now official feature requests. One (SPR-6928) is actually scheduled in Spring 3.1 (cool!). I’ve updated the post and added all open tickets. Please vote!

This post is somewhat a response to InfoQ’s Comparison of Spring MVC and JAX-RS.
Recently I have completed a migration from a JAX-RS implementation of a web service to Spring 3.0 MVC annotation-based @Controllers. The aforementioned post on InfoQ was published a few days after my migration so I’m dumping below the list of problems I had, along with solutions.

Full list of issues:

• Same relative paths in multiple controllers not supported
• @ExceptionHandler is controller-centric
• Standard content negotiation can’t respond with a fixed response type
• JSR 303 bean validation not applied in @Controllers
• Formatting responses (i.e. Date) not working when using Spring formatter annotations

Same relative paths in multiple @Controllers not supported
Consider two Controllers where I use a versioned URL and a web.xml file that uses two URL mappings:

@Controller
public class AdminController {
   @RequestMapping("/v1/{userId}")
   public SomeResponse showUserDetails(String userId) {
      ...
   }
}

@Controller
public class UserController {
   @RequestMapping("/v1/{userId}")
   public SomeOtherResponse showUserStreamtring userId) {
      ...
   }
}
In web.xml:
	<servlet-mapping>
	  <servlet-name>public-api</servlet-name>
	  <url-pattern>/public</url-pattern>
	</servlet-mapping>
	<servlet-mapping>
	  <servlet-name>admin-api</servlet-name>
	  <url-pattern>/admin</url-pattern>
	</servlet-mapping>

Here I want to implement handling for /admin/v1/{userId} as well as /user/v1/{userId}. Each controller serves a different purpose (admin and public) and I want to keep the servlet application contexts separate for each servlet mapping (for instance add authentication and different view resolvers to all /admin/* URLs).
Deploying this configuration barfs the following exception:

java.lang.IllegalStateException: Cannot map handler [UserController] to URL path
[/v1/{userId}]: There is already handler [com.test.controller. UserController@6b177115]
mapped.

I would like Spring MVC to account for web.xml servlet mappings before deciding that two identical @RequestMappings in different controllers refer to the same URI. My solution was to have all @RequestMappings specifying the full path and resorting to a custom view to work around this issue).

@ExceptionHandler is controller-centric
RESTful web services respond with the usual HTTP responses (200s, 400s, 500s, etc.). These responses are being mapped from application exceptions and it only makes sense to specify the same @ExceptionHandler for all @Controllers since most (if not all) exceptions handled by them are for the same business domain. Currently SpringMVC only supports declaring and @ExceptionHander per controller and not globally per webapp. This forces the implementors to extend their @Controller classes from a common one where the @ExceptionHandler is specified or, to use composition and delegate exception handling to one service which leads to duplication. Both are ugly. There are several ways to deal with exceptions (check SPR-5622; please guys, add this to the main Spring documentation) but ideally I’d like to see a global scoped handler like @ExceptionHandler(scope=APPLICATION) where I can remap once framework exceptions to more suitable ones that translate into proper error codes (like working around SPR-5622 like issues).

Standard content negotiation can’t respond with a fixed response type (SPR-6937)
Here’s a simple requirement: Implement an RSS feed that serves a user’s activity stream.
Here’s another requirement (make that best practice):

Service methods should have as arguments primitives, wrappers or other POJOs and
 respond with void, primitives, wrappers or other POJOs.

I want testability, readability and portability for service classes and using framework objects for arguments/return objects adds unacceptable clutter.
As such, no framework objects are allowed as arguments or return types in a @Controller and in this case I don’t want to work with ModelAndViews just to be able to select the desired View.
Since this new requirement (add RSS view) belongs to the user domain, I want to add a method in the UserController class that returns a POJO which should be always unmarshalled into RSS regardless of request content type (there goes ContentNegotiationViewResolver). The same controller has other methods returning POJO view objects that I want marshalled into JSON or plain text response body and an HTTP response status. This is already available in RESTEasy via the elegant Produces annotation. The same RESTful controller should respond with different response types for different requests since I want to keep its domain centered around the user (and not response types).

...
public class UserController {
...
   @RequestHandler("/rss/{userId}")
   public SomePOJO getRssFeed(String userId) {
      ...
      //returns a POJO that should always be resolved by an RSS view
   }

   @RequestMapping("/activity/{userId}")
   public SomeOtherPOJO showUserStream(String userId) {
      ...
      //return POJO that should always be resolved by a JSON view
   }
...
}

It turns out this is not immediately possible. To my knowledge none of the included view resolvers provide a straight mapping between URLs and views so I had to write one that configures like this:

<bean class="UrlToViewViewResolver">
   <map name="urlToViewMap">
      <property key="/user/rss/*" value="rssView"/>
      <property key="/user/activity/*" value="jsonView"/>
      ...
   </map>
</bean>

The implementation (not included) is trivial and based on AntPathMatcher, a utility class readily available in Spring.

JSR 303 bean validation not applied in @Controllers (SPR-6928, scheduled for 3.1)
On Springsource’s weblog there’s a nice tutorial on how to get validation and type conversion in Spring 3.0 here. When it comes to validation though, Spring only supports it “after binding user input to it”. It is not possible to validate arguments and return types for a RESTful controller (no web forms or user input of any kind there). This means this won’t work out of the box:

   @RequestHandler("/admin/", method = RequestMethod.PUT )
   public void updateUser(@Valid User user) {
      ...
      //returns a POJO that should always be resolved by an RSS view
   }

since in a RESTful scenario the User object won’t be bound from a form but unmarshalled from XML/JSON/CSV or any other wire format.
To enable validation for a RESTful controller I chose to use @Valid at the method level and add an around advice that intercepts all methods annotated as such then use a Validator to validate arguments and return objects.

<aop:config proxy-target-class="true">
	<aop:pointcut id="validatableMethodExecution"
        expression="execution(* com.mycorp..*.*(..)) and @annotation(javax.validation.Valid)"/>

	<aop:aspect id="validateArgumentsAndReturnObject" ref="validatorAspect">
		<aop:around pointcut-ref="validatableMethodExecution" method="validate"/>
	</aop:aspect>
</aop:config>

public class ModelValidatorAspect {
	@Autowired
	private Validator validator;

    public Object validate(ProceedingJoinPoint pjp) throws Throwable {
        Set<ConstraintViolation> violations =new HashSet<ConstraintViolation>();
        for(Object obj : pjp.getArgs()) {
            if(obj != null) {
                violations.addAll(validator.validate(obj));
            }
        }
        if(!violations.isEmpty()) {
            throw new BadRequestException(composeErrorMessage(violations));
        }

        Object ret = pjp.proceed();

        //validate return object and throw exception if the validation fails.
        if(ret != null) {
            violations = validator.validate(ret);
        }
        if(!violations.isEmpty()) {
            throw new BusinessValidationException(violationsStr);
        }
   }
}

This could get more complex if you have deal with collections of validatable objects but it’s not hard to enhance the code to handle that.

Formatting responses (i.e. Date) not working using Spring formatter annotations
Spring comes with some niceties like @DateTimeFormat and @NumberFormat and naturally I want to rely on them by annotating the class attributes of return types used by @Controllers.
That’s not quite working since the formatters work with form-backed model objects. However, if you use a MappingJacksonJsonView there is a way of formatting using Jackson’s own @JsonSerialize.

Although at times it felt that the switch to SpringMVC was too much, the features that come with it (like abstracting web-related properties as annotations) will turn your RESTful controllers into annotated POJOs focused on the business domain which ultimately means maintainable code.

Update Mar.04 I left out several points about JSON marshalling that didn’t work as expected (Jackson marshaller is a handful).
- One enhancement I suggested to ease the pain of configuring JSON marshalling is creating a namespace similar to oxm (SPR-6943)
- The second was to add Jetisson support (SPR-6938) as an alternative to Jackson JSON marshaller.

I’m keen on your feedback regarding these issues so if you have better solutions please comment.

I want to write testable code and occasionally I bump into frameworks that make it challenging to unit test. Ideally I want to inject a service stub into my code then control the stub’s behavior based on my testing needs.
Commons Http Client from Jakarta facilitates integration with HTTP services but how to easily unit test code that depends on the HttpClient library? Turns out it’s not that hard.
I’ll cover both 1.3 and the newer 1.4 versions of the library since the older v1.3 is still widely used.
Here’s some typical service (HttpClient v1.3) we want to test. It returns the remote HTML page title:

public class RemoteHttpService {
   private HttpClient client;

   public String getPageTitle(String uri)  throws IOException {
     String contentHtml = fetchContent(uri);
     Pattern p = Pattern.compile("<title>(.*)</title>");
     Matcher m = p.matcher(contentHtml);
     if(m.find()) {
        return m.group(1);
     }
     return null;
   }

   private String fetchContent(String uri)  throws IOException {
      HttpMethod method = new GetMethod("http://blog.newsplore.com/" + uri);
      int responseStatus = client.executeMethod(method);
      if(responseStatus != 200) {
        throw new IllegalStateException("Expected HTTP response status 200 " +
"but instead got [" + responseStatus + "]");
      }
      byte[] responseBody = method.getResponseBody();
      return new String(responseBody, "UTF-8");
   }

   public void setHttpClient(HttpClient client) {
      this.client = client;
   }
}

with the HttpClient is injected at runtime (via some IoC container or explicitly).
To be able to unit-test this code we have to come-up with a stubbed version of the HttpClient and emulate the GET method.

What we really want is to be able to control the byte array returned by method.getResponseBody() and although we can inject a mocked HttpClient, the actual class that is responsible for producing the response body is GetMethod which is instantiated (and completely encapsulated) in the method body so we can’t inject a stubbed GetMethod version directly.
The solution is to create a simple HttpClient mock that is able to control the response status and response body. Here it is:

package org.apache.commons.httpclient;

import org.apache.commons.httpclient.*;

public class HttpClientMock extends HttpClient {
   private int expectedResponseStatus;
   private String expectedResponseBody;
   public HttpClientMock (int responseStatus, String responseBody) {
      this.expectedResponseStatus = responseStatus;
      this.expectedResponseBody = responseBody;
   }
   @Override
   public int executeMethod(HttpMethod method) {
      ((HttpMethodBase)method).setResponseStream(
new ByteArrayInputStream(expectedResponseBody.getBytes("UTF-8")));
      return expectedResponseStatus;
   }
}

There are two things to notice: Firstly, I placed HttpClientMock in the org.apache.commons.httpclient package to be able to have access to the protected member HttpMethodBase:setResponseStream. Secondly, I downcasted HttpMethod to HttpMethodBase to get access to setResponseStream (it’s safe since all concrete HttpMethods (GetMethod, PostMethod, etc) inherit from it).

Now I can unit test RemoteHttpService like this:

public class RemoteHttpServiceTest {
   private RemoteHttpService remoteHttpService = new RemoteHttpService();

   @Test
   public void extractTitleResponseOK()  throws IOException {
      String pageTitle = "A Title";
      String responseBody = "<title>" + pageTitle + "</title>";
      HttpClientMock mockHttpClient = new HttpClientMock(200, responseBody);
      remoteHttpService.setHttpClient(mockHttpClient);
      String resultedPageTitle = remoteHttpService.getPageTitle(""); //home page
      Assert.assertTrue(pageTitle.equals(resultedPageTitle));
   }
}

I can also add a negative test:

   @Test(expected = IllegalStateException.class)
   public void respondsWithInternalServerError()  throws IOException {
      HttpClientMock mockHttpClient = new HttpClientMock(500, "");
      remoteHttpService.setHttpClient(mockHttpClient);
      remoteHttpService.getPageTitle("");
   }

For HttpClient version 4.x here’s the refactored service code :

   private String fetchContent(String uri) {
        HttpGet httpget = new HttpGet("http://blog.newsplore.com/" + uri);
        HttpResponse response = client.execute(httpget);
        int responseStatus = response.getStatusLine().getStatusCode();
        if(responseStatus != 200) {
           throw new IllegalStateException("Expected HTTP response status 200 " +
"but instead got [" + responseStatus + "]");
        }
        InputStream responseStream = response.getEntity().getContent();
        byte[] responseBytes = IOUtils.toByteArray(responseStream);
        return new String(responseBytes, "UTF-8");
   }

Turns out that implementing a HttpClient mock by extending the DefaultHttpClient (the provided implementation of HttpClient) is not easy. The method I want to override is public HttpResponse execute(HttpUriRequest request) which is marked ‘final’ and the code in DefaultHttpClient looks just too complex to inject stubbed behavior.
What we actually want is to be able to control the return of the aforementioned ‘execute’ method since it encapsulates the response status and body (better encapsulation than HttpClient 1.3 if you noticed).
Luckily in this case we can use a readily available mocking framework, Mockito that can stub this entire method and return an HttpResponse that we instantiate with the needed status and response body. Here’s how to rewrite the tests using it:

@Test
public void extractTitleResponseOK() throws IOException {
	String pageTitle = "A Title";
	String responseBody = "";
	HttpResponse response = prepareResponse(200, responseBody);
	HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
	Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
			.thenReturn(response);

	remoteHttpService.setHttpClient(mockHttpClient);
	String resultedPageTitle = remoteHttpService.getPageTitle(""); // home page
	Assert.assertTrue(pageTitle.equals(resultedPageTitle));
}

@Test(expected = IllegalStateException.class)
public void respondsWithInternalServerError() throws IOException {
	HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
	HttpResponse response = prepareResponse(500, "");
	Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
			.thenReturn(response);
	remoteHttpService.setHttpClient(mockHttpClient);
	remoteHttpService.getPageTitle("");
}

private HttpResponse prepareResponse(int expectedResponseStatus,
               String expectedResponseBody) {
      HttpResponse response = new BasicHttpResponse(new BasicStatusLine(
    		  new ProtocolVersion("HTTP", 1, 1), expectedResponseStatus, ""));
      response.setStatusCode(expectedResponseStatus);
      try {
		response.setEntity(new StringEntity(expectedResponseBody));
	} catch (UnsupportedEncodingException e) {
		throw new IllegalArgumentException(e);
	}
	return response;
}

The meat of the above code is

HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
HttpResponse response = prepareResponse(...);
Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
	.thenReturn(response);

that instructs the mocked HttpClient to return a HttpResponse that is instantiated in the test context (seeded if you like) when ‘execute’ is called (check Mockito documentation for more details on the syntax).

That’s it. Happy testing.

In my constant search for free weather data for Spincloud, a short while ago I have found a gem: free forecast data offered by the progressive Norwegian Meteorologic Institute. The long range forecast coverage is fairly thorough and covers most more than 2700 locations worldwide. I am happy to announce that I have extended spincloud.com to include it.

The data is refreshed every hour and the forecast range is available for the next seven days. You can bookmark any location or subscribe to weather reports via RSS.
On a different but related note, I can only be thankful for the free data offered by various meteorological organizations that allows Spincloud to exist and I believe in freeing public data (weather related and otherwise) as it belongs to the public that finances government and inter-government agencies in the first place. The Norwegian Met Institute is a great example for freeing its data and I am saluting them for making the right decision.
Now if only all such progressive Meteorological institutes around the world would agree on a common format for disseminating their data, it would make developers like me a tad happier…

I admit that I regard automation as a dull but vital part in the success of a project. Automation had evolved into Continuous Integration, a powerful toolset allowing frequent and regular building and testing of the code. I won’t get into what CI is (check the internets). Instead, I am going to explore a couple of aspects of CI that can be added to the artifacts of the development process and note some others that cannot.

Continuous performance
You wrote performance tests. You can run performance tests by firing a battery of tests from a client machine and targeted on an arbitrary environment where the application lives. The test results are collected on the client and you can publish them on a web server. Why not automating this completely then? To accomplish this, automate the execution, gathering and publishing of the performance results. Daily performance indicators not only increase visibility of the progress of the application but it becomes much easier to fix a performance degradation on a daily changeset than between two releases. There are a couple of factors that may add complexity to establishing performance tests:
Dealing with dependencies
The obvious rule of thumb is to minimize dependencies. However, if there still are dependencies on other (perhaps external) systems, use mocks and to isolate the system you’re testing for performance. We’re talking about nightly performance tests so don’t put unnecessary stress where you shouldn’t.
Finally, the main artifact of the final integration (done once per iteration) is a running environment where all components run together, in a production-like setup. Use this environment to run your system performance test where you measure the current performance against the baseline.
Measuring relative performance
The environment you’re using for the nightly PT cycle most likely will not be a perfect mirror of production (especially true when dealing with geographically distributed systems). Use common-sense to establish the ratio between the two environments then derive rough production performance numbers using it (assuming a linear CPU/Throughput relationship).

Continuous deployment
This is as simple as it sounds: automate the install. Make it dead easy to deploy the application in any environment by providing installation scripts. Simplify the configuration down to a single file that is self-documented and easily understood by non-programmers (read Operations Teams). The goal here is to unlock a powerful tool: making the application installable and upgradable with a click of a button. If all the other pieces of the continuum are in place then you could confidently deploy your application in production it on a much tighter release cycle, even on a daily basis. Deployment and integration become tasks in the background rather than first-class events.

More continuous
Since I am just fresh off the Agile Testing Days conference and I have learned a few more Cs from the distinguished speakers which I term as Soft Cs since they involve constant human engagement:
- Continuous learning (Declan Whelan)
- Continuous process improvement (Stuart Reid)
- Continuous acceptance testing (i.e. stakeholder signoff at the end of every sprint) – Anko Tijman
- Continuous customer involvement (Anko Tijman)

Sep. 15 came and went and I didn’t realize that this blog is one year old. I didn’t post in the last few months but this is because I have been busy moving across the world from Toronto back to Europe (I’m in Berlin now), becoming a father (“the best job in the world”) and taking a new job (LBS, yes!).
Things are happening, wheels are in motion and there’s still a lot to write about.

Stay tuned!