Unit testing with Commons HttpClient library

I want to write testable code and occasionally I bump into frameworks that make it challenging to unit test. Ideally I want to inject a service stub into my code then control the stub’s behavior based on my testing needs.
Commons Http Client from Jakarta facilitates integration with HTTP services but how to easily unit test code that depends on the HttpClient library? Turns out it’s not that hard.
I’ll cover both 1.3 and the newer 1.4 versions of the library since the older v1.3 is still widely used.
Here’s some typical service (HttpClient v1.3) we want to test. It returns the remote HTML page title:

public class RemoteHttpService {
   private HttpClient client;
   
   public String getPageTitle(String uri)  throws IOException {
     String contentHtml = fetchContent(uri);
     Pattern p = Pattern.compile("<title>(.*)</title>");
     Matcher m = p.matcher(contentHtml);
     if(m.find()) {
        return m.group(1);
     }
     return null;
   }

   private String fetchContent(String uri)  throws IOException {
      HttpMethod method = new GetMethod("http://blog.newsplore.com/" + uri);
      int responseStatus = client.executeMethod(method);
      if(responseStatus != 200) {
        throw new IllegalStateException("Expected HTTP response status 200 " +
"but instead got [" + responseStatus + "]");
      }
      byte[] responseBody = method.getResponseBody();
      return new String(responseBody, "UTF-8");
   }

   public void setHttpClient(HttpClient client) {
      this.client = client;
   }
}

with the HttpClient is injected at runtime (via some IoC container or explicitly).
To be able to unit-test this code we have to come-up with a stubbed version of the HttpClient and emulate the GET method.

What we really want is to be able to control the byte array returned by method.getResponseBody() and although we can inject a mocked HttpClient, the actual class that is responsible for producing the response body is GetMethod which is instantiated (and completely encapsulated) in the method body so we can’t inject a stubbed GetMethod version directly.
The solution is to create a simple HttpClient mock that is able to control the response status and response body. Here it is:

package org.apache.commons.httpclient;

import org.apache.commons.httpclient.*;

public class HttpClientMock extends HttpClient {
   private int expectedResponseStatus;
   private String expectedResponseBody;
   public HttpClientMock (int responseStatus, String responseBody) {
      this.expectedResponseStatus = responseStatus;
      this.expectedResponseBody = responseBody;
   }
   @Override
   public int executeMethod(HttpMethod method) {
      ((HttpMethodBase)method).setResponseStream(
new ByteArrayInputStream(expectedResponseBody.getBytes("UTF-8")));
      return expectedResponseStatus;
   }
}

There are two things to notice: Firstly, I placed HttpClientMock in the org.apache.commons.httpclient package to be able to have access to the protected member HttpMethodBase:setResponseStream. Secondly, I downcasted HttpMethod to HttpMethodBase to get access to setResponseStream (it’s safe since all concrete HttpMethods (GetMethod, PostMethod, etc) inherit from it).

Now I can unit test RemoteHttpService like this:

public class RemoteHttpServiceTest {
   private RemoteHttpService remoteHttpService = new RemoteHttpService();

   @Test
   public void extractTitleResponseOK()  throws IOException {
      String pageTitle = "A Title";
      String responseBody = "<title>" + pageTitle + "</title>";
      HttpClientMock mockHttpClient = new HttpClientMock(200, responseBody);
      remoteHttpService.setHttpClient(mockHttpClient);
      String resultedPageTitle = remoteHttpService.getPageTitle(""); //home page
      Assert.assertTrue(pageTitle.equals(resultedPageTitle));
   }
}

I can also add a negative test:

   @Test(expected = IllegalStateException.class)
   public void respondsWithInternalServerError()  throws IOException {
      HttpClientMock mockHttpClient = new HttpClientMock(500, "");
      remoteHttpService.setHttpClient(mockHttpClient);
      remoteHttpService.getPageTitle("");
   }

For HttpClient version 4.x here’s the refactored service code :

   private String fetchContent(String uri) {
        HttpGet httpget = new HttpGet("http://blog.newsplore.com/" + uri); 
        HttpResponse response = client.execute(httpget);
        int responseStatus = response.getStatusLine().getStatusCode();
        if(responseStatus != 200) {
           throw new IllegalStateException("Expected HTTP response status 200 " +
"but instead got [" + responseStatus + "]");
        }
        InputStream responseStream = response.getEntity().getContent();
        byte[] responseBytes = IOUtils.toByteArray(responseStream);
        return new String(responseBytes, "UTF-8");
   }

Turns out that implementing a HttpClient mock by extending the DefaultHttpClient (the provided implementation of HttpClient) is not easy. The method I want to override is public HttpResponse execute(HttpUriRequest request) which is marked ‘final’ and the code in DefaultHttpClient looks just too complex to inject stubbed behavior.
What we actually want is to be able to control the return of the aforementioned ‘execute’ method since it encapsulates the response status and body (better encapsulation than HttpClient 1.3 if you noticed).
Luckily in this case we can use a readily available mocking framework, Mockito that can stub this entire method and return an HttpResponse that we instantiate with the needed status and response body. Here’s how to rewrite the tests using it:

@Test
public void extractTitleResponseOK() throws IOException {
	String pageTitle = "A Title";
	String responseBody = "";
	HttpResponse response = prepareResponse(200, responseBody);
	HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
	Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
			.thenReturn(response);

	remoteHttpService.setHttpClient(mockHttpClient);
	String resultedPageTitle = remoteHttpService.getPageTitle(""); // home page
	Assert.assertTrue(pageTitle.equals(resultedPageTitle));
}

@Test(expected = IllegalStateException.class)
public void respondsWithInternalServerError() throws IOException {
	HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
	HttpResponse response = prepareResponse(500, "");
	Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
			.thenReturn(response);
	remoteHttpService.setHttpClient(mockHttpClient);
	remoteHttpService.getPageTitle("");
}

private HttpResponse prepareResponse(int expectedResponseStatus, 
               String expectedResponseBody) {
      HttpResponse response = new BasicHttpResponse(new BasicStatusLine(
    		  new ProtocolVersion("HTTP", 1, 1), expectedResponseStatus, ""));
      response.setStatusCode(expectedResponseStatus);
      try {
		response.setEntity(new StringEntity(expectedResponseBody));
	} catch (UnsupportedEncodingException e) {
		throw new IllegalArgumentException(e);
	}
	return response;
}

The meat of the above code is

HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
HttpResponse response = prepareResponse(...);
Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
	.thenReturn(response);

that instructs the mocked HttpClient to return a HttpResponse that is instantiated in the test context (seeded if you like) when ‘execute’ is called (check Mockito documentation for more details on the syntax).

That’s it. Happy testing.