Handling errors in a REST way is seemingly simple enough: upon requesting a resource, when an error occurs, a proper status code and a body that contains a parseable message and using the content-type of the request should be returned.
The default error pages in Tomcat are ugly. Not only they expose too much of the server internals, they are only HTML formatted and making them a poor choice if a RESTful web service is deployed in that Tomcat container. Substituting them to simple static pages is still no enough since I want a dynamic response containing error information.

Here’s how to do it in 3 simple steps:

Step 1
In web.xml

   <servlet>
      <servlet-name>jsp</servlet-name>
      <url-pattern>/error.jsp>
   </servlet>
   <!-- other servlet mappings --&gt&

  <error-page>
    <error-code>404</error-code>
    <location>/error.jsp</location>
  </error-page>
  <error-page>
    <error-code>400</error-code>
    <location>/error.jsp</location>
  </error-page>
...
  <error-page>
    <exception-type>java.lang.Throwable</error-code>
    <location>/error.jsp</location>
  </error-page>

The servlet called jsp comes out-of the box with the Tomcat container and is the one responsible for processing JSPs . Make sure this mapping comes first, especially if you have a catch-all url-pattern that should map all other URLs to your webapp.

Step 2
Create a file called errors.jsp in the root of your web application:

<%@ page contentType="application/json" pageEncoding="UTF-8"%>
{
   status:<%=request.getAttribute("javax.servlet.error.status_code") %>,
   reason:<%=request.getAttribute("javax.servlet.error.message") %>
}

Step 3
Hook the exception handling using @ExceptionHandler:

@Controller
public class SomeController {
...

@ExceptionHandler(BusinessException.class, ValidationException.class, ...)
public void handleApplicationExceptions(Throwable exception, HttpServletResponse response) {
   String formattedErrorForFrontEnd = formatErrMessage(exception);
   if(exception instanceof BusinessException) {
      response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
formattedErrorForFrontEnd);
   } else if (exception instanceof ValidationException) {
      response.sendError(HttpServletResponse.SC_BAD_REQUEST, formattedErrorForFrontEnd);
   }
   ...
}

//other @ExceptionHandlers skipped

   private String formatErrMessage(Throwable exception) {
      //skipped
   }
}

I used @ExceptionHandler to map between application exceptions and error codes. You can apply message formatting as well if the default messages are not sufficient. Also, consider having an @ExceptionHandler(Throwable.class) for the “all other exceptions” case.

As you can see, there is no need to write another Spring Controller that handles exceptional paths. The error-page mappings in web.xml, a 2 line JSP and the standard @ExceptionHandler combination was sufficient.

Update Mar.04 Thanks to @ewolff some of the points described below are now official feature requests. One (SPR-6928) is actually scheduled in Spring 3.1 (cool!). I’ve updated the post and added all open tickets. Please vote!

This post is somewhat a response to InfoQ’s Comparison of Spring MVC and JAX-RS.
Recently I have completed a migration from a JAX-RS implementation of a web service to Spring 3.0 MVC annotation-based @Controllers. The aforementioned post on InfoQ was published a few days after my migration so I’m dumping below the list of problems I had, along with solutions.

Full list of issues:

• Same relative paths in multiple controllers not supported
• @ExceptionHandler is controller-centric
• Standard content negotiation can’t respond with a fixed response type
• JSR 303 bean validation not applied in @Controllers
• Formatting responses (i.e. Date) not working when using Spring formatter annotations

Same relative paths in multiple @Controllers not supported
Consider two Controllers where I use a versioned URL and a web.xml file that uses two URL mappings:

@Controller
public class AdminController {
   @RequestMapping("/v1/{userId}")
   public SomeResponse showUserDetails(String userId) {
      ...
   }
}

@Controller
public class UserController {
   @RequestMapping("/v1/{userId}")
   public SomeOtherResponse showUserStreamtring userId) {
      ...
   }
}
In web.xml:
	<servlet-mapping>
	  <servlet-name>public-api</servlet-name>
	  <url-pattern>/public</url-pattern>
	</servlet-mapping>
	<servlet-mapping>
	  <servlet-name>admin-api</servlet-name>
	  <url-pattern>/admin</url-pattern>
	</servlet-mapping>

Here I want to implement handling for /admin/v1/{userId} as well as /user/v1/{userId}. Each controller serves a different purpose (admin and public) and I want to keep the servlet application contexts separate for each servlet mapping (for instance add authentication and different view resolvers to all /admin/* URLs).
Deploying this configuration barfs the following exception:

java.lang.IllegalStateException: Cannot map handler [UserController] to URL path
[/v1/{userId}]: There is already handler [com.test.controller. UserController@6b177115]
mapped.

I would like Spring MVC to account for web.xml servlet mappings before deciding that two identical @RequestMappings in different controllers refer to the same URI. My solution was to have all @RequestMappings specifying the full path and resorting to a custom view to work around this issue).

@ExceptionHandler is controller-centric
RESTful web services respond with the usual HTTP responses (200s, 400s, 500s, etc.). These responses are being mapped from application exceptions and it only makes sense to specify the same @ExceptionHandler for all @Controllers since most (if not all) exceptions handled by them are for the same business domain. Currently SpringMVC only supports declaring and @ExceptionHander per controller and not globally per webapp. This forces the implementors to extend their @Controller classes from a common one where the @ExceptionHandler is specified or, to use composition and delegate exception handling to one service which leads to duplication. Both are ugly. There are several ways to deal with exceptions (check SPR-5622; please guys, add this to the main Spring documentation) but ideally I’d like to see a global scoped handler like @ExceptionHandler(scope=APPLICATION) where I can remap once framework exceptions to more suitable ones that translate into proper error codes (like working around SPR-5622 like issues).

Standard content negotiation can’t respond with a fixed response type (SPR-6937)
Here’s a simple requirement: Implement an RSS feed that serves a user’s activity stream.
Here’s another requirement (make that best practice):

Service methods should have as arguments primitives, wrappers or other POJOs and
 respond with void, primitives, wrappers or other POJOs.

I want testability, readability and portability for service classes and using framework objects for arguments/return objects adds unacceptable clutter.
As such, no framework objects are allowed as arguments or return types in a @Controller and in this case I don’t want to work with ModelAndViews just to be able to select the desired View.
Since this new requirement (add RSS view) belongs to the user domain, I want to add a method in the UserController class that returns a POJO which should be always unmarshalled into RSS regardless of request content type (there goes ContentNegotiationViewResolver). The same controller has other methods returning POJO view objects that I want marshalled into JSON or plain text response body and an HTTP response status. This is already available in RESTEasy via the elegant Produces annotation. The same RESTful controller should respond with different response types for different requests since I want to keep its domain centered around the user (and not response types).

...
public class UserController {
...
   @RequestHandler("/rss/{userId}")
   public SomePOJO getRssFeed(String userId) {
      ...
      //returns a POJO that should always be resolved by an RSS view
   }

   @RequestMapping("/activity/{userId}")
   public SomeOtherPOJO showUserStream(String userId) {
      ...
      //return POJO that should always be resolved by a JSON view
   }
...
}

It turns out this is not immediately possible. To my knowledge none of the included view resolvers provide a straight mapping between URLs and views so I had to write one that configures like this:

<bean class="UrlToViewViewResolver">
   <map name="urlToViewMap">
      <property key="/user/rss/*" value="rssView"/>
      <property key="/user/activity/*" value="jsonView"/>
      ...
   </map>
</bean>

The implementation (not included) is trivial and based on AntPathMatcher, a utility class readily available in Spring.

JSR 303 bean validation not applied in @Controllers (SPR-6928, scheduled for 3.1)
On Springsource’s weblog there’s a nice tutorial on how to get validation and type conversion in Spring 3.0 here. When it comes to validation though, Spring only supports it “after binding user input to it”. It is not possible to validate arguments and return types for a RESTful controller (no web forms or user input of any kind there). This means this won’t work out of the box:

   @RequestHandler("/admin/", method = RequestMethod.PUT )
   public void updateUser(@Valid User user) {
      ...
      //returns a POJO that should always be resolved by an RSS view
   }

since in a RESTful scenario the User object won’t be bound from a form but unmarshalled from XML/JSON/CSV or any other wire format.
To enable validation for a RESTful controller I chose to use @Valid at the method level and add an around advice that intercepts all methods annotated as such then use a Validator to validate arguments and return objects.

<aop:config proxy-target-class="true">
	<aop:pointcut id="validatableMethodExecution"
        expression="execution(* com.mycorp..*.*(..)) and @annotation(javax.validation.Valid)"/>

	<aop:aspect id="validateArgumentsAndReturnObject" ref="validatorAspect">
		<aop:around pointcut-ref="validatableMethodExecution" method="validate"/>
	</aop:aspect>
</aop:config>

public class ModelValidatorAspect {
	@Autowired
	private Validator validator;

    public Object validate(ProceedingJoinPoint pjp) throws Throwable {
        Set<ConstraintViolation> violations =new HashSet<ConstraintViolation>();
        for(Object obj : pjp.getArgs()) {
            if(obj != null) {
                violations.addAll(validator.validate(obj));
            }
        }
        if(!violations.isEmpty()) {
            throw new BadRequestException(composeErrorMessage(violations));
        }

        Object ret = pjp.proceed();

        //validate return object and throw exception if the validation fails.
        if(ret != null) {
            violations = validator.validate(ret);
        }
        if(!violations.isEmpty()) {
            throw new BusinessValidationException(violationsStr);
        }
   }
}

This could get more complex if you have deal with collections of validatable objects but it’s not hard to enhance the code to handle that.

Formatting responses (i.e. Date) not working using Spring formatter annotations
Spring comes with some niceties like @DateTimeFormat and @NumberFormat and naturally I want to rely on them by annotating the class attributes of return types used by @Controllers.
That’s not quite working since the formatters work with form-backed model objects. However, if you use a MappingJacksonJsonView there is a way of formatting using Jackson’s own @JsonSerialize.

Although at times it felt that the switch to SpringMVC was too much, the features that come with it (like abstracting web-related properties as annotations) will turn your RESTful controllers into annotated POJOs focused on the business domain which ultimately means maintainable code.

Update Mar.04 I left out several points about JSON marshalling that didn’t work as expected (Jackson marshaller is a handful).
- One enhancement I suggested to ease the pain of configuring JSON marshalling is creating a namespace similar to oxm (SPR-6943)
- The second was to add Jetisson support (SPR-6938) as an alternative to Jackson JSON marshaller.

I’m keen on your feedback regarding these issues so if you have better solutions please comment.

I want to write testable code and occasionally I bump into frameworks that make it challenging to unit test. Ideally I want to inject a service stub into my code then control the stub’s behavior based on my testing needs.
Commons Http Client from Jakarta facilitates integration with HTTP services but how to easily unit test code that depends on the HttpClient library? Turns out it’s not that hard.
I’ll cover both 1.3 and the newer 1.4 versions of the library since the older v1.3 is still widely used.
Here’s some typical service (HttpClient v1.3) we want to test. It returns the remote HTML page title:

public class RemoteHttpService {
   private HttpClient client;

   public String getPageTitle(String uri)  throws IOException {
     String contentHtml = fetchContent(uri);
     Pattern p = Pattern.compile("<title>(.*)</title>");
     Matcher m = p.matcher(contentHtml);
     if(m.find()) {
        return m.group(1);
     }
     return null;
   }

   private String fetchContent(String uri)  throws IOException {
      HttpMethod method = new GetMethod("http://blog.newsplore.com/" + uri);
      int responseStatus = client.executeMethod(method);
      if(responseStatus != 200) {
        throw new IllegalStateException("Expected HTTP response status 200 " +
"but instead got [" + responseStatus + "]");
      }
      byte[] responseBody = method.getResponseBody();
      return new String(responseBody, "UTF-8");
   }

   public void setHttpClient(HttpClient client) {
      this.client = client;
   }
}

with the HttpClient is injected at runtime (via some IoC container or explicitly).
To be able to unit-test this code we have to come-up with a stubbed version of the HttpClient and emulate the GET method.

What we really want is to be able to control the byte array returned by method.getResponseBody() and although we can inject a mocked HttpClient, the actual class that is responsible for producing the response body is GetMethod which is instantiated (and completely encapsulated) in the method body so we can’t inject a stubbed GetMethod version directly.
The solution is to create a simple HttpClient mock that is able to control the response status and response body. Here it is:

package org.apache.commons.httpclient;

import org.apache.commons.httpclient.*;

public class HttpClientMock extends HttpClient {
   private int expectedResponseStatus;
   private String expectedResponseBody;
   public HttpClientMock (int responseStatus, String responseBody) {
      this.expectedResponseStatus = responseStatus;
      this.expectedResponseBody = responseBody;
   }
   @Override
   public int executeMethod(HttpMethod method) {
      ((HttpMethodBase)method).setResponseStream(
new ByteArrayInputStream(expectedResponseBody.getBytes("UTF-8")));
      return expectedResponseStatus;
   }
}

There are two things to notice: Firstly, I placed HttpClientMock in the org.apache.commons.httpclient package to be able to have access to the protected member HttpMethodBase:setResponseStream. Secondly, I downcasted HttpMethod to HttpMethodBase to get access to setResponseStream (it’s safe since all concrete HttpMethods (GetMethod, PostMethod, etc) inherit from it).

Now I can unit test RemoteHttpService like this:

public class RemoteHttpServiceTest {
   private RemoteHttpService remoteHttpService = new RemoteHttpService();

   @Test
   public void extractTitleResponseOK()  throws IOException {
      String pageTitle = "A Title";
      String responseBody = "<title>" + pageTitle + "</title>";
      HttpClientMock mockHttpClient = new HttpClientMock(200, responseBody);
      remoteHttpService.setHttpClient(mockHttpClient);
      String resultedPageTitle = remoteHttpService.getPageTitle(""); //home page
      Assert.assertTrue(pageTitle.equals(resultedPageTitle));
   }
}

I can also add a negative test:

   @Test(expected = IllegalStateException.class)
   public void respondsWithInternalServerError()  throws IOException {
      HttpClientMock mockHttpClient = new HttpClientMock(500, "");
      remoteHttpService.setHttpClient(mockHttpClient);
      remoteHttpService.getPageTitle("");
   }

For HttpClient version 4.x here’s the refactored service code :

   private String fetchContent(String uri) {
        HttpGet httpget = new HttpGet("http://blog.newsplore.com/" + uri);
        HttpResponse response = client.execute(httpget);
        int responseStatus = response.getStatusLine().getStatusCode();
        if(responseStatus != 200) {
           throw new IllegalStateException("Expected HTTP response status 200 " +
"but instead got [" + responseStatus + "]");
        }
        InputStream responseStream = response.getEntity().getContent();
        byte[] responseBytes = IOUtils.toByteArray(responseStream);
        return new String(responseBytes, "UTF-8");
   }

Turns out that implementing a HttpClient mock by extending the DefaultHttpClient (the provided implementation of HttpClient) is not easy. The method I want to override is public HttpResponse execute(HttpUriRequest request) which is marked ‘final’ and the code in DefaultHttpClient looks just too complex to inject stubbed behavior.
What we actually want is to be able to control the return of the aforementioned ‘execute’ method since it encapsulates the response status and body (better encapsulation than HttpClient 1.3 if you noticed).
Luckily in this case we can use a readily available mocking framework, Mockito that can stub this entire method and return an HttpResponse that we instantiate with the needed status and response body. Here’s how to rewrite the tests using it:

@Test
public void extractTitleResponseOK() throws IOException {
	String pageTitle = "A Title";
	String responseBody = "";
	HttpResponse response = prepareResponse(200, responseBody);
	HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
	Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
			.thenReturn(response);

	remoteHttpService.setHttpClient(mockHttpClient);
	String resultedPageTitle = remoteHttpService.getPageTitle(""); // home page
	Assert.assertTrue(pageTitle.equals(resultedPageTitle));
}

@Test(expected = IllegalStateException.class)
public void respondsWithInternalServerError() throws IOException {
	HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
	HttpResponse response = prepareResponse(500, "");
	Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
			.thenReturn(response);
	remoteHttpService.setHttpClient(mockHttpClient);
	remoteHttpService.getPageTitle("");
}

private HttpResponse prepareResponse(int expectedResponseStatus,
               String expectedResponseBody) {
      HttpResponse response = new BasicHttpResponse(new BasicStatusLine(
    		  new ProtocolVersion("HTTP", 1, 1), expectedResponseStatus, ""));
      response.setStatusCode(expectedResponseStatus);
      try {
		response.setEntity(new StringEntity(expectedResponseBody));
	} catch (UnsupportedEncodingException e) {
		throw new IllegalArgumentException(e);
	}
	return response;
}

The meat of the above code is

HttpClient mockHttpClient = Mockito.mock(HttpClient.class);
HttpResponse response = prepareResponse(...);
Mockito.when(mockHttpClient.execute(Mockito.any(HttpUriRequest.class)))
	.thenReturn(response);

that instructs the mocked HttpClient to return a HttpResponse that is instantiated in the test context (seeded if you like) when ‘execute’ is called (check Mockito documentation for more details on the syntax).

That’s it. Happy testing.

It took a while since the previous feature update to Spincloud. I have done a number of upgrades to the underlying tech and some intensive code refactoring but nothing visible. The time has come for another eye candy: heat maps. It is a map overlay that shows a color-translated temperature layer based on interpolated values of the current weather conditions. It gives a quick indication of the average temperature across all land masses where data is available.

Naturally in areas where data is more dense, the visual representation is better. The toughest job was to figure-out the interpolation math. I still have to refine the algorithm currently based on the inverse distance weighting algorithm which interpolates scattered points on a surface. Apparently a better one to use is kriging but I have yet to implement it.
The temperature map is generated from current temperatures and is updated once every hour. You can toggle the overlay by clicking the “Temp” button found on the top of the active map area.

In the first part I’ve left-off with some good news: successful deployment in the local GAE container. In this second part I’ll talk about the following:

• - Loading data and browsing
• - Table indexing
• - Limitations of datastore queries
• - More datastore limitations and JPA issues
• - Deployment
• - Performance
• - Production monitoring
• - Usage and quotas
• - Other limitations
• - Final thoughts

Loading data and browsing
After finishing-off the first successful deployment, next on the agenda was testing out the persistence tier but for this I needed some data to work with in the GAE datastore.
Along with the local container Google makes available a local datastore but so far the only way to populate the datastore is described here and it uses python. Furthermore, currently there is one documented way to browse the local datastore using the local development console but this again requires the python environment. There’s a hint from Google though that there will be a data viewer for the Java SDK. In the mean time I voted up the feature request.

Back to bulk loading, after voting on the feature request to have a bulk uploader, I decided not to use the python solution but to handcraft a loader that will fill-in two tables: COUNTRY and COUNTRY_STATE (for federated countries like US and CAN). Since there’s no way to schedule worker threads (but is on it’s way) the only mechanism to trigger the data loading is via URLs. I’m using Spring 3.0 and so it wasn’t hard to create a new @Controller, some methods to handle data loading then map them to URLs:

@Controller
public class BulkLoadController {
	@RequestMapping("/importCountries")
	public ModelAndView importCountries() {
           ...
        }
	@RequestMapping("/importProvinces")
	public ModelAndView importProvinces() {
           ...
        }
}

Since my SQL script that creates and loads the COUNTRY table is no good (Google’s BigTable is not RDBMS) I chose to translate all INSERT operations to CSV.

Before
INSERT INTO COUNTRY (NAME, TWO_L_CODE, THREE_L_CODE, CODE, TIMEZONE)
  VALUES("Antarctica","AQ","ATA",10,12);

After
Antarctica,AQ,ATA,10,12

You don’t need to create the table, the GAE container runtime takes care of that both locally and in the hosted environment.
Once I created the CSV file (I named it countries.txt), I made it available to the webapp by placing it in the root of the war folder as the container has support for reading files from the application folders. So the country importer method will look like this (exception handling removed for brevity):

@RequestMapping("/importCountries")
public ModelAndView importCountries() {
	FileInputStream countriesF = new FileInputStream("countries.txt");
	BufferedReader br = new BufferedReader(new InputStreamReader(countriesF));
	String line = null;
	while((line = br.readLine()) != null) {
		persistCountry(line);
	}
        return new ModelAndView("success");
}

The only thing I had to do was to implement persistCountry which I did by simply adding a new method on the DAO that would persist a Country entity and injecting the DAO in BulkLoadController. I fired the URL request and I got the OK page back. Since I couldn’t browse the datastore I resorted to looking in the local_db. file located in WEB-INF/appengine-generated folder and I could see the country names.
Handling Transactions
One notable mention regarding transactions is that you can’t save multiple objects in a single transaction if they are all root entities (see comment). I bumped into this while sorting the following exception:

Caused by: java.lang.IllegalArgumentException: can't operate on multiple entity groups in a single transaction. found
 both Element {
  type: "stations"
  id: 500
}
 and Element {
  type: "stations"
  id: 599
}

To relief the database from unneeded burden, I’d batch multiple entity updates then issue a single transaction commit. With GAE’s datastore this is not possible so I had to open and commit transactions for every entity update. This resulted in notably high CPU usage both locally and on the hosted environment but I couldn’t find any better way of handling this. In my case I had to remove any batching logic from my respective classes and annotate the methods responsible with insert/updates (like persistCountry’) with @org.springframework.transaction.annotation.Transactional like in the ‘persistCountry’ method above or use programmatic transaction management for uploading weather stations as we’ll see immediately.

There are two more tables I still had to bulk-provision: PROVINCE and STATION but the pattern should the same. So it seemed.
When I tried to provision the PROVINCE table I got no errors but no rows in the database either. My class looks like this:

@Entity
public class Province implements Serializable {

  private ProvincePK countryCodePK;
  private String provinceName;
  private float timeZone;

  @EmbeddedId
  public ProvincePK getProvincePK() {
    return provincePK;
  }
...

@Embeddable
public class ProvincePK implements Serializable {

  private String stateCode;
  private String countryCode;
...

From the documentation [via groups] I learned that the composite PKs are not supported. However it would have been useful to indicate this either in the enhancement step (still hate it) or deployment. So issue 1658 was born. Vote it if you want it implemented.
The solution is not easy in most cases but I reverted to a single field PK in the Province entity since it was possible in my case. I had the provinces imported after this fix.
Now the harder part. I am using ASCII resources from the internet to import weather stations; In all there are about 20000 stations imported from several resources, both HTTP and FTP. The first thing I learned was that the AppEngine has a request timeout of 30 seconds. I learned this the hard way after trying to perform the same import I’m using in Spincloud only I was using a URL request to trigger it (I’m normally using JMX but it’s not supported by the hosted AppEngine). The error I bumped into was:

com.google.apphosting.runtime.HardDeadlineExceededError: This request (8c8ca701dccad394)
 started at 2009/06/05 07:30:05.088 UTC and was still executing at 2009/06/05 07:30:33.944 UTC.

Notice the timeline, there is an automatic timeout enforced by the hosted runtime.
Since we can’t use long running tasks, the alternative is to chain requests that will incrementally add batches of items into the database and we’ll use a request parameter to control batch increments.

	@RequestMapping("/updateWeatherStations")
	public ModelAndView updateWeatherStations(HttpServletRequest request) {
		int position = Integer.parseInt(request.getParameter("position"));
		wmoStationsImporter.saveWeatherStations(position);
	}

The method saveWeatherStations will save a set of items into the datastore; the ‘position’ parameter indicates the start index of the item list; the batch size is constant and so with each request the ‘position’ parameter will have to increment with the batch size.

private static final int BATCH_SIZE = 50;
public void fetchWMOData(int position) {
	int recordNo = 0;
	int startIndex = position * BATCH_SIZE;
	String nextLine = null;
	BufferedReader isr = new BufferedReader(new InputStreamReader(getStationsFileInputStream()));
	String nextLine = null;
	while ((nextLine = isr.readLine()) != null) {
		recordNo++;
		if (recordNo  startIndex + BATCH_SIZE) {
			logger.info("Finished batch");
			return;
		}
		//parse record and save weather station
	}
}

Some observations with the above code: first, I’m reading a local ASCII file that contains stations I want to save into the datastore. Secondly I found a reasonable value for the BATCH_SIZE; the higher the number, the more time the processing takes (reading from file, parsing record and saving entity into the datastore) and you may hit again the 30 seconds limit. I’m also using local files to load the ASCII data since the request limit is so tight and I don’t want to wait for resources to be downloaded off the internet for most part.

What’s left is to “weld” requests together using a shell script. The number of requests is the total number of records divided by BATCH_SIZE.

#/bin/sh
count=25
while [ "$count" -le 133 ]
do
  wget "http:///updateStations?position=$count"
  let "count += 1"
  sleep 8
done

To play nicely with the hosted environment and not to get throttled, I’m pausing 8 seconds after each request.
In conclusion I really need a better way to handle bulk loading. Best is to be able to use SQL data dumps to load the datastore.

Table indexing
The datastore indexes each column so you don’t need to do it explicitly. However, for all JPA queries that use more than one field in the criteria, composite field indexing is needed. The local environment will create an datastore index config file for you (WEB-INF/datastore-indexes.xml) and moreover, it will create the indexing configuration but only after actually executing a query at runtime. This is not optimal and I’d like to see a way to auto-generate the indexes by processing JPA queries (as part of the enhance step maybe?). More about the GAE’s datastore indexing here
Spatial indexing is not supported.

Limitations of datastore queries
If you’re thinking of a straight migration of all existing JPA queries to the datastore be warned. GAE is using a datastore based on the BigTable storage system. BigTable follows a different philosophy than traditional RDBMS and so it doesn’t support many of the standard relational aspects of a database. Here are the documented limitations.I have found that queries that use UPPER and LIKE are not supported either.

org.datanucleus.store.appengine.query.DatastoreQuery$UnsupportedDatastoreFeatureException:
Problem with query : Unsupported method <upper> while parsing expression: InvokeExpression {[null].upper(PrimaryExpression {station, usState})}

org.datanucleus.store.appengine.query.DatastoreQuery$UnsupportedDatastoreOperatorException:
Problem with query <SELECT station FROM com.newsplore.weather.bo.Station station WHERE
 UPPER(station.name) LIKE :stationName ORDER BY station.name>:
App Engine datastore does not support operator  LIKE

I can live without the UPPER but queries using LIKE are a must and I was quite surprised to find that they are not supported (TODO: study the BigMap . I have yet to find a replacement for LIKE…
Plain SQL is also unsupported (not a relational datastore…). As well, spatial extensions -if supported- are not documented. I’m using several spatial SQL queries in Spincloud, mostly to select data from a geographic area.

More datastore limitations and JPA issues
- Primary keys cannot be int types, I had to switch a bunch to Long.
- JPA queries need fully qualified class names; only “select s from com.package.Station s” will work.
- Inconsistent NullPointerException when accessing some entities the first time:

ERROR MainHandlerExceptionResolver:18  Exception processing page: Name is null
java.lang.NullPointerException: Name is null
	at java.lang.Enum.valueOf(Enum.java:195)
	at com.google.appengine.api.datastore.dev.CompositeIndexManager$IndexSource.valueOf(CompositeIndexManager.java:64)

After repeating the request the entities are fetched OK.
- No composite PKs allowed.
- java.sql.TImestamp not supported. The following exception is thrown when accessing an entity containing such type:

nested exception is org.springframework.dao.InvalidDataAccessApiUsageException: java.sql.Timestamp is not a support
ed property type.; nested exception is java.lang.IllegalArgumentException: java.sql.Timestamp is not a supported prop
erty type.]

I had to change all java.sql.Timestamp fields to java.util.Date. I’m a bit baffled about this one since Timestamp is a subtype of Date. I also changed the @Temporal annotations to TemporalType.DATE.
- No support for uniqueness constraints. I removed all “unique=true” clauses in @Column(name=”SOME_COLUMN”, unique=true)
- I couldn’t make @ManyToOne relationships work. I have such relationship for instance between Station (many) and Province (one). Within the transaction that persists a Station, a Province is associated with a Station so first the Province is loaded from the datastore by PK then set it into the new Station object. The error is:

Caused by: java.lang.IllegalArgumentException: can't operate on multiple entity groups in a single transaction. found both Element {
  type: "PROVINCE"
  name: "NO"
}
 and Element {
  type: "stations"
  id: 21
}

This is the same type of constraint that prevents working on multiple root entities within the same transaction.
- Not equals and IN operators are not supported
- I’m using method-level annotations for entities and I noticed that both the attribute and the getter/setter have to follow the JavaBean rules. With all other JPA providers I can have an arbitrary named field as long as accessors follow JavaBean rules.

Deployment
You can use the neat Eclipse plugin to push the webapp to the hosted GAE or use an included utility called appcfg.sh, your choice. One important limitation is that the total number of files that are allowed to be uploaded is 1000. This issue was solved in the Python environment with the introduction of the zipserve module but there’s no equivalent on the Java side yet. In the mean time, one way around it is to deploy a second webapp containing only static files and update the web pages accordingly (or use a real CDN).

Performance
Static pages perform quite well. Querying the datastore with indexed data ididn’t strike me as blinking fast. I can pull for instance 200 weather stations from the datastore (through JPA) and render the results on a JSP web page but if I bump the number of records to 1000 I bump into another limitation of the AppEngine which is a limit on the datastore request:

ERROR MainHandlerExceptionResolver:18  Exception processing page: datastore timeout:
operation took too long.
com.google.appengine.api.datastore.DatastoreTimeoutException: datastore timeout: operation took too long.
	at com.google.appengine.api.datastore.DatastoreApiHelper.translateError(DatastoreApiHelper.java:37)
	at com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(DatastoreApiHelper.java:55)
...

According to my non-scientific measurement the datastore timeout is around 5 seconds. A query that returns 1000 records would exceed this deadline. Arguably one shouldn’t issue queries that return such a high number of records but it’s better to be aware of this constraint.
I have also experienced an OutOfMemoryException on the hosted environment and I couldn’t find any facility to redeploy or somehow reset the runtime from the browser’s admin interface.

Production monitoring
Google provides an online web console to monitor the usage, check logs, check quotas and browse the live datastore. I found it useful but a bit more work is needed on the datastore browser to support GUI-oriented querying.

Usage and quotas
Google offers free usage up to a limit, afterwards there’s a paid scheme. The usage is free for up to 5 million page views per month which seems quite reasonable.

Other limitations
A list of what’s runnable in the AppEngine is found here Of all, I’ll focus on just one: saving files on the hosted file system is forbidden. I happen to save lots of image tiles for the radar layer so I’ll be forced to use the BigTable to store the image tiles, not necessarily a bad thing as it forces me to learn more about BigTable. On the other hand I have to point out that this limitation adds to the challenges of migrating to the AppEngine cloud.

Final thoughts
It’s quite clear that Google had an aggressive timeline for releasing GAE for Java. GAE(J) is still incomplete and there are some glaring omissions that should be corrected sooner rather than later:
- datastore backup, browser and bulk uploader tool
- spatial extensions support in the datastore. This is a must since given the opportunity for integration with Google Maps.
- background processing (coming soon)
- extending queries to support the LIKE operator or an alternative
- better JPA support. I mean better error messages and better documentation with unsupported features, proper Level 2 cache integration with memcache.
The roadmap is laid-out here

In conclusion, moving an existing project to Google’s infrastructure is not a trivial job. The differences between a traditional database and BigMap concepts are at times irreconcilable and code refactoring may prove to be too big a task. I happen to agree with Mr. Cranrell’s remarks regarding the challenges with regards to migrating to a non-relational datastore (perfect timing, the article was posted just today) but at the same time I have to appreciate the (revolutionary?) efforts from Google (BigMap) and Amazon (SimpleDB) to shift the market focus from RDBMS to perhaps better alternatives.

The AppEngine for Java is a worthy player in the new cloud services industry that is shaping as we speak. It is especially relevant for greenfield projects that don’t come with the standard baggage of constraints (most notably the relational database mapping) and where new and innovative ways of architecting an application can be explored.

A short two months back I posted an article describing how to upgrade to Spring 3.0 M2. Spring folks are releasing at breakneck speed and so I got busy again upgrading spincloud.com to Spring 3.0 M3 released at the beginning of May. Just yesterday (June 3rd) the team released Spring Security 3.0 M1 and I decided to roll this in Spincloud as well.

Upgrading Spring Security from 2.0.4 to 3.0.0 M1
For Spring Security 3.0.0 M1 I’m doing a “soft” upgrade since I had done my homework when I migrated from Acegi. I won’t use any of the new 3.0 features, just getting ready to use them.

To digress a bit, Spring Security a technology that is harder to swallow due to its breadth. To simplify the picture, Spring Security (former Acegi) provides three major security concerns to enterprise applications:
- Authorization of method execution (either through standard JSR-250 JEE security annotations or via specific annotation-based method security).
- HTTP Request authorization (mapping URLs to accessible roles using Ant style or regexp’ed paths, dubbed channel security).
- Website authentication (integrating Single Sign On (SSO) services by supporting major SSO providers, HTTP BASIC authentication, OpenId authentication and other types).
For Spincloud I’m using OpenId for authentication and Channel Security to secure website pages.

To perform the upgrade you have to first grab the new jars. Here’s the Maven config:

	<repository>
		<id>Springframework milestone</id>
		<url>http://maven.springframework.org/milestone</url>
	</repository>
	<properties>
		<spring.version>3.0.0.M3</spring.version>
		<spring-security.version>3.0.0.M1</spring-security.version>
	</properties>
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-core</artifactId>
			<version>${spring-security.version}</version>
			<scope>compile</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-config</artifactId>
			<version>${spring-security.version}</version>
			<scope>compile</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-openid</artifactId>
			<version>${spring-security.version}</version>
			<scope>compile</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-taglibs</artifactId>
			<version>${spring-security.version}</version>
			<scope>compile</scope>
		</dependency>
		<dependency>
			<groupId>org.springframework.security</groupId>
			<artifactId>spring-security-web</artifactId>
			<version>${spring-security.version}</version>
			<scope>compile</scope>
		</dependency>

Make sure your build process refers only to these jars.
in 3.0.0 M1, the package structure was changed. Here’s what I had to do to upgrade the packages:
Before

import org.springframework.security.Authentication;
import org.springframework.security.context.SecurityContext;
import org.springframework.security.context.SecurityContextHolder;
import org.springframework.security.context.SecurityContextImpl;
import org.springframework.security.providers.openid.OpenIDAuthenticationStatus;
import org.springframework.security.providers.openid.OpenIDAuthenticationToken;
import org.springframework.security.userdetails.UserDetails;
import org.springframework.security.userdetails.UserDetailsService;

After:

import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContext;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.core.context.SecurityContextImpl;
import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.core.userdetails.UserDetailsService;
import org.springframework.security.openid.OpenIDAuthenticationStatus;
import org.springframework.security.openid.OpenIDAuthenticationToken;

The security namespace has changed too so I updated my security XML file namespace and I had only to change the packages for standard Spring Security beans defined to match the new package structure.

Upgrading Spring from 3.0.0 M2 to M3
This should be easy. First, grab the new jars through Maven by using the same Maven config I described at the beginning of this post and change the spring.version to 3.0.0.M3. I’m planning to move more XML configuration to annotations and M3 enables more of that (@Lazy for instance). I’m still waiting for the scheduling namespace due to appear in this milestone but pushed to RC1 apparently. I’m craving for some annotation-based timer configuration too…

Build and deployment
At first, the build failed on me with the follwing error:

java.lang.NoClassDefFoundError: org/springframework/asm/ClassVisitor
	at org.springframework.context.support.AbstractRefreshableApplicationContext.customizeBeanFactory(AbstractRefreshableApplicationContext.java:205)
	at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:125)
	at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:419)

This is because M3 (finally) repackages asm. So I deleted the asm libraries I used to fix the M2 issues and added the repackaged Spring asm library:

		<dependency>
			<groupId>org.springframework</groupId>
			<artifactId>org.springframework.asm</artifactId>
			<version>${spring.version}</version>
		</dependency>

If you’re getting this error on container start-up:

SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.parsing.BeanDefinitionParsingException: Configuration problem: Unable to locate Spring NamespaceHandler for XML schema namespace [http://www.springframework.org/schema/security]
Offending resource: ServletContext resource [/WEB-INF/classes/applicationContext-security.xml]

means you have forgotten to add security-config jar (see the Maven config in the Security section).

That’s all. Done in 3 hours with no problems at all.

When Google announced that Java is the second language that the Appengine will support I almost didn’t believe it given the surge of the new languages and the perception that Java entered legacy but the JVM is a powerful tried-and-true environment and Google saw the potential of using it for what it is bound to

become: a runtime environment for the new and exciting languages (see JRuby and Grails). The JVM is the new gateway drug in the world of languages.

Note: I’ll break down this review into two posts as it’s too extensive to cover everything at once. This first part is about initial setup, JPA and some local webapp deployment issues. In the second part I’ll describe how to load data into the datastore, data indexing, how I reached some of the limitations of the AppEngine and how to get around some of them.

I managed to snatch an evaluation account and last few days I have been playing with it. (as of today Google opened the registrations to all). My goal was (again) simple: to port Spincloud to the AppEngine. The prize: free hosting which amounts to $20/month in my case (see my post about Java hosting providers), some cool monitoring tools that I can access on the web and of course the transparent super-scalability that Google is touting, in case I get techcrunched, slashdotted or whatever (slim chances but still…).
I am a couple of weeks into the migration effort and I can conclude that the current state of the AppEngine is not quite ready for Spincloud as I’ll detail below but it comes quite close. The glaring omission was scheduling but Google announced today that the AppEngine will support it (awesome!) and I plan to use it from day one.

My stated goal is running Spincloud on AppEngine. Here are the technologies I use in Spincloud:
- Spring 3.0 (details here)
- SpringAOP
- SpringSecurity with OpenId
- Webapp management with JMX (I know, it won’t fly)
- Job scheduling using Spring’s Timer Task abstraction (not optimistic about this one either)
- SiteMesh (upgraded to 2.4.2 as indicated)
- Spatial database support including spatial indexing
- JPA 2.0 (back story here)
- Level 2 JPA cache, query cache.
- Native SQL query support including native SQL spatial queries.
- Image processing (spincloud scraps images and processes pixel-level information to get some weather data)

That’s a lot of stuff but first things first. I recommend creating a new repository branch (I use Subversion) dedicated to the new environment as there may be many things to change and you still want to continue the development unaffected on your mainline.

I downloaded the latest SDK and exploded it under the tools/ folder. Make sure you install the Eclipse plugin as it’s pretty useful.

After the initial setup I started-out with the JPA part since this seemed to be the most challenging piece of technology that I had to change. AppEngine uses DataNucleus JPA provider, a poor choice in my opinion since they are a JDO provider turned to JPA (much like KODO/OpenJPA) and so they drag a lot of JDO baggage along. I haven’t heard of them before and they don’t seem to be that popular (why not Hibernate or the new RI: EclipseLink? Update: because JPA is RDBMS oriented while JDO is not and the backing store of GAE is BigTable which is not RDBMS). DataNucleus JPA requires a pre-compilation step that does bytecode enhancement which I completely dislike as it’s archaic (all modern JPA providers don’t have this anymore) but I had no choice. So I’ve followed the reference documentation. The only thing not described there is the ant integration.
Here is what I added to my build.xml

<project name="weather"  default="build"  basedir=".">
...
  <property name="sdk.dir" value="/Users/florin/tools/appengine-java-sdk-1.2.1"/>
  <import file="${sdk.dir}/config/user/ant-macros.xml" />

  <path id="enhancer.classpath">
    <pathelement path="${bin.dir}"/>
    <fileset dir="${sdk.dir}/lib">
      <include name="appengine-tools-api.jar"/>
    </fileset>
    <fileset dir="${lib.dir}">
      <include name="gisjts/jts*.jar"/>
      <include name="appengine/appengine-api-1.0-sdk-1.2.1.jar"/>
    </fileset>
  </path>

  <target name="enhance" depends="compile">
    <enhance verbose="true" classpathref="enhancer.classpath">
      <fileset dir="${bin.dir}">
        <include name="com/newsplore/weather/bo/*.class"/>
      </fileset>
    </enhance>
  </target>

  <target name="build-appengine" depends="compile, enhance, jar"/>
...

Essentially this is to accommodate the enhancement step (notice how the “enhance” target is interposed between compile and jar in the build-appengine target).
I ran the “build-appengine” target and noticed the first exception:

  [enhance] Please see the logs [/tmp/enhance64409.log] for further information.

That’s a convoluted way of looking at the errors (why not showing the exception in the console?). Here’s the actual error found in the said file:

java.lang.RuntimeException: Unexpected exception
...
   Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: java.lang.NoClassDefFoundError: Lorg/apache/log4j/Logger;
  at java.lang.Class.getDeclaredFields0(Native Method)

This was because one of my business objects used log4j. Adding the log4j jar file didn’t help and the solution was to remove the logging from the class altogether.

After cleanup I got this error:

  [enhance] SEVERE: Class "com.newsplore.weather.bo.GeometryUserType" was not found in the
CLASSPATH. Please check your specification and your CLASSPATH.
  [enhance] org.datanucleus.exceptions.ClassNotResolvedException: Class
"com.newsplore.weather.bo.GeometryUserType" was not found in the CLASSPATH.
    Please check your specification and your CLASSPATH.

That was because GeometryUserType (a class I used to handle Geometry types in Hibernate) was in the com.newsplore.weather.bo package and the enhancer went through all classes in this package thinking they are all @Entities and failed with this one. Weird, I thought that the enhancer was smarter and filtered-out all non @Entities… The solution is either to specify all the classes to be enhanced in the “enhance” ant target or move the non-entities to other packages (a good practice). In my case I deleted the class since I wasn’t using it anyway (I used this utility class before Hibernate became a JPA provider).
After this cleanup, the enhance target spat yet another error:

[enhance]Class com.newsplore.weather.bo.UsForecast has property weatherIconByCode declared
    in MetaData, but its getter method doesnt exist in the class!
  [enhance] May 27, 2009 4:03:50 PM org.datanucleus.enhancer.DataNucleusEnhancer main
[enhance] SEVERE: DataNucleus Enhancer completed with an error. Please review the enhancer
log for full details. Some classes may have been enhanced but some caused errors
  [enhance] Class com.newsplore.weather.bo.UsForecast has property weatherIconByCode
declared in MetaData, but its getter method doesnt exist in the class!
  [enhance] org.datanucleus.metadata.InvalidMetaDataException: Class
com.newsplore.weather.bo.UsForecast has property weatherIconByCode declared in MetaData,
 but its getter method doesnt exist in the class!
  [enhance]     at org.datanucleus.metadata.ClassMetaData
.populateMemberMetaData(ClassMetaData.java:497)

So a domain object (UsForecast) has a property called weatherIconByCode. Hmm, I didnt remember of such property but looking at the source I saw that the offender was a private utility method called getWeatherIconByCode(String code) that was being processed by the enhancer as if it was a property (I’m using annotations bound to getters and not to fields). Since adding @Transient didn’t fix it, the solution is to rename the method not to start with get/set (or move it in an utility class which you shouldn’t do if you follow DDD).
To my relief, after this fix the build didn’t yield any more errors so I had a nice webapp folder ready to be deployed. AppEngine has a local sandbox (based on Jetty) that should be used before deploying into the cloud. You can start the appserver using an ant target but I chose to get more control and create a script that accomplishes the same thing. Here it is:

cd build/webapp
java -classpath <appengine-sdk-home>/lib/appengine-tools-api.jar
com.google.appengine.tools.development.DevAppServerMain --port=8080 --address=localhost

Save this in a file called startgae.sh (mind the path, you should be able to cd to the webapp folder from it) then make it executable.
Using this script I fired-up the container and got no startup error. This was weird since I knew I was using JMX (via Spring and annotations) and as well the task scheduler (using Quartz) was seemingly up and running.
I hit the home page to see what has been deployed and I get a 404 error. My web.xml has this mapping:

  <servlet>
    <servlet-name>weather</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
  </servlet>

  <servlet-mapping>
	<servlet-name>weather</servlet-name>
	<url-pattern>/</url-pattern>
  </servlet-mapping>

but this doesn’t work in the local servlet container. This was quite annoying as not only the home page doesn’t work, the static folders (css/, js/, img/) are routed through the weather servlet too. Surprisingly as it turns out, this mapping worked perfectly in the AppEngine environment. To be able to continue with the work in the local environment, I had to define fine grained mappings for all the URLs.
After this change, the first URL that started working was the /faq which provided the first breakthrough. Although only the presentation tier is in use on the /faq page, the fact that the page worked provided a bunch of good news: Spring 3.0 DI container, SiteMesh, Spring Security and SpringAOP were all correctly running.

Stay tuned for the second part detailing more JPA, datastore indexing, a bunch of the AppEngine limitations, a bit of local environment hacking and how to use “request welding” to accomplish long running tasks.

I have been thinking to write about this subject a while back when project Spincloud was still under development. I was even thinking about making this the first post on my blog.
The idea is simple: you have location-based data (POIs for instance) stored in some database (preferably a spatial DB) and now you want to perform a select statement that will indicate the area that should include the points we want. In case of Spincloud’s weather map, we want the weather reported by the stations located within a given area determined by the Google Map viewport that the user is currently browsing.
In all my examples I’ll use SQL Spatial Extensions support, specifically MSQL spatial extensions.
Here’s a visual representation of the spatial select (the red grid is the area where we want to fetch the data):

This is quite easy to accomplish by issuing a spatial select statement on the database:

select * from POI where Contains(GeomFromText
    ('POLYGON ((-30 32, 30 -8, -89 -8, -89 32, -30 32))', LOCATION))

But what about selecting an area that crosses the 180 degrees longitude? Let’s say we want to select data in an area around New Zealand that starts at 170 degrees latitude and ends at -160 degrees latitude going East. The selected area will look like this:

If we are to describe this with a polygon, we may be tempted write it like this using Spatial SQL parlance:

POLYGON ((-160 -23, -160 -55, 170 -55, 170 -23, -160 -23))

but you’d be wrong. In fact this polygon describes an area on the map that excludes exactly our area of interest:

This happens specifically because the 180 degrees meridian is crossed (the next point of longitude beyond 180 degrees is -180 degrees).
To fix this special case, you have to test for this occurrence and if so we’ll construct two polygons, split by the 180 degrees meridian (notice the exaggerated division between the polygons east and west of the 180°):

Two spatial selects will have to be issued in order to account for areas on each side of the 180° meridian.

If you’re using Google Maps then you most likely use getBounds() to get the bounding box. The API reference specifically states that the returned box is “a rectangle in geographical coordinates, including one that crosses the 180 degrees meridian”. I use the excellent Vividsolutions JTS for all GIS related code so to construct the polygons I use the following logic:

public List getPOIsInArea(float neLng, float neLat, float swLng, float swLat) {
  GeometryFactory gf = new GeometryFactory();

  if((neLng < swLng) ) { //cross lat or long 180 degree boundary
   Polygon eastPol = gf.createPolygon(gf.createLinearRing(new Coordinate[] {
    new Coordinate(neLng, neLat),
    new Coordinate(-180, neLat),
    new Coordinate(-180, swLat),
    new Coordinate(neLng, swLat),
    new Coordinate(neLng, neLat)
     }), new LinearRing[]{});

   Polygon westPol = gf.createPolygon(gf.createLinearRing(new Coordinate[] {
    new Coordinate(swLng, swLat),
    new Coordinate(180, swLat),
    new Coordinate(180, neLat),
    new Coordinate(swLng, neLat),
    new Coordinate(swLng, swLat)
     }), new LinearRing[]{});

     //select POIs inside both eastPol and westPol
  } else {
      Polygon pol = gf.createPolygon(gf.createLinearRing(new Coordinate[] {
     new Coordinate(neLng, neLat),
     new Coordinate(neLng, swLat),
     new Coordinate(swLng, swLat),
     new Coordinate(swLng, neLat),
     new Coordinate(neLng, neLat),
     }), new LinearRing[]{});

     //select POIs from pol
  }
}

The above code creates two polygons if the 180° meridian is crossed (E longitude is greater than the W longitude for the bounding box):

East polygon:
POLYGON ((-160 -23, -180 -23, -180 -55, -160 -55, -160 -23))

West polygon:
POLYGON ((160 -55, 180 -55, 180 -23, 160 -23, 160 -55))

Both polygons will be used when selecting POIs from the database. The resulted POI list will be of course the union between the POIs in the east and west polygons.

What’s remaining is to present the data on the map but since the spatial select may return more points than the maximum we want to show on the map (i.e. 1000 returned versus 20 we want shown), we’ll have to employ some balanced point reduction step (we call it clustering) before the POIs are presented on the map. I’ll talk about point clustering and using the cool k-means algorithm in a future post.

As I’m using the ubiquitous Hibernate 3.3 as the JPA 1.0 provider for Spincloud, I decided to try out another one. I had tried OpenJPA (spawned from Kodo JDO) when they only supported build-time bytecode enhancement and it was a pain to make it work. It worked all right but boy what a pain. There’s now an agent to provide on-the-fly enhancement but I’ll take transparent enhancement anytime.
I’ve heard about EclipseLink before. The project started when Oracle donated the respectable TopLink project to the Eclipse foundation. If the solid reputation behind TopLink was a good enough argument for me to try it, the announcement that it will be the JPA 2.0 reference implementation convinced me that I should try it out.
My goal is to evaluate if EclipseLink is production-ready. I’m applying a complex set of evaluation criteria (joking): if it can run Spincloud then it is (I was inspired by Seifer’s interview on Infoq about Ruby VMs; when asked what was the criteria for qualifying if a Ruby VM is production ready, he answered: if it runs Rails).

I have the following JPA requirements:
- column mappings, one-to-one, one-to-many
- supports BLOB fields
- supports NamedQueries and NamedNativeQueries
- support for object cache and query cache
- deployment/operational nice to have: ease of maintaining compatibility with both EclipseLink and Hibernate in the source code and runtime. Ideally I should plug-in any JPA provider without changing a single line of code. This was not attainable as I’ll explain below.

I started by downloading the binaries. I’m using Maven to bring the jars so I’ve followed the instructions here. I only changed the version since I wanted to use v1.1.0:

  <dependency>
    <groupId>org.eclipse.persistence</groupId>
    <artifactId>eclipselink</artifactId>
    <version>1.1.0</version>
    <scope>provided</scope>
  </dependency>

There’s a single jar file called eclipselink-1.1.0.jar downloaded which is a nice change from the multitude of Hibernate jars I was accustomed with.

Next, I created a new persistence.xml file since I didn’t know how different it would be from the one tied to Hibernate. Initially I just changed the JPA provider to:

  <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>

as the documentation stated.
Then I had a look at my current NamedQueries and NamedNativeQueries that looked like this:

@org.hibernate.annotations.NamedQueries({
	@org.hibernate.annotations.NamedQuery(name="findStationByPK",
	        query="select station from Station station where station.id=:stationPK"
	        , cacheable=true, cacheRegion="weatherStationsRegion"),
...

and I recalled that I did this in order to make use of Hibernate’s Query cache facility, alas a non-standard way of cache enablement (not until JPA 2.0 anyway). I had to put query caching to rest and switched all my Named*Queries to standard JPA:

@javax.persistence.NamedQueries({
	@javax.persistence.NamedQuery(name="findStationByPK",
	        query="select station from Station where id=:stationPK"),
...

I’m not using the integration unit tests I have due to the issues I had with the Spring integration test framework I’m using after switching to Spring 3.0 so I’m using ad-hoc testing for now. After rebuilding/deploying the project I didn’t notice any error on Tomcat start-up so I just tried the GUI, displaying the weather in a map area. I hit the first error:

Exception [EclipseLink-7060] (Eclipse Persistence Services - 1.1.0.r3639-SNAPSHOT):
org.eclipse.persistence.exceptions.ValidationException
Exception Description: Cannot acquire data source [java:comp/env/ds/MeteoDS].
Internal Exception: javax.naming.NamingException: This context must be accessed
throughout a java: URL

I dug through the documentation (what’s with this documentation and what does “No dynamic weaving (instrumentation) – static weaving of entities is still available via EclipseLink” means, why JPAEclipseLinkSessionCustomizer example is broken and finally why use teh deprecated SessionCustomizer?) and found that I have to use a session customizer to make it work in Tomcat. I didn’t like this one bit (it just works in Hibernate) but I followed the instructions and EclipseLink could establish the connection with the Tomcat-configured data source as expected. This was the first duly noted turn-off.
I redeployed and browsed the weather map just to hit another problem:

  Exception Description: Syntax error parsing the query [findStationByPK:
    select station from   Station where id=:stationPK], line 1, column 28:
    syntax error at [where].
 Internal Exception: MismatchedTokenException(65!=66)

Hmm, no compile-time JPA query check. Not good. I checked the JPA query, it was:

  @javax.persistence.NamedQuery(name="findStationByPK",
       query="select station from Station where id=:stationPK"),
...

which looked OK but looking closely I realized that it was not. The correct syntax is (notice the station. indirection:

@javax.persistence.NamedQueries({
	@javax.persistence.NamedQuery(name="findStationByPK",
	        query="select station from Station station where station.id=:stationPK"),
	@javax.persistence.NamedQuery(name="findByStationId",
...

Hibernate was happy with the “non-standard” syntax while EclipseLink is not. No big deal, it wasn’t in many places anyway. I scanned all my queries and fixed them. Redeployed and hit this:

  Exception Description: Syntax error parsing the query [findByStationId: from
    Station station where station.stationId=:stationId], line 1, column 0:
    unexpected token [from].

My JPA query is:

	@javax.persistence.NamedQuery(name="findByStationId",
	        query="from Station station where station.stationId=:stationId")

Again a Hibernate artifact. H3 supports the convenience of using queries that start with “from entity…” if they return entities (single or lists). The fix was simply to add “select station” in front of the JPA QL:

	@javax.persistence.NamedQuery(name="findByStationId",
	        query="select station from Station station where
                station.stationId=:stationId")

Not a biggie either, standard is better and Hibernate can parse this kind of query. Again, rebuilt/redeployed and queried the map again. This time the error was this:

[EL Warning]: 2009-03-21 19:02:42.135--UnitOfWork(2001070194)--Exception [EclipseLink-4002]
(Eclipse Persistence Services -
 1.1.0.r3639-SNAPSHOT): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
You have an error in your SQL syntax; check
the manual that corresponds to your MySQL server version for the right syntax to use
near ':area) , LOCATION) and obs_time
> :minSynopDateToConsiderStr UNION select distin' at line 1
Error Code: 1064
Call: select distinct STATION_FK as STATION_IDENTIFIER, LOCATION,
obs_time as REPORT_TIME  from SYNOP where Contains(GeomFromText(:area) , LOCATION) and
obs_time > :minSynopDateToConsiderStr UNION
select distinct STATION_FK as STATION_IDENTIFIER,
LOCATION, REPORT_TIME as REPORT_TIME from METAR where
Contains(GeomFromText(:area) , LOCATION) and REPORT_TIME > :minMetarDateToConsiderStr

I couldn’t find anything at first in the documentation (getting really pissed at the documentation style, can’t find anything useful) but I finally found this:

Support for the EclipseLink # convention is helpful if you are already familiar with
EclipseLink queries or if you are migrating EclipseLink queries to a JPA application.

I’m not “already familiar with EclipseLink queries” and no, it’s not helpful. So now I have to use #area instead of :area named parameter. This is really smart. Why in the world would one want to change the column with sharp? I’m starting to suspect clumsiness or plain careless coding. I still want to maintain Hibernate/EclipseLink pluggability so I have no choice but creating a static class that holds all NamedNativeQueries and parametrizes the named parameter prefix:

class NativeQueries {
public final static String WEATHER_SNAPSHOT_IN_AREA_NATIVE_QUERY ="select distinct
STATION_FK as STATION_IDENTIFIER, LOCATION, obs_time as REPORT_TIME from SYNOP where
Contains (GeomFromText(@named-param-prefix@area) " +
 ", LOCATION) and obs_time > @named-param-prefix@minSynopDateToConsiderStr UNION " +
"select distinct STATION_FK as STATION_IDENTIFIER, LOCATION, REPORT_TIME
as REPORT_TIME from METAR where Contains(GeomFromText(@named-param-prefix@area) " +
", LOCATION) and REPORT_TIME > @named-param-prefix@minMetarDateToConsiderStr";
...
}

Then I have to use ant to perform a pre-compilation task that replaces @named-param-prefix@ with either : or #. This really got the best of me as I’ve hit a Java compilation quirk too: you have to delete all classes that rely on the NativeQueries class since they will not recompile if the values of the public static final String values change; you have to delete all .class files and force a full compilation for them.
After I did this build change it all started working. Victory at last. Or was it? Well, no; I still have to add query caching as I heavily use spatial queries to fetch area weather (activated every time the map is dragged/zoomed). There’s the @Cache but it’s for entity caching which is good but what I also need is query caching à la Hibernate:

 @org.hibernate.annotations.NamedQuery(name="qName", query="<query>",
    cacheable=true, cacheRegion="some-cache-region")

After another round of thorough search of the documentation I found what I needed but it wasn’t what I expected. EclipseLink doesn’t support annotation-level query cache configuration. Instead you have to code-in (or use AOP to) cache writes after the queries are executed, via its ReadQuery class.
Needless to say I didn’t do it. I gave up on it right there.
On a whole this felt a lot like my (undocumented) evaluation I did over a year ago with OpenJPA (no dice too). Poor documentation (worse than EL’s) and lack of (then) runtime weaving made it a nightmare to deploy. I haven’t tried it ever since.

Prognosis: negative

This is really a shame, EclipseLink seems to be quite fast and has a footprint similar to Hibernate but:
- the documentation is awful. Guys, look at Hibernate and follow their lead. I badly need a single page HTML up-to-date documentation that I can freely browse. I can find almost anything in a matter of seconds using Hibernate’s single page HTML doc and Ctrl-F. I feel so strongly about this, I’d make an addition to the Software Craftsmanship Manifesto: “Maintain top-notch documentation”. Spring and Hibernate projects got it right and this is one of the secrets of their success. Take note.
- Use : (column) for named parameters in NamedNativeQuery; it is consistent with NamedQuery syntax (and hopefully standardized in JPA 2.0).
- Use compile-time Named*Query syntax check. It adds a great deal to predictability.
- Add the Session Customizer fix into of the base platform and make it transparent to the user. Nobody wants to write integration code to fix oddities of popular containers. “Out of the box” is valuable.
- Work on a better query caching support or push hard for a JPA 2.0 beta since there’s standardized second-level caching.

Oh and guys, some documentation pages don’t render properly on Safari, the TOCs overlap the content. Unacceptable.

Until then I’m sticking with Hibernate 3.3. It just works.

Selecting a web hosting provider is a tough job for any web developer that wants to put a Java/JEE web application online. The choice is much simpler when it comes to publishing a PHP web site and there are a load of cheap (and sometimes quite reliable) PHP hosting providers to choose from with LAMP being the de facto standard in the web hosting world. But when it comes to Java hosting providers the picture becomes blurrier. The common thing that all these environments need is a Java container. The most popular choice is Tomcat but there are providers that use Resin, Weblogic or Websphere (the latter two are full fledged JEE containers).
With the raise of lightweight J2EE servers started by the Spring folks, a little revolution began in the Java world: running enterprise-grade JEE webapps without the need for an EJB container; a servlet container is enough. Tomcat should fill the bill for just about any Java web application that doesn’t use EJBs. The advantage that comes from running a servlet container is the smaller footprint compared with an EJB container. This is critical when it comes to selecting hosting environments since you’re paying for resources (especially RAM) that have to be sized to accommodate the memory requirements of the web application.

Self hosting: Home-based dedicated server (near $0)

You can use your home internet connection and a PC where the web container runs and deploys your web application. There is an obvious advantage since the server is physically located close to the development team: complete control and physical access to the machine. The other big plus is the price: you’re already paying for the internet connection and chances are you already have a PC that can be transformed into a server. And chances are that you already have a development environment that can be the basis of the production deployment. Here are the shortcomings: - Bandwidth limitations: the webserver where the webapp lives will upload content to the web browsers that access the application. The upload bandwidth is smaller than the download bandwidth and will result in longer load times for web clients. - Some internet providers cap the internet traffic so you may run beyond the limit.

- You may violate contractual obligations you agreed with your Internet provider by running your own web server.
- The internet connection is not reliable: provider’s downtime directly translates in downtime for your webapp.

Managed hosting ($6 and up)
This is the cheapest solution but it poses the most restrictions. The main limitation is the inability to deploy webapps that depend on Java bytecode manipulation at runtime, mainly Hibernate and Spring. This is due to classloading security constraints that prevent the bytecode manipulation libraries (CGLIB or javassist) to perform runtime code enhancements.
Working with managed hosting environments poses more limitations:
- Deployment: some popular hosting providers deploy once a day: you drop the war file or the exploded war in a specific folder on the hosted environment and it will be deployed at a specific time picked by the provider (more here). If it fails, you have to wait another day till the app will be redeployed or pick a provider that supports instant drop&deploy.
- Debugging a failed deployment: I couldn’t make log4j write the log into a location on my hosted file space so I had to resort to a JDBC appender to actually see what causes a failed deployment. This required two deployments (one to try a location on the hosted folder and a second one to use the JDBC appender; that’s two days due to the deployment constraint detailed above) and the issue turned out to be the classloader security issue preventing CGLIB to change the bytecode. I have only tried to deploy a war file with Hibernate and Spring dependencies in Godaddy’s Java hosting environment (their Linux Deluxe plan) before I gave up on managed Java hosting.
- Sharing the environment with other webapps. Your app will be deployed along with other applications and compete for the same resources.
- Access to server logs and traffic stats is very limited. You can only use Google Analytics (or similar) but not Awstats (at least not for free).
- Outgoing traffic may be restricted or blocked.

Unmanaged hosting solutions
Unmanaged doesn’t necessarily mean that the hosting company doesn’t manage anything, they just leave the actual OS administration in your hands so you’ll have a physical server under your complete control, everything else from network management to hardware maintenance is done by the hosting company. You/your team has to have solid server administration knowledge: Linux, Apache, Java AppServers, software firewall (iptables), backup/restore, monitoring, security, patching, DNS and more. A web administrator inside your team usually takes care of all these details. After the machine is up and running you are free to install and run any Java container and deploy your application.
Below are the options when choosing to run a dedicated server.

Colocated server ($70 and up)

You can colocate the server which means that you buy then ship your own rack-mountable server to a data centre where it will be installed by on-site technicians. The monthly fee you’ll pay starts at around $70 (prices are per one unit which is a slot in the rack) and includes the rack space rental, network connectivity, location security, 24/7 power (not counting mishaps), etc. Denial of Service prevention services are usually provided as part of the package. There is complete freedom in terms of the deployment environment since you own it. However, there are two problems that come with this solution:

- Initial cost associated with buying rack-mountable servers that usually start from $1.5k (check SuperMicro, they’re good). These machines contain redundant components to cope with failures.
- Maintenance: When components break you have to manage their replacement. If the whole machine crashes then your website will be offline until the machine is replaced.

Rent a dedicated server ($100 and up)

Instead of buying your own server, you can choose to rent a dedicated server from the hosting provider. Kevin Rose unveils how digg.com was initially hosted on dedicated servers rented from Ev1Servers.net (see full article). The benefit of this solution is the lower initial cost compared to colocation, beginning at around $100/month and of course the complete control over the environment. The drawbacks are the same as colocation except for servicing the box that falls under the hosting provider’s responsibility.

Virtual Private Server ($20 and up)

A VPS ” is a method of partitioning a physical server computer into multiple servers”. Each of these partitions behaves like a full fledged server, has an IP address and can be reachable from the internet. The computers that are VPS partitioned, sit on the hosting provider’s infrastructure. The provider is responsible for network connectivity, bandwidth and continuity of service in case of physical server failure. There are two main VPS software providers: Virtuozzo and Xen. Of the two, I recommend Xen-based VPS providers since the virtualized machines cannot be oversold and thus you’ll get consistent performance for the money (a detailed explanation here). The main benefit of the VPS is the excellent price/performance ratio starting as low as $20/month. Another big advantage is the ability to “resize” the virtual partition on demand in a matter of minutes (you’ll pay more of course), add more virtual machines to your deployment () as well as “relocating” it to a different data center

if the provider has multiple locations (West Coast and East Coast for instance). Root access is also provided. The drawbacks are:
- the operating systems that you can run are controlled by the hosting provider and usually are Linux distributions (here’s an example). However, the choices are very popular Linux distros.
- you have to have server administration expertise.
I run this blog, spincloud.com and the rest of my web properties on a VPS. I use the excellent Linode after I had switched from Slicehost, having performance issues. I am quite happy with Linode, their service rocks and they’ve been delivering consistent performance since day one.

Cloud services (EC2: $100 and up)

Cloud computing is today’s trend. The company that kick-started the trend was Amazon with their Elastic Computing Cloud. EC2 is actually an advanced form of VPS: you can run your own virtual machine image (or AMI) or pick one from the existing stock. After the AMI is booted and configured, you have the liberty of installing any Java environment. You get root access too. From the high profile websites, Smugmug uses EC2 and S3 and they save money doing so. In fact Smugmug was one of the early adopters of Amazon’s cloud services.

Drawbacks:
- If you reboot the machine or if it simply crashes then all data is lost, including any database records you happen to have. This was a real problem for developers until Amazon added the possibility to mount a persistent S3 volume in an EC2 instance. S3 stands for Simple Storage Service and is Amazon’s offering for cloud-based storage.
- You have to learn new terminology and set of skills that are applicable to Amazon’s problem-set only. You’ll also tie the solution to a single vendor.
The cost for an EC2+S3 starts at close to $100/month.
If you’re thinking of using the dirt-cheap (starts free) Google AppEngine for deploying a Java application you’re out of luck, they only support Python. Update: Google has added Java support to Appengine. at the begining of April. I have a dev account and I’ll evaluate it in a separate post.

Common concerns when selecting a hosting provider
Price
Don’t pick a service just because it’s cheap. Perform some due diligence on the company, where their data centers are located and how good their service is. You don’t want your website hosted on the wrong side of the world. Monitor WebHostingTalk forums; hosting providers often run promotions there.
Look for monthly plans and don’t commit to yearly plans since you may end-up being unhappy with the provider and with no choice to get out of the contract.
Be suspicious when you see associated setup fees coming with setting-up an account. They are not refundable and good hosting providers simply don’t have them.

Uptime
You want no interruption in service for your web servers so uptime is a must. You can’t afford to have downtime since it translates directly into lost customers and ultimately lost business. Look for hosting providers that offer SLAs (Service level agreements) and have proven reliability. Netcraft is a neat tool that measures the reliability of hosting providers on a monthly basis. Look for the providers that guarantee five nines (99.999%) uptime, that is five minutes a year.

Technical support
My personal experience is that I used the tech support while I was using a shared web hosting. I haven’t used any after moving to the VPS.

Transfer volume
Perform some basic calculations for the required traffic based on the expected traffic and data volume sent out. Providers bundle 200GB/month or more in their offerings these days. This should be enough for most of the cases. Make sure your website doesn’t get cut-off if your traffic spikes and/or you’re not paying a ridiculous amount of money per GB if you go beyond your quota.

Security
All hosting providers offer a form of protection against DDOS but you are responsible for the software that runs on the server. I got the online forum I was running for Newsplorer hacked a couple of years ago because I didn’t upgrade the forum software to patch a security vulnerability that was promptly exploited.

Backup
Hosting providers offer backups or you can do it yourself with a bit of effort and some cron knowledge. Save the backed-up files on a different location (I use S3 for my backup needs but there are free solutions too).