Reviewing Google AppEngine for Java (Part 1)

appengine_lowres

When Google announced that Java is the second language that the Appengine will support I almost didn’t believe it given the surge of the new languages and the perception that Java entered legacy but the JVM is a powerful tried-and-true environment and Google saw the potential of using it for what it is bound to

become: a runtime environment for the new and exciting languages (see JRuby and Grails). The JVM is the new gateway drug in the world of languages.

Note: I’ll break down this review into two posts as it’s too extensive to cover everything at once. This first part is about initial setup, JPA and some local webapp deployment issues. In the second part I’ll describe how to load data into the datastore, data indexing, how I reached some of the limitations of the AppEngine and how to get around some of them.

I managed to snatch an evaluation account and last few days I have been playing with it. (as of today Google opened the registrations to all). My goal was (again) simple: to port Spincloud to the AppEngine. The prize: free hosting which amounts to $20/month in my case (see my post about Java hosting providers), some cool monitoring tools that I can access on the web and of course the transparent super-scalability that Google is touting, in case I get techcrunched, slashdotted or whatever (slim chances but still…).
I am a couple of weeks into the migration effort and I can conclude that the current state of the AppEngine is not quite ready for Spincloud as I’ll detail below but it comes quite close. The glaring omission was scheduling but Google announced today that the AppEngine will support it (awesome!) and I plan to use it from day one.

My stated goal is running Spincloud on AppEngine. Here are the technologies I use in Spincloud:
– Spring 3.0 (details here)
– SpringAOP
– SpringSecurity with OpenId
– Webapp management with JMX (I know, it won’t fly)
– Job scheduling using Spring’s Timer Task abstraction (not optimistic about this one either)
– SiteMesh (upgraded to 2.4.2 as indicated)
– Spatial database support including spatial indexing
– JPA 2.0 (back story here)
– Level 2 JPA cache, query cache.
– Native SQL query support including native SQL spatial queries.
– Image processing (spincloud scraps images and processes pixel-level information to get some weather data)

That’s a lot of stuff but first things first. I recommend creating a new repository branch (I use Subversion) dedicated to the new environment as there may be many things to change and you still want to continue the development unaffected on your mainline.

I downloaded the latest SDK and exploded it under the tools/ folder. Make sure you install the Eclipse plugin as it’s pretty useful.

After the initial setup I started-out with the JPA part since this seemed to be the most challenging piece of technology that I had to change. AppEngine uses DataNucleus JPA provider, a poor choice in my opinion since they are a JDO provider turned to JPA (much like KODO/OpenJPA) and so they drag a lot of JDO baggage along. I haven’t heard of them before and they don’t seem to be that popular (why not Hibernate or the new RI: EclipseLink? Update: because JPA is RDBMS oriented while JDO is not and the backing store of GAE is BigTable which is not RDBMS). DataNucleus JPA requires a pre-compilation step that does bytecode enhancement which I completely dislike as it’s archaic (all modern JPA providers don’t have this anymore) but I had no choice. So I’ve followed the reference documentation. The only thing not described there is the ant integration.
Here is what I added to my build.xml

<project name="weather"  default="build"  basedir=".">
...
  <property name="sdk.dir" value="/Users/florin/tools/appengine-java-sdk-1.2.1"/>
  <import file="${sdk.dir}/config/user/ant-macros.xml" />

  <path id="enhancer.classpath">
    <pathelement path="${bin.dir}"/>
    <fileset dir="${sdk.dir}/lib">
      <include name="appengine-tools-api.jar"/>
    </fileset>
    <fileset dir="${lib.dir}">
      <include name="gisjts/jts*.jar"/>
      <include name="appengine/appengine-api-1.0-sdk-1.2.1.jar"/>
    </fileset>
  </path>

  <target name="enhance" depends="compile">
    <enhance verbose="true" classpathref="enhancer.classpath">
      <fileset dir="${bin.dir}">
        <include name="com/newsplore/weather/bo/*.class"/>
      </fileset>
    </enhance>
  </target>

  <target name="build-appengine" depends="compile, enhance, jar"/>
...

Essentially this is to accommodate the enhancement step (notice how the “enhance” target is interposed between compile and jar in the build-appengine target).
I ran the “build-appengine” target and noticed the first exception:

  [enhance] Please see the logs [/tmp/enhance64409.log] for further information.

That’s a convoluted way of looking at the errors (why not showing the exception in the console?). Here’s the actual error found in the said file:

java.lang.RuntimeException: Unexpected exception
...
   Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: java.lang.NoClassDefFoundError: Lorg/apache/log4j/Logger;
  at java.lang.Class.getDeclaredFields0(Native Method)

This was because one of my business objects used log4j. Adding the log4j jar file didn’t help and the solution was to remove the logging from the class altogether.

After cleanup I got this error:

  [enhance] SEVERE: Class "com.newsplore.weather.bo.GeometryUserType" was not found in the
CLASSPATH. Please check your specification and your CLASSPATH.
  [enhance] org.datanucleus.exceptions.ClassNotResolvedException: Class
"com.newsplore.weather.bo.GeometryUserType" was not found in the CLASSPATH.
    Please check your specification and your CLASSPATH.

That was because GeometryUserType (a class I used to handle Geometry types in Hibernate) was in the com.newsplore.weather.bo package and the enhancer went through all classes in this package thinking they are all @Entities and failed with this one. Weird, I thought that the enhancer was smarter and filtered-out all non @Entities… The solution is either to specify all the classes to be enhanced in the “enhance” ant target or move the non-entities to other packages (a good practice). In my case I deleted the class since I wasn’t using it anyway (I used this utility class before Hibernate became a JPA provider).
After this cleanup, the enhance target spat yet another error:

[enhance]Class com.newsplore.weather.bo.UsForecast has property weatherIconByCode declared
    in MetaData, but its getter method doesnt exist in the class!
  [enhance] May 27, 2009 4:03:50 PM org.datanucleus.enhancer.DataNucleusEnhancer main
[enhance] SEVERE: DataNucleus Enhancer completed with an error. Please review the enhancer
log for full details. Some classes may have been enhanced but some caused errors
  [enhance] Class com.newsplore.weather.bo.UsForecast has property weatherIconByCode
declared in MetaData, but its getter method doesnt exist in the class!
  [enhance] org.datanucleus.metadata.InvalidMetaDataException: Class
com.newsplore.weather.bo.UsForecast has property weatherIconByCode declared in MetaData,
 but its getter method doesnt exist in the class!
  [enhance]     at org.datanucleus.metadata.ClassMetaData
.populateMemberMetaData(ClassMetaData.java:497)

So a domain object (UsForecast) has a property called weatherIconByCode. Hmm, I didnt remember of such property but looking at the source I saw that the offender was a private utility method called getWeatherIconByCode(String code) that was being processed by the enhancer as if it was a property (I’m using annotations bound to getters and not to fields). Since adding @Transient didn’t fix it, the solution is to rename the method not to start with get/set (or move it in an utility class which you shouldn’t do if you follow DDD).
To my relief, after this fix the build didn’t yield any more errors so I had a nice webapp folder ready to be deployed. AppEngine has a local sandbox (based on Jetty) that should be used before deploying into the cloud. You can start the appserver using an ant target but I chose to get more control and create a script that accomplishes the same thing. Here it is:

cd build/webapp
java -classpath <appengine-sdk-home>/lib/appengine-tools-api.jar
com.google.appengine.tools.development.DevAppServerMain --port=8080 --address=localhost

Save this in a file called startgae.sh (mind the path, you should be able to cd to the webapp folder from it) then make it executable.
Using this script I fired-up the container and got no startup error. This was weird since I knew I was using JMX (via Spring and annotations) and as well the task scheduler (using Quartz) was seemingly up and running.
I hit the home page to see what has been deployed and I get a 404 error. My web.xml has this mapping:

  <servlet>
    <servlet-name>weather</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
  </servlet>

  <servlet-mapping>
	<servlet-name>weather</servlet-name>
	<url-pattern>/</url-pattern>
  </servlet-mapping>

but this doesn’t work in the local servlet container. This was quite annoying as not only the home page doesn’t work, the static folders (css/, js/, img/) are routed through the weather servlet too. Surprisingly as it turns out, this mapping worked perfectly in the AppEngine environment. To be able to continue with the work in the local environment, I had to define fine grained mappings for all the URLs.
After this change, the first URL that started working was the /faq which provided the first breakthrough. Although only the presentation tier is in use on the /faq page, the fact that the page worked provided a bunch of good news: Spring 3.0 DI container, SiteMesh, Spring Security and SpringAOP were all correctly running.

Stay tuned for the second part detailing more JPA, datastore indexing, a bunch of the AppEngine limitations, a bit of local environment hacking and how to use “request welding” to accomplish long running tasks.

  • http://www.datanucleus.org Andy

    DataNucleus is a *wise* choice. It allows persistence to any type of datastore (LDAP, XML, ODF, Excel, ODBMS, RDBMS, BigTable, Hadoop, JSON), not just the limited type (RDBMS) allowed by Hibernate, EclipseLink, and OpenJPA. Just because *you* haven’t heard of it before isn’t a reason for anything. It has been around since 2003 (called JPOX) and is well enough known. In fact it’s been “very popular” since 2004. Use of JDO makes plenty of sense on this datastore too.

    > DataNucleus JPA requires a pre-compilation step that does bytecode enhancement which I completely dislike as it’s archaic

    Check your facts. DataNucleus requires bytecode enhancement of classes. This is a common feature these days, present in anything AOP. It allows you to do this as part of compilation, as a post-compile step, or at runtime. Ant integration is in the our docs, as is the Maven integration, and the Eclipse integration, and the runtime enhancement.

    As ever, the DN forum would be a more suitable place if you have comments about it as a product, since its developers aren’t going to come across your blog as a matter of course.

  • http://blog.newsplore.com/?fbconnect_action=myhome&userid=2 Florin

    Transparent bytecode enhancement is a common feature, explicit enhancement is not. With respect to AOP, if you refer to AspectJ, there’s an option to do runtime bytecode enhancement as well as through a static precompilation step. SpringAOP doesn’t require static bytecode enhancement (it uses dynamic proxying via either JDK proxies or CGLIB). But to *mandate* a precompilation step is a restriction of GAE+DN as it complicates the build and deployment.
    It’s true, DN supports non-RDBMS stores and I’ve updated the post to reflect that. It’s great to hear that it’s been around from 2003 and it became “very popular”. Somewhere along the way Hibernate showed us all how ORM should be done and it become so “very popular” that it’s been adopted as a standard (after already being the de facto standard).

  • http://www.datanucleus.org Andy

    “Transparent bytecode enhancement” …. you mean what I already wrote “runtime enhancement”, that DN already supports. OpenJPA supports the same post-compile and runtime options. Don’t see you complaining about that. Heck EclipseLink also does.

  • Matt

    Hello – I like your blog but cannot figure out how to subscribe to your feed. Do you have a feed url? -Thanks

  • http://blog.newsplore.com/?fbconnect_action=myhome&userid=2 Florin

    Hi Matt,
    Here’s the RSS feed URL: http://feeds.feedburner.com/EverythingBeta
    I added an RSS icon onthe top-right corner of the blog for better reach.

  • http://www.astwoodconsulting.co.uk/ excelmacrotraining

    Somewhere along the way Hibernate showed us all how ORM should be done and it become so  very popular Thanks to share this blog.I really appreciate.

  • Jonathan Elano

    Even without their discounts http://www.jvmhost.com seems to be the cheapest Java provider on the market.

  • http://www.midlandpropertylawyers.co.uk conveyancing birmingham

    This is a nice content.I like this content.This is a nice content.I appreciate to this content.The written skill is so good.Thanks to share this blog.Keep sharing. 

  • Adrian

    Is there any other fix for getWeatherIconByCode(String code) than renaming the method not to start with get/set. How do you implement then readonly calculated properties?