In my constant search for free weather data for Spincloud, a short while ago I have found a gem: free forecast data offered by the progressive Norwegian Meteorologic Institute. The long range forecast coverage is fairly thorough and covers most more than 2700 locations worldwide. I am happy to announce that I have extended spincloud.com to include it.

The data is refreshed every hour and the forecast range is available for the next seven days. You can bookmark any location or subscribe to weather reports via RSS.
On a different but related note, I can only be thankful for the free data offered by various meteorological organizations that allows Spincloud to exist and I believe in freeing public data (weather related and otherwise) as it belongs to the public that finances government and inter-government agencies in the first place. The Norwegian Met Institute is a great example for freeing its data and I am saluting them for making the right decision.
Now if only all such progressive Meteorological institutes around the world would agree on a common format for disseminating their data, it would make developers like me a tad happier…

It took a while since the previous feature update to Spincloud. I have done a number of upgrades to the underlying tech and some intensive code refactoring but nothing visible. The time has come for another eye candy: heat maps. It is a map overlay that shows a color-translated temperature layer based on interpolated values of the current weather conditions. It gives a quick indication of the average temperature across all land masses where data is available.

Naturally in areas where data is more dense, the visual representation is better. The toughest job was to figure-out the interpolation math. I still have to refine the algorithm currently based on the inverse distance weighting algorithm which interpolates scattered points on a surface. Apparently a better one to use is kriging but I have yet to implement it.
The temperature map is generated from current temperatures and is updated once every hour. You can toggle the overlay by clicking the “Temp” button found on the top of the active map area.

There are three forces that shape a project: domain, process and technology. Add the “-driven” suffix to any of them and you’ll perhaps recognize some of the methods used in projects you’ve been involved in and yet, as soon as one takes too much of a lead against the other two, failure will follow almost inevitably. At the intersection of these three forces we find familiar terms and concepts but first a word or two about each of them:

Technology
Back in the days of the tech bubble, tech was allmighty. Buzzwords like Java, EJBs, PHP defined entire projects. It was the time where software became accessible to a much larger audience. The new wave of enthusiastic geeks embraced everything from new languages to professional certifications to the then-nascent open source. I admit having my share of technology-driven projects back in the day…

Process
Process brings structure and pace to a project. The two complementary components of a project process are methodology and integration. We are all too familiar with methodology: waterfall, RUP or agile methods of software development are vastly documented and practiced; integration largely is defect management, testing, build, deployment and documentation. Today they all come together in what is called continuous integration where all these concepts become interrelated in a repetitive process that produces accountability and visibility into the progress of a project. Process is also the one force that tends to disappear first after a project is finished.

Domain
The domain captures entities and business logic, is driven by the business requirements and is in no way influenced by the two other forces. The most successful way to employ the domain is through Domain Driven Design where the main focus of software development is neither technology nor process but the business requirements.

There are times in the life cycle of a project where these forces have to be kept separate but most of the time they shape each other. At their confluence there are several artifacts that are paramount to a project’s success with code being at their core.

Code
Although each force is important during a project life-cycle, they all work towards producing code. No matter how many UML diagrams, software architecture documents or burn down charts are produced the most valuable and tangible value that a project creates is runnable code.

APIs
Application Programming Interfaces represent the external view of runnable software products. APIs state the intent of the software. They should be cared-for and refined during a project lifecycle, until they become the best representation of the domain they are modeling. APIs also serve for integration purposes and so they should be extended to blend into the domain of the consumers as needed.

Features
Features are defined within a project scope. They can be fully defined prior to coding in a waterfall process or, defined and/or shaped during the project development in an Agile setup. Features specify a system’s requirements (they are called user stories in an agile project).

Test, Build, Deploy, Run
Projects have to be monitored since inception to their natural end. Just like a child’s development stages, projects need to touch milestones, assess velocity and health. They do so by constant testing, building, deploying and running.

Roles
Project roles that drive each of these forces can also be mapped in the above figure. Take the Agile roles for instance:

A fine balance
Imagine any of the forces represented by circles in the picture becoming disproportionally larger than the others. As a result, one or more of the artifacts at the intersection of the forces will become disproportionally smaller than the others upsetting the equilibrium. Whether is less features, less APIs or less testing, the end result is still unbalanced.

As always, people (roles) will drive the forces. Balancing them requires skill from all parts involved and the success or failure of a project depends on giving enough power to each not to overturn the balance. If ancient romans could do it two millennia ago, so can we.

When Google announced that Java is the second language that the Appengine will support I almost didn’t believe it given the surge of the new languages and the perception that Java entered legacy but the JVM is a powerful tried-and-true environment and Google saw the potential of using it for what it is bound to

become: a runtime environment for the new and exciting languages (see JRuby and Grails). The JVM is the new gateway drug in the world of languages.

Note: I’ll break down this review into two posts as it’s too extensive to cover everything at once. This first part is about initial setup, JPA and some local webapp deployment issues. In the second part I’ll describe how to load data into the datastore, data indexing, how I reached some of the limitations of the AppEngine and how to get around some of them.

I managed to snatch an evaluation account and last few days I have been playing with it. (as of today Google opened the registrations to all). My goal was (again) simple: to port Spincloud to the AppEngine. The prize: free hosting which amounts to $20/month in my case (see my post about Java hosting providers), some cool monitoring tools that I can access on the web and of course the transparent super-scalability that Google is touting, in case I get techcrunched, slashdotted or whatever (slim chances but still…).
I am a couple of weeks into the migration effort and I can conclude that the current state of the AppEngine is not quite ready for Spincloud as I’ll detail below but it comes quite close. The glaring omission was scheduling but Google announced today that the AppEngine will support it (awesome!) and I plan to use it from day one.

My stated goal is running Spincloud on AppEngine. Here are the technologies I use in Spincloud:
- Spring 3.0 (details here)
- SpringAOP
- SpringSecurity with OpenId
- Webapp management with JMX (I know, it won’t fly)
- Job scheduling using Spring’s Timer Task abstraction (not optimistic about this one either)
- SiteMesh (upgraded to 2.4.2 as indicated)
- Spatial database support including spatial indexing
- JPA 2.0 (back story here)
- Level 2 JPA cache, query cache.
- Native SQL query support including native SQL spatial queries.
- Image processing (spincloud scraps images and processes pixel-level information to get some weather data)

That’s a lot of stuff but first things first. I recommend creating a new repository branch (I use Subversion) dedicated to the new environment as there may be many things to change and you still want to continue the development unaffected on your mainline.

I downloaded the latest SDK and exploded it under the tools/ folder. Make sure you install the Eclipse plugin as it’s pretty useful.

After the initial setup I started-out with the JPA part since this seemed to be the most challenging piece of technology that I had to change. AppEngine uses DataNucleus JPA provider, a poor choice in my opinion since they are a JDO provider turned to JPA (much like KODO/OpenJPA) and so they drag a lot of JDO baggage along. I haven’t heard of them before and they don’t seem to be that popular (why not Hibernate or the new RI: EclipseLink? Update: because JPA is RDBMS oriented while JDO is not and the backing store of GAE is BigTable which is not RDBMS). DataNucleus JPA requires a pre-compilation step that does bytecode enhancement which I completely dislike as it’s archaic (all modern JPA providers don’t have this anymore) but I had no choice. So I’ve followed the reference documentation. The only thing not described there is the ant integration.
Here is what I added to my build.xml

<project name="weather"  default="build"  basedir=".">
...
  <property name="sdk.dir" value="/Users/florin/tools/appengine-java-sdk-1.2.1"/>
  <import file="${sdk.dir}/config/user/ant-macros.xml" />

  <path id="enhancer.classpath">
    <pathelement path="${bin.dir}"/>
    <fileset dir="${sdk.dir}/lib">
      <include name="appengine-tools-api.jar"/>
    </fileset>
    <fileset dir="${lib.dir}">
      <include name="gisjts/jts*.jar"/>
      <include name="appengine/appengine-api-1.0-sdk-1.2.1.jar"/>
    </fileset>
  </path>

  <target name="enhance" depends="compile">
    <enhance verbose="true" classpathref="enhancer.classpath">
      <fileset dir="${bin.dir}">
        <include name="com/newsplore/weather/bo/*.class"/>
      </fileset>
    </enhance>
  </target>

  <target name="build-appengine" depends="compile, enhance, jar"/>
...

Essentially this is to accommodate the enhancement step (notice how the “enhance” target is interposed between compile and jar in the build-appengine target).
I ran the “build-appengine” target and noticed the first exception:

  [enhance] Please see the logs [/tmp/enhance64409.log] for further information.

That’s a convoluted way of looking at the errors (why not showing the exception in the console?). Here’s the actual error found in the said file:

java.lang.RuntimeException: Unexpected exception
...
   Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: java.lang.NoClassDefFoundError: Lorg/apache/log4j/Logger;
  at java.lang.Class.getDeclaredFields0(Native Method)

This was because one of my business objects used log4j. Adding the log4j jar file didn’t help and the solution was to remove the logging from the class altogether.

After cleanup I got this error:

  [enhance] SEVERE: Class "com.newsplore.weather.bo.GeometryUserType" was not found in the
CLASSPATH. Please check your specification and your CLASSPATH.
  [enhance] org.datanucleus.exceptions.ClassNotResolvedException: Class
"com.newsplore.weather.bo.GeometryUserType" was not found in the CLASSPATH.
    Please check your specification and your CLASSPATH.

That was because GeometryUserType (a class I used to handle Geometry types in Hibernate) was in the com.newsplore.weather.bo package and the enhancer went through all classes in this package thinking they are all @Entities and failed with this one. Weird, I thought that the enhancer was smarter and filtered-out all non @Entities… The solution is either to specify all the classes to be enhanced in the “enhance” ant target or move the non-entities to other packages (a good practice). In my case I deleted the class since I wasn’t using it anyway (I used this utility class before Hibernate became a JPA provider).
After this cleanup, the enhance target spat yet another error:

[enhance]Class com.newsplore.weather.bo.UsForecast has property weatherIconByCode declared
    in MetaData, but its getter method doesnt exist in the class!
  [enhance] May 27, 2009 4:03:50 PM org.datanucleus.enhancer.DataNucleusEnhancer main
[enhance] SEVERE: DataNucleus Enhancer completed with an error. Please review the enhancer
log for full details. Some classes may have been enhanced but some caused errors
  [enhance] Class com.newsplore.weather.bo.UsForecast has property weatherIconByCode
declared in MetaData, but its getter method doesnt exist in the class!
  [enhance] org.datanucleus.metadata.InvalidMetaDataException: Class
com.newsplore.weather.bo.UsForecast has property weatherIconByCode declared in MetaData,
 but its getter method doesnt exist in the class!
  [enhance]     at org.datanucleus.metadata.ClassMetaData
.populateMemberMetaData(ClassMetaData.java:497)

So a domain object (UsForecast) has a property called weatherIconByCode. Hmm, I didnt remember of such property but looking at the source I saw that the offender was a private utility method called getWeatherIconByCode(String code) that was being processed by the enhancer as if it was a property (I’m using annotations bound to getters and not to fields). Since adding @Transient didn’t fix it, the solution is to rename the method not to start with get/set (or move it in an utility class which you shouldn’t do if you follow DDD).
To my relief, after this fix the build didn’t yield any more errors so I had a nice webapp folder ready to be deployed. AppEngine has a local sandbox (based on Jetty) that should be used before deploying into the cloud. You can start the appserver using an ant target but I chose to get more control and create a script that accomplishes the same thing. Here it is:

cd build/webapp
java -classpath <appengine-sdk-home>/lib/appengine-tools-api.jar
com.google.appengine.tools.development.DevAppServerMain --port=8080 --address=localhost

Save this in a file called startgae.sh (mind the path, you should be able to cd to the webapp folder from it) then make it executable.
Using this script I fired-up the container and got no startup error. This was weird since I knew I was using JMX (via Spring and annotations) and as well the task scheduler (using Quartz) was seemingly up and running.
I hit the home page to see what has been deployed and I get a 404 error. My web.xml has this mapping:

  <servlet>
    <servlet-name>weather</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
  </servlet>

  <servlet-mapping>
	<servlet-name>weather</servlet-name>
	<url-pattern>/</url-pattern>
  </servlet-mapping>

but this doesn’t work in the local servlet container. This was quite annoying as not only the home page doesn’t work, the static folders (css/, js/, img/) are routed through the weather servlet too. Surprisingly as it turns out, this mapping worked perfectly in the AppEngine environment. To be able to continue with the work in the local environment, I had to define fine grained mappings for all the URLs.
After this change, the first URL that started working was the /faq which provided the first breakthrough. Although only the presentation tier is in use on the /faq page, the fact that the page worked provided a bunch of good news: Spring 3.0 DI container, SiteMesh, Spring Security and SpringAOP were all correctly running.

Stay tuned for the second part detailing more JPA, datastore indexing, a bunch of the AppEngine limitations, a bit of local environment hacking and how to use “request welding” to accomplish long running tasks.

I launched Spincloud on New Year’s day. I didn’t expect a flood of traffic to hit the site since -let’s face it- there was nothing groundbreaking. Spincloud was born from my idea of having access to the world weather in one step or less and I’m pleased with the results so far. Spincloud gains adoption every day and this is only due to the people that saw something new and exciting in it. There are blog posts, traffic graphs and other goodies to talk about so let’s begin.

I announced spincloud’s launch on my personal blog then Keir from googlemapsmania broke the news to the community. And so the first spike of traffic was logged on Jan 2nd. 

Then followed a month of relative silence during which I took a much needed break from the project. I couldn’t help but refactoring and cleaning-up the code a bit. I had two minor releases deployed in prod in the mean time and everything went smooth.

I had a new feature I wanted to have online before the initial release, the meteoalarm mashup that displays weather warnings across Europe but chose not to delay it (a Good Decision in hindsight) . I started working on it by mid January , did a lot of digging for administrative borders data then getting the meteoalarm warning data right (those guys don’t have any integration point). I was done two weeks later.

In the mean time I was trying to raise the awareness about spincloud. I submitted the site so some mapping/mashup themed websites, improved the indexing (still working on it). Then Wednesday last week the European bloggers finally found it. UK’s Mapperz ran a nice little story that seems to have set the things in motion. A number of other bloggers picked-up the story and a surprisingly good feedback came from Italy: Matteo and italiasw.com did a great job while  gisdk.blogspot.com of Denmark  noticed the RSS facilities. Then the referral machine kicked in,  killerstartups.com, programmableweb.com amongst the sites linking. And thus the second spike registered on the traffic meter:

This one was three times bigger than the first spike and it was yet again all thanks to the blogger community that saw something useful in spincloud.

Things are looking up.  There are a few sites in Asia following spincloud: drchoi.or.kr and wangtam.com and I see that 16 people linked to spincloud on delicious. Yey!

The traffic tripled on an average from last month and I’m delighted. The server hums happily for now and I have all reasons to keep it that way.

Huzzah!

	=

Update Nov.25.2009: The twitter name has changed to @meteo_alarm. Please update your bookmark or follow list.

Between code refactoring and cooking new ideas, I took some time to add a new feature to Spincloud. Inspired by the crowd-sourced weather updates triggered by really bad snow storms in the UK, I (finally) started to see the advantages to using Twitter. So I took the Meteoalarm data that I mashed-up on Spincloud’s map and streamed it to Twitter.

So here it is: Meteoalarm on Twitter via Spincloud = @Meteoalarm @Meteo_alarm. The stream is updated twice a day and it presents a concise text-only representation of the color encoded weather warnings  in Europe for today and tomorrow.

I started working on Spincloud back in 2003 when I got interested in regular expressions. I liked them so much I started looking for large sets of freely available data that I could crunch. Weather data was quite appealing; not only I’d parse it but I could get instant feedback by just looking at the parsed data, after all we understand temperatures and wind speeds and the like. I’d get to use other technologies I couldn’t experiment at work where I was doing the regular J2EE stuff and I kept feeling the itch to do new things.

And so Spincloud was born; it’s mission: to track the weather worldwide. It didn’t have a name back then but as things started to shape-up, I realized that the experiment was slowly turning into a real project. Then Google published its Maps API and I just knew that my small little pet project just found the perfect playground. It’s been a while since I started with this and I had no intention of publishing it initially. It has grown beyond its initial purpose and I feel it’s time to set it free.

This is the third version of Spincloud, the first two iterations were not publicly released. This is not an open source project; however, I’m planning on blogging about the technical challenges I’m encountering and are interesting enough to share.

Here are the main features of Spincloud:

Update: googlemapsmania.com just ran a story about Spincloud here. Thanks guys!