Reviewing Google AppEngine for Java (Part 1)

appengine_lowres

When Google announced that Java is the second language that the Appengine will support I almost didn’t believe it given the surge of the new languages and the perception that Java entered legacy but the JVM is a powerful tried-and-true environment and Google saw the potential of using it for what it is bound to

become: a runtime environment for the new and exciting languages (see JRuby and Grails). The JVM is the new gateway drug in the world of languages.

Note: I’ll break down this review into two posts as it’s too extensive to cover everything at once. This first part is about initial setup, JPA and some local webapp deployment issues. In the second part I’ll describe how to load data into the datastore, data indexing, how I reached some of the limitations of the AppEngine and how to get around some of them.

I managed to snatch an evaluation account and last few days I have been playing with it. (as of today Google opened the registrations to all). My goal was (again) simple: to port Spincloud to the AppEngine. The prize: free hosting which amounts to $20/month in my case (see my post about Java hosting providers), some cool monitoring tools that I can access on the web and of course the transparent super-scalability that Google is touting, in case I get techcrunched, slashdotted or whatever (slim chances but still…).
I am a couple of weeks into the migration effort and I can conclude that the current state of the AppEngine is not quite ready for Spincloud as I’ll detail below but it comes quite close. The glaring omission was scheduling but Google announced today that the AppEngine will support it (awesome!) and I plan to use it from day one.

My stated goal is running Spincloud on AppEngine. Here are the technologies I use in Spincloud:
– Spring 3.0 (details here)
– SpringAOP
– SpringSecurity with OpenId
– Webapp management with JMX (I know, it won’t fly)
– Job scheduling using Spring’s Timer Task abstraction (not optimistic about this one either)
– SiteMesh (upgraded to 2.4.2 as indicated)
– Spatial database support including spatial indexing
– JPA 2.0 (back story here)
– Level 2 JPA cache, query cache.
– Native SQL query support including native SQL spatial queries.
– Image processing (spincloud scraps images and processes pixel-level information to get some weather data)
Continue reading “Reviewing Google AppEngine for Java (Part 1)”

Choosing a Java hosting provider

TL;DR: Go for a VPS. I now use Digitalocean, they’re awesome.

Selecting a web hosting provider is a tough job for any web developer that wants to put a Java/JEE web application online. The choice is much simpler when it comes to publishing a PHP web site and there are a load of cheap (and sometimes quite reliable) PHP hosting providers to choose from with LAMP being the de facto standard in the web hosting world. But when it comes to Java hosting providers the picture becomes blurrier. The common thing that all these environments need is a Java container. The most popular choice is Tomcat but there are providers that use Resin, Weblogic or Websphere (the latter two are full fledged JEE containers).
With the raise of lightweight J2EE servers started by the Spring folks, a little revolution began in the Java world: running enterprise-grade JEE webapps without the need for an EJB container; a servlet container is enough. Tomcat should fill the bill for just about any Java web application that doesn’t use EJBs. The advantage that comes from running a servlet container is the smaller footprint compared with an EJB container. This is critical when it comes to selecting hosting environments since you’re paying for resources (especially RAM) that have to be sized to accommodate the memory requirements of the web application.

Self hosting: Home-based dedicated server (near $0)
homepcYou can use your home internet connection and a PC where the web container runs and deploys your web application. There is an obvious advantage since the server is physically located close to the development team: complete control and physical access to the machine. The other big plus is the price: you’re already paying for the internet connection and chances are you already have a PC that can be transformed into a server. And chances are that you already have a development environment that can be the basis of the production deployment. Here are the shortcomings:- Bandwidth limitations: the webserver where the webapp lives will upload content to the web browsers that access the application. The upload bandwidth is smaller than the download bandwidth and will result in longer load times for web clients.
– Some internet providers cap the internet traffic so you may run beyond the limit.
– You may violate contractual obligations you agreed with your Internet provider by running your own web server.
– The internet connection is not reliable: provider’s downtime directly translates in downtime for your webapp.
Continue reading “Choosing a Java hosting provider”

Spincloud Labs: Political boundaries overlay in Google maps (Part 2)

Update Sep.21.2011: I took the code in the two parts and made a github project off of it called Gborders. The code is simpler and there are more options to generate the borders overlay based on geographic regions. Happy forks!

In Part 1 we imported world political borders into a database table. In this second part we’ll use the table and generate a script that will be used to add the borders overlay to Google Maps. We’ll use the cool Ruby and some fancy GIS words along the way.

We left-off with a database table containing all borders. The goal today is to produce a Javascript file that will be used for overlaying polygons representing countries, over a map using the Google Maps API.

Let’s examine the table data first (I’m using mysql through the command line):

mysql> desc world_boundaries;
+-----------+--------------+------+-----+---------+-------+
| Field     | Type         | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| ID        | int(11)      | NO   | PRI | NULL    |       |
| FIPS      | varchar(255) | YES  |     | NULL    |       |
| ISO2      | varchar(255) | YES  |     | NULL    |       |
| ISO3      | varchar(255) | YES  |     | NULL    |       |
| UN        | int(11)      | YES  |     | NULL    |       |
| NAME      | varchar(255) | YES  |     | NULL    |       |
| AREA      | int(11)      | YES  |     | NULL    |       |
| POP2005   | bigint(20)   | YES  |     | NULL    |       |
| REGION    | int(11)      | YES  |     | NULL    |       |
| SUBREGION | int(11)      | YES  |     | NULL    |       |
| LON       | double       | YES  |     | NULL    |       |
| LAT       | double       | YES  |     | NULL    |       |
| ogc_geom  | geometry     | YES  |     | NULL    |       |
+-----------+--------------+------+-----+---------+-------+

A lot of information here but we’ll need just these fields: name (country name), iso2 (two letter country codes) the_geom, iso2 (border geometry) and region (grouping countries by regions). To check the data-set let’s examine a small country. I’ll pick the tiny State of Vatican for its size:

mysql> select iso2, AsText(ogc_geom), region from world_boundaries where iso2='VA';
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
| iso2 | AsText(ogc_geom)                                                                                                                                  | region |
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
| VA   | MULTIPOLYGON(((12.445090330889 41.903117521785,12.451653339581 41.907989033391,12.456660170954 41.901426024699,12.445090330889 41.903117521785))) |    150 |
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
1 row in set (0.00 sec)

The only surprise here is that the country border is described as a MULTIPLOYGON spatial type which describe a collection of polygons that don’t interset. This is in order to accommodate countries that have islands under ownership.

Let’s see how these points look on the map. We’ll use the excellent polygon encoder utility written by Mark McClure. Copy Vatican’s point set in the “Input Text” input box (choose lng/lat option):

12.445090330889, 41.903117521785
12.451653339581, 41.907989033391
12.456660170954, 41.901426024699
12.445090330889, 41.903117521785

Here’s the result:
va_polygon
Not very accurate but we’ll have to live with it, it’s a free data set after all…
Continue reading “Spincloud Labs: Political boundaries overlay in Google maps (Part 2)”

Spincloud Labs: Political boundaries overlay in Google maps (Part 1)

Update Sep.21.2011: I took the code in this post and part deux and made a github project off of it called Gborders. The code is simpler and there are more options to generate the borders overlay based on geographic regions. Happy forks!

One thing I needed when designing the Meteoalarm mashup for Spincloud were the political boundaries for all European countries. With them at hand, I would use the polygon overlay from the Coogle API and fill the country polygons with the respective weather warning colors.

This first part is a tutorial on how to import world political borders into a MySQL database table.
The second part in these series will use this table to create a script that will add the country borders overlay to Google Maps.

So after learning that Google Maps API doesn’t give programatic access to the political boundaries I moved on to doing it myself. I read about the encoded polylines and figured that I only missed the country boundaries as encoded polygons so I started searching for readily-available resources. Sure enough there were nowhere to be found.

The next best thing was to find the country boundaries in some GIS format and generate the polyligons myself. The information seems to be sparse but after a bit of research I found was the CIA World Databank but the format is incomplete, it only defines the polygons with no country names or any other administrative hook that I would be able to use. I then spent hours trying to find a better resource until I remembered that Mapping Hacks (an excellent book by the way) has data set for the examples in the book. I had the first breakthrough, the website makes available updated world borders files here, they are mirrored from thematicmapping.org.

We’ll use a utility called shp2mysql to export the boundaries to SQL. I’m on a MacBook and no shp2mysql binaries for my platform were available in the original package so we’ll have to compile it first then run it to be able to export the shapefile to SQL.

Below are detailed all steps needed to to produce the borders SQL file on Leopard 10.5.x (note that you have to have the Developer Tools installed). In short, copy and run the following script on your mac in a file called process_boundaries.sh. The script creates a file called wb_dump.sql which is ready to be imported into database:
Continue reading “Spincloud Labs: Political boundaries overlay in Google maps (Part 1)”

Spincloud now integrates with Europe’s weather warning service Meteoalarm

Spincloud + Meteoalarm
A short month after the initial public release of Spincloud, I am pleased to announce the addition of a new feature: integration with the European weather warning service Meteoalarm which provides the most relevant information needed to prepare for extreme weather, expected to occur over Europe.  The colors indicate the severity of the danger and its possible impact: Red: very dangerous, Orange: dangerous, Yelllow: potentially dangerous, Green: no warnings.

Currently only the country-level warnings are displayed. You can access the new feature by clicking the “Meteoalarm” button located on the top-left corner of the map. Note that the map has to be positioned over Europe for this buttton to be enabled.
More information about Meteoalarm here.

Important note: When using this new feature, the best experience is with Google’s Chrome, Mozilla or Safari browsers. Enabling the Meteoalarm overlay is much slower on IE browser which sometimes results in the browser reporting a slow-running script on slower hardware. Wait a few more seconds, the new overlay will render eventually.

@BreakingNewsOn map

I’m a bit of a news junkie and when I heard that a twitter stream called @BreakingNewsOn existed I did the natural geekish thing, mash it up. Yahoo pipes was my friend here. I haven’t used the Pipes before but that didn’t quite matter, Yahoo did a great job keeping this playgroundsy product in the child realm so even a caveman could do it. The mashup result -two pipe connections later- is beyond the link:

http://pipes.yahoo.com/newsplore/breakingnewson

Gmail themes, skin deep

Today was my lucky day according to least one popular blog. I got the Themes tab enabled in my gmail account, wohoo! I only wish they could change the font sizes too, I’m up for a microscopic look’n feel, really small and cozy-cute.

My current favorite is the Shiny theme:

Now, if only googlers would add a Cover Flow option instead of that dull message list. Oh, and it should come gratis as usual.

I learned something today too. To take a screenshot on a MacBook, press Shift-Command-4 then select the portion of the screen you want to capture, then find the pic saved conveniently on your Desktop in png format. The karate stance is only required if you want to execute the entire screen capture kata.

[my gmail inbox via Techcrunch]

My very first mac: one with the machine

I just bought my first MacBook yesterday. My current laptop (a beaten down Toshiba Satellite) was long overdue and I realized that I didn’t want yet another boring Win notebook, it just doesn’t do it for me anymore. I’m in front of a PC every day at
work and I surely don’t want to see the same things while at home. And so I was determined to buy a Mac. My first Mac. I didn’t know that much about Apple and still don’t (imagine that)…
I waited until the new MacBooks were announced (no rush, I don’t know much about these machines anyway) and I started to do my research. A $1500 laptop seems a reasonable price for a good laptop and I had two requirements: not that big and not that heavy. MBPs seemed too expensive so my choices were the 2.0 or 2.4 MacBook. I eventually choose the 2.4 version mostly because of the backlit keyboard.

Continue reading “My very first mac: one with the machine”