When Google announced that Java is the second language that the Appengine will support I almost didn’t believe it given the surge of the new languages and the perception that Java entered legacy but the JVM is a powerful tried-and-true environment and Google saw the potential of using it for what it is bound to

become: a runtime environment for the new and exciting languages (see JRuby and Grails). The JVM is the new gateway drug in the world of languages.

Note: I’ll break down this review into two posts as it’s too extensive to cover everything at once. This first part is about initial setup, JPA and some local webapp deployment issues. In the second part I’ll describe how to load data into the datastore, data indexing, how I reached some of the limitations of the AppEngine and how to get around some of them.

I managed to snatch an evaluation account and last few days I have been playing with it. (as of today Google opened the registrations to all). My goal was (again) simple: to port Spincloud to the AppEngine. The prize: free hosting which amounts to $20/month in my case (see my post about Java hosting providers), some cool monitoring tools that I can access on the web and of course the transparent super-scalability that Google is touting, in case I get techcrunched, slashdotted or whatever (slim chances but still…).
I am a couple of weeks into the migration effort and I can conclude that the current state of the AppEngine is not quite ready for Spincloud as I’ll detail below but it comes quite close. The glaring omission was scheduling but Google announced today that the AppEngine will support it (awesome!) and I plan to use it from day one.

My stated goal is running Spincloud on AppEngine. Here are the technologies I use in Spincloud:
- Spring 3.0 (details here)
- SpringAOP
- SpringSecurity with OpenId
- Webapp management with JMX (I know, it won’t fly)
- Job scheduling using Spring’s Timer Task abstraction (not optimistic about this one either)
- SiteMesh (upgraded to 2.4.2 as indicated)
- Spatial database support including spatial indexing
- JPA 2.0 (back story here)
- Level 2 JPA cache, query cache.
- Native SQL query support including native SQL spatial queries.
- Image processing (spincloud scraps images and processes pixel-level information to get some weather data)

That’s a lot of stuff but first things first. I recommend creating a new repository branch (I use Subversion) dedicated to the new environment as there may be many things to change and you still want to continue the development unaffected on your mainline.

I downloaded the latest SDK and exploded it under the tools/ folder. Make sure you install the Eclipse plugin as it’s pretty useful.

After the initial setup I started-out with the JPA part since this seemed to be the most challenging piece of technology that I had to change. AppEngine uses DataNucleus JPA provider, a poor choice in my opinion since they are a JDO provider turned to JPA (much like KODO/OpenJPA) and so they drag a lot of JDO baggage along. I haven’t heard of them before and they don’t seem to be that popular (why not Hibernate or the new RI: EclipseLink? Update: because JPA is RDBMS oriented while JDO is not and the backing store of GAE is BigTable which is not RDBMS). DataNucleus JPA requires a pre-compilation step that does bytecode enhancement which I completely dislike as it’s archaic (all modern JPA providers don’t have this anymore) but I had no choice. So I’ve followed the reference documentation. The only thing not described there is the ant integration.
Here is what I added to my build.xml

<project name="weather"  default="build"  basedir=".">
...
  <property name="sdk.dir" value="/Users/florin/tools/appengine-java-sdk-1.2.1"/>
  <import file="${sdk.dir}/config/user/ant-macros.xml" />

  <path id="enhancer.classpath">
    <pathelement path="${bin.dir}"/>
    <fileset dir="${sdk.dir}/lib">
      <include name="appengine-tools-api.jar"/>
    </fileset>
    <fileset dir="${lib.dir}">
      <include name="gisjts/jts*.jar"/>
      <include name="appengine/appengine-api-1.0-sdk-1.2.1.jar"/>
    </fileset>
  </path>

  <target name="enhance" depends="compile">
    <enhance verbose="true" classpathref="enhancer.classpath">
      <fileset dir="${bin.dir}">
        <include name="com/newsplore/weather/bo/*.class"/>
      </fileset>
    </enhance>
  </target>

  <target name="build-appengine" depends="compile, enhance, jar"/>
...

Essentially this is to accommodate the enhancement step (notice how the “enhance” target is interposed between compile and jar in the build-appengine target).
I ran the “build-appengine” target and noticed the first exception:

  [enhance] Please see the logs [/tmp/enhance64409.log] for further information.

That’s a convoluted way of looking at the errors (why not showing the exception in the console?). Here’s the actual error found in the said file:

java.lang.RuntimeException: Unexpected exception
...
   Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: java.lang.NoClassDefFoundError: Lorg/apache/log4j/Logger;
  at java.lang.Class.getDeclaredFields0(Native Method)

This was because one of my business objects used log4j. Adding the log4j jar file didn’t help and the solution was to remove the logging from the class altogether.

After cleanup I got this error:

  [enhance] SEVERE: Class "com.newsplore.weather.bo.GeometryUserType" was not found in the
CLASSPATH. Please check your specification and your CLASSPATH.
  [enhance] org.datanucleus.exceptions.ClassNotResolvedException: Class
"com.newsplore.weather.bo.GeometryUserType" was not found in the CLASSPATH.
    Please check your specification and your CLASSPATH.

That was because GeometryUserType (a class I used to handle Geometry types in Hibernate) was in the com.newsplore.weather.bo package and the enhancer went through all classes in this package thinking they are all @Entities and failed with this one. Weird, I thought that the enhancer was smarter and filtered-out all non @Entities… The solution is either to specify all the classes to be enhanced in the “enhance” ant target or move the non-entities to other packages (a good practice). In my case I deleted the class since I wasn’t using it anyway (I used this utility class before Hibernate became a JPA provider).
After this cleanup, the enhance target spat yet another error:

[enhance]Class com.newsplore.weather.bo.UsForecast has property weatherIconByCode declared
    in MetaData, but its getter method doesnt exist in the class!
  [enhance] May 27, 2009 4:03:50 PM org.datanucleus.enhancer.DataNucleusEnhancer main
[enhance] SEVERE: DataNucleus Enhancer completed with an error. Please review the enhancer
log for full details. Some classes may have been enhanced but some caused errors
  [enhance] Class com.newsplore.weather.bo.UsForecast has property weatherIconByCode
declared in MetaData, but its getter method doesnt exist in the class!
  [enhance] org.datanucleus.metadata.InvalidMetaDataException: Class
com.newsplore.weather.bo.UsForecast has property weatherIconByCode declared in MetaData,
 but its getter method doesnt exist in the class!
  [enhance]     at org.datanucleus.metadata.ClassMetaData
.populateMemberMetaData(ClassMetaData.java:497)

So a domain object (UsForecast) has a property called weatherIconByCode. Hmm, I didnt remember of such property but looking at the source I saw that the offender was a private utility method called getWeatherIconByCode(String code) that was being processed by the enhancer as if it was a property (I’m using annotations bound to getters and not to fields). Since adding @Transient didn’t fix it, the solution is to rename the method not to start with get/set (or move it in an utility class which you shouldn’t do if you follow DDD).
To my relief, after this fix the build didn’t yield any more errors so I had a nice webapp folder ready to be deployed. AppEngine has a local sandbox (based on Jetty) that should be used before deploying into the cloud. You can start the appserver using an ant target but I chose to get more control and create a script that accomplishes the same thing. Here it is:

cd build/webapp
java -classpath <appengine-sdk-home>/lib/appengine-tools-api.jar
com.google.appengine.tools.development.DevAppServerMain --port=8080 --address=localhost

Save this in a file called startgae.sh (mind the path, you should be able to cd to the webapp folder from it) then make it executable.
Using this script I fired-up the container and got no startup error. This was weird since I knew I was using JMX (via Spring and annotations) and as well the task scheduler (using Quartz) was seemingly up and running.
I hit the home page to see what has been deployed and I get a 404 error. My web.xml has this mapping:

  <servlet>
    <servlet-name>weather</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
  </servlet>

  <servlet-mapping>
	<servlet-name>weather</servlet-name>
	<url-pattern>/</url-pattern>
  </servlet-mapping>

but this doesn’t work in the local servlet container. This was quite annoying as not only the home page doesn’t work, the static folders (css/, js/, img/) are routed through the weather servlet too. Surprisingly as it turns out, this mapping worked perfectly in the AppEngine environment. To be able to continue with the work in the local environment, I had to define fine grained mappings for all the URLs.
After this change, the first URL that started working was the /faq which provided the first breakthrough. Although only the presentation tier is in use on the /faq page, the fact that the page worked provided a bunch of good news: Spring 3.0 DI container, SiteMesh, Spring Security and SpringAOP were all correctly running.

Stay tuned for the second part detailing more JPA, datastore indexing, a bunch of the AppEngine limitations, a bit of local environment hacking and how to use “request welding” to accomplish long running tasks.

Selecting a web hosting provider is a tough job for any web developer that wants to put a Java/JEE web application online. The choice is much simpler when it comes to publishing a PHP web site and there are a load of cheap (and sometimes quite reliable) PHP hosting providers to choose from with LAMP being the de facto standard in the web hosting world. But when it comes to Java hosting providers the picture becomes blurrier. The common thing that all these environments need is a Java container. The most popular choice is Tomcat but there are providers that use Resin, Weblogic or Websphere (the latter two are full fledged JEE containers).
With the raise of lightweight J2EE servers started by the Spring folks, a little revolution began in the Java world: running enterprise-grade JEE webapps without the need for an EJB container; a servlet container is enough. Tomcat should fill the bill for just about any Java web application that doesn’t use EJBs. The advantage that comes from running a servlet container is the smaller footprint compared with an EJB container. This is critical when it comes to selecting hosting environments since you’re paying for resources (especially RAM) that have to be sized to accommodate the memory requirements of the web application.

Self hosting: Home-based dedicated server (near $0)

You can use your home internet connection and a PC where the web container runs and deploys your web application. There is an obvious advantage since the server is physically located close to the development team: complete control and physical access to the machine. The other big plus is the price: you’re already paying for the internet connection and chances are you already have a PC that can be transformed into a server. And chances are that you already have a development environment that can be the basis of the production deployment. Here are the shortcomings: - Bandwidth limitations: the webserver where the webapp lives will upload content to the web browsers that access the application. The upload bandwidth is smaller than the download bandwidth and will result in longer load times for web clients. - Some internet providers cap the internet traffic so you may run beyond the limit.

- You may violate contractual obligations you agreed with your Internet provider by running your own web server.
- The internet connection is not reliable: provider’s downtime directly translates in downtime for your webapp.

Managed hosting ($6 and up)
This is the cheapest solution but it poses the most restrictions. The main limitation is the inability to deploy webapps that depend on Java bytecode manipulation at runtime, mainly Hibernate and Spring. This is due to classloading security constraints that prevent the bytecode manipulation libraries (CGLIB or javassist) to perform runtime code enhancements.
Working with managed hosting environments poses more limitations:
- Deployment: some popular hosting providers deploy once a day: you drop the war file or the exploded war in a specific folder on the hosted environment and it will be deployed at a specific time picked by the provider (more here). If it fails, you have to wait another day till the app will be redeployed or pick a provider that supports instant drop&deploy.
- Debugging a failed deployment: I couldn’t make log4j write the log into a location on my hosted file space so I had to resort to a JDBC appender to actually see what causes a failed deployment. This required two deployments (one to try a location on the hosted folder and a second one to use the JDBC appender; that’s two days due to the deployment constraint detailed above) and the issue turned out to be the classloader security issue preventing CGLIB to change the bytecode. I have only tried to deploy a war file with Hibernate and Spring dependencies in Godaddy’s Java hosting environment (their Linux Deluxe plan) before I gave up on managed Java hosting.
- Sharing the environment with other webapps. Your app will be deployed along with other applications and compete for the same resources.
- Access to server logs and traffic stats is very limited. You can only use Google Analytics (or similar) but not Awstats (at least not for free).
- Outgoing traffic may be restricted or blocked.

Unmanaged hosting solutions
Unmanaged doesn’t necessarily mean that the hosting company doesn’t manage anything, they just leave the actual OS administration in your hands so you’ll have a physical server under your complete control, everything else from network management to hardware maintenance is done by the hosting company. You/your team has to have solid server administration knowledge: Linux, Apache, Java AppServers, software firewall (iptables), backup/restore, monitoring, security, patching, DNS and more. A web administrator inside your team usually takes care of all these details. After the machine is up and running you are free to install and run any Java container and deploy your application.
Below are the options when choosing to run a dedicated server.

Colocated server ($70 and up)

You can colocate the server which means that you buy then ship your own rack-mountable server to a data centre where it will be installed by on-site technicians. The monthly fee you’ll pay starts at around $70 (prices are per one unit which is a slot in the rack) and includes the rack space rental, network connectivity, location security, 24/7 power (not counting mishaps), etc. Denial of Service prevention services are usually provided as part of the package. There is complete freedom in terms of the deployment environment since you own it. However, there are two problems that come with this solution:

- Initial cost associated with buying rack-mountable servers that usually start from $1.5k (check SuperMicro, they’re good). These machines contain redundant components to cope with failures.
- Maintenance: When components break you have to manage their replacement. If the whole machine crashes then your website will be offline until the machine is replaced.

Rent a dedicated server ($100 and up)

Instead of buying your own server, you can choose to rent a dedicated server from the hosting provider. Kevin Rose unveils how digg.com was initially hosted on dedicated servers rented from Ev1Servers.net (see full article). The benefit of this solution is the lower initial cost compared to colocation, beginning at around $100/month and of course the complete control over the environment. The drawbacks are the same as colocation except for servicing the box that falls under the hosting provider’s responsibility.

Virtual Private Server ($20 and up)

A VPS ” is a method of partitioning a physical server computer into multiple servers”. Each of these partitions behaves like a full fledged server, has an IP address and can be reachable from the internet. The computers that are VPS partitioned, sit on the hosting provider’s infrastructure. The provider is responsible for network connectivity, bandwidth and continuity of service in case of physical server failure. There are two main VPS software providers: Virtuozzo and Xen. Of the two, I recommend Xen-based VPS providers since the virtualized machines cannot be oversold and thus you’ll get consistent performance for the money (a detailed explanation here). The main benefit of the VPS is the excellent price/performance ratio starting as low as $20/month. Another big advantage is the ability to “resize” the virtual partition on demand in a matter of minutes (you’ll pay more of course), add more virtual machines to your deployment () as well as “relocating” it to a different data center

if the provider has multiple locations (West Coast and East Coast for instance). Root access is also provided. The drawbacks are:
- the operating systems that you can run are controlled by the hosting provider and usually are Linux distributions (here’s an example). However, the choices are very popular Linux distros.
- you have to have server administration expertise.
I run this blog, spincloud.com and the rest of my web properties on a VPS. I use the excellent Linode after I had switched from Slicehost, having performance issues. I am quite happy with Linode, their service rocks and they’ve been delivering consistent performance since day one.

Cloud services (EC2: $100 and up)

Cloud computing is today’s trend. The company that kick-started the trend was Amazon with their Elastic Computing Cloud. EC2 is actually an advanced form of VPS: you can run your own virtual machine image (or AMI) or pick one from the existing stock. After the AMI is booted and configured, you have the liberty of installing any Java environment. You get root access too. From the high profile websites, Smugmug uses EC2 and S3 and they save money doing so. In fact Smugmug was one of the early adopters of Amazon’s cloud services.

Drawbacks:
- If you reboot the machine or if it simply crashes then all data is lost, including any database records you happen to have. This was a real problem for developers until Amazon added the possibility to mount a persistent S3 volume in an EC2 instance. S3 stands for Simple Storage Service and is Amazon’s offering for cloud-based storage.
- You have to learn new terminology and set of skills that are applicable to Amazon’s problem-set only. You’ll also tie the solution to a single vendor.
The cost for an EC2+S3 starts at close to $100/month.
If you’re thinking of using the dirt-cheap (starts free) Google AppEngine for deploying a Java application you’re out of luck, they only support Python. Update: Google has added Java support to Appengine. at the begining of April. I have a dev account and I’ll evaluate it in a separate post.

Common concerns when selecting a hosting provider
Price
Don’t pick a service just because it’s cheap. Perform some due diligence on the company, where their data centers are located and how good their service is. You don’t want your website hosted on the wrong side of the world. Monitor WebHostingTalk forums; hosting providers often run promotions there.
Look for monthly plans and don’t commit to yearly plans since you may end-up being unhappy with the provider and with no choice to get out of the contract.
Be suspicious when you see associated setup fees coming with setting-up an account. They are not refundable and good hosting providers simply don’t have them.

Uptime
You want no interruption in service for your web servers so uptime is a must. You can’t afford to have downtime since it translates directly into lost customers and ultimately lost business. Look for hosting providers that offer SLAs (Service level agreements) and have proven reliability. Netcraft is a neat tool that measures the reliability of hosting providers on a monthly basis. Look for the providers that guarantee five nines (99.999%) uptime, that is five minutes a year.

Technical support
My personal experience is that I used the tech support while I was using a shared web hosting. I haven’t used any after moving to the VPS.

Transfer volume
Perform some basic calculations for the required traffic based on the expected traffic and data volume sent out. Providers bundle 200GB/month or more in their offerings these days. This should be enough for most of the cases. Make sure your website doesn’t get cut-off if your traffic spikes and/or you’re not paying a ridiculous amount of money per GB if you go beyond your quota.

Security
All hosting providers offer a form of protection against DDOS but you are responsible for the software that runs on the server. I got the online forum I was running for Newsplorer hacked a couple of years ago because I didn’t upgrade the forum software to patch a security vulnerability that was promptly exploited.

Backup
Hosting providers offer backups or you can do it yourself with a bit of effort and some cron knowledge. Save the backed-up files on a different location (I use S3 for my backup needs but there are free solutions too).

In Part 1 we imported world political borders into a database table. In this second part we’ll use the table and generate a script that will be used to add the borders overlay to Google Maps. We’ll use the cool Ruby and some fancy GIS words along the way.

We left-off with a database table containing all borders. The goal today is to produce a Javascript file that will be used for overlaying polygons representing countries, over a map using the Google Maps API.

Let’s examine the table data first (I’m using mysql through the command line):

mysql> desc world_boundaries;
+-----------+--------------+------+-----+---------+-------+
| Field     | Type         | Null | Key | Default | Extra |
+-----------+--------------+------+-----+---------+-------+
| ID        | int(11)      | NO   | PRI | NULL    |       |
| FIPS      | varchar(255) | YES  |     | NULL    |       |
| ISO2      | varchar(255) | YES  |     | NULL    |       |
| ISO3      | varchar(255) | YES  |     | NULL    |       |
| UN        | int(11)      | YES  |     | NULL    |       |
| NAME      | varchar(255) | YES  |     | NULL    |       |
| AREA      | int(11)      | YES  |     | NULL    |       |
| POP2005   | bigint(20)   | YES  |     | NULL    |       |
| REGION    | int(11)      | YES  |     | NULL    |       |
| SUBREGION | int(11)      | YES  |     | NULL    |       |
| LON       | double       | YES  |     | NULL    |       |
| LAT       | double       | YES  |     | NULL    |       |
| ogc_geom  | geometry     | YES  |     | NULL    |       |
+-----------+--------------+------+-----+---------+-------+

A lot of information here but we’ll need just these fields: name (country name), iso2 (two letter country codes) the_geom, iso2 (border geometry) and region (grouping countries by regions). To check the data-set let’s examine a small country. I’ll pick the tiny State of Vatican for its size:

mysql> select iso2, AsText(ogc_geom), region from world_boundaries where iso2='VA';
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
| iso2 | AsText(ogc_geom)                                                                                                                                  | region |
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
| VA   | MULTIPOLYGON(((12.445090330889 41.903117521785,12.451653339581 41.907989033391,12.456660170954 41.901426024699,12.445090330889 41.903117521785))) |    150 |
+------+---------------------------------------------------------------------------------------------------------------------------------------------------+--------+
1 row in set (0.00 sec)

The only surprise here is that the country border is described as a MULTIPLOYGON spatial type which describe a collection of polygons that don’t interset. This is in order to accommodate countries that have islands under ownership.

Let’s see how these points look on the map. We’ll use the excellent polygon encoder utility written by Mark McClure. Copy Vatican’s point set in the “Input Text” input box (choose lng/lat option):

12.445090330889, 41.903117521785
12.451653339581, 41.907989033391
12.456660170954, 41.901426024699
12.445090330889, 41.903117521785

Here’s the result:

Not very accurate but we’ll have to live with it, it’s a free data set after all…

We can also get a glimpse of how the Javascript code containing the encoded polygon looks like by using the same utility. Click on the “Show Code” button; you should see this:

...
var polyline1_1 = new GPolyline.fromEncoded({
  color: "#0000ff",
  weight: 4,
  opacity: 0.8,
  points: "mew~Fyt}jAm]_h@~g@i^qIhgA",
  levels: "PHIP",
  zoomFactor: 2,
  numLevels: 18
});
map.addOverlay(polyline1_1);

The last line adds this polygon (i.e. is a polyline but more on that later) to the map.
The “points” property of the GPolyline is an encoded string which we’ll have to generate. The full documentation on how the encoding is performed read Google’s polyline documentation.
Let’s use a bit of pseudocode to describe the steps we want performed:

for each ogc_geom of countries fetched from DB
  generate encoded points
  create GPolygon Javascript code snippet
  collect snippet to Javascript file
end

First things first. Since I’m learning Ruby, I’m going to use it to implement these requirements. The extras I’m gonna need are Mysql (for DB connectivity) and GeoRuby (for processing spatial data types). Installing MySql gem to Ruby is easy (sudo gem install mysql) that is if you don’t want to get into using 64bit Mysql distributions (you don’t) so let’s skip that. Install GeoRuby with a”sudo gem install GeoRuby” command.
To verify if both mysql and GeoRuby are enabled, open an irb session then type:

>> require 'mysql'
=> true
>> require 'geo_ruby'
=> true

They should both render true.
A word about encoding polygons. Google’s encoded polyline/polygon API optimizes the rendering by enabling encoding more or less points giving different zoom levels (higher zoom, more points will be used). A well known optimization method is described by the Douglas-Peucker algorithm. However, to make my life easy I devised a simpler algorithm for dealing with this; I term it the straight point reduction algorithm and it replaces the D-P algorithm by uniformly reducing all points up-front and encoding the polygons always using the same number of points for all zoom levels. The rendering of the polygons is not impacted and the actual looks of the country borders is virtually unchanged at all zoom levels. I have found that using one in five points yields respectable performance and doesn’t impact the shape of the borders. Finally the algorithm used to encode coordinates is described here. The generated javascript has three functions: initializing the overlay (initOverlay) and adding/removing (addBordersOverlay/removeBordersOverlay)
To check out the code download the .rb file or click to expand.

require 'mysql'
require 'geo_ruby'

include GeoRuby::SimpleFeatures

def generate_js_border_overlay(output_file)
  factory = GeometryFactory.new
  wkt_parser = EWKTParser.new(factory)

  borderDB = Mysql::new('localhost', 'root', '', 'bordersdb')
  #region 150 is Europe
  res = borderDB.query("select name, iso2, AsText(ogc_geom), region from
                world_boundaries where region=150")
  encoded_polygon_desc = "";
  remove_warnings_layer = "function removeBordersOverlay() {n"
  add_warnings_layer = "function addBordersOverlay() {n"
  init_borders = "function initBorders() {n"
  res.each do |row|
    name, iso2, multi_polygon, region = *row
    processed_polygon = wkt_parser.parse(multi_polygon);
    encoded_polygon_desc << "var encodedPolygon_#{iso2};n"
    add_warnings_layer << "map.addOverlay(encodedPolygon_#{iso2});n"
    remove_warnings_layer << "map.removeOverlay(encodedPolygon_#{iso2});n"
    init_borders << "encodedPolygon_#{iso2} =
      new GPolygon.fromEncoded({n
      polylines: ["
    factory.geometry.each do |landmass|
      landmass.rings.each do |ring|
        encoded = encode_by_reducing_pointcount(ring.points)
        init_borders << "{color: "white",n
        weight: 5,n
        points: '"
        init_borders << encoded[0].gsub(/\/, '&&') + "',n
        levels: '#{encoded[1]}',n
        zoomFactor: 2,n numLevels: 2}"
        init_borders << "," unless ring == landmass.rings.last
                 && landmass == factory.geometry.last
        init_borders << "n"
      end
    end
    init_borders << "],nfill:true,n
    opacity:0.7,n
    color: 'white'n
    });"
  end

  add_warnings_layer << "n}"
  remove_warnings_layer << "n}"
  init_borders << "n}"
  File.open(output_file, 'w') {|f|
    f.write(encoded_polygon_desc + "n")
    f.write(add_warnings_layer + "n")
    f.write(remove_warnings_layer + "n")
    f.write(init_borders)
  }
end

def encode_by_reducing_pointcount(points)
  dlat = plng = plat = dlng = 0
  res = ["",""]
  index = -1
  for point in points
    index += 1
    #straight point reduction algorithm: use every 5th point only
    #use all points if their total count is less than 16
    next if index.modulo(5) != 0 && points.size > 16
    late5 = (point.y * 1e5).floor
    lnge5 = (point.x * 1e5).floor
    dlat = late5 - plat;
    dlng = lnge5 - plng;
    plat = late5;
    plng = lnge5;
    res[0] << encode_signed_number(dlat)
    res[0] << encode_signed_number(dlng)
    res[1] << encode_number(3)
  end
  return res
end

def encode_signed_number(num)
  sig_num = num << 1
  sig_num = ~sig_num if sig_num < 0
  encode_number(sig_num)
end

def encode_number(num)
  res = ""
  while num  >= 0x20 do
    res << ((0x20 | (num & 0x1f)) + 63).chr
    num >>= 5
  end
  res << (num + 63).chr
  return res
end

generate_js_border_overlay("/tmp/bordersOverlay.js")

select name, iso2, AsText(ogc_geom), region from world_boundaries where iso2='ES'")

I’m picking a nicely colored background too, electric lime that is; in the ruby script I’ll update the two occurrences of “color: ” to “color: #CCFF00″.After the script is ran, the file /tmp/bordersOverlay.js is created.

Let’s put it all together. This is a live embedded map that uses the generated JS script:
[include file=/wp-content/uploads/2009/03/embedmap.html iframe=true width=100% width=520 height=440 scrolling=no]
The usage is straightforward: include the JS file in the HTML snippet

 <script type="text/javascript" src="bordersOverlay.js"></script>

then call these two functions in your JS code after initializing the GMap2 object:

initBorders();
addBordersOverlay();

You can also peek at the source code here.
If you zoom-in towards a border area you can observe how approximate the true border is followed. This is due to the straight point reduction algorithm detailed earlier. Spain’s islands are also colored correctly (Balearic islands to the East of the mainland and the Canary Islands which become visible if you browse off the West coast of Africa).
You can also dynamically change the polygon colors as needed. For the above case we’ll use the following code:

encodedPolygon_ES.setFillStyle({color:blue",opacity:1})');

That’s all.
The only concern is client-side performance. If there are many polygons to render, the browser may take a long time to render the countries overlay. This is quite apparent when trying Spincloud‘s Europe weather warnings (Meteoalarm) overlay by using Internet Explorer. Position the map over Europe and click the Meteoalarm button located in the top-left corner of the map. Rendering may take a long time (Chrome handles this the best, it’s mighty fast).
I’ll close this tutorial with some colorful maps created using variations of the above code.

All countries in the Mediterranean Basin colored by region:

Countries in G7:

Finally, all OPEC member countries:

One thing I needed when designing the Meteoalarm mashup for Spincloud were the political boundaries for all European countries. With them at hand, I would use the polygon overlay from the Coogle API and fill the country polygons with the respective weather warning colors.

This first part is a tutorial on how to import world political borders into a MySQL database table.
The second part in these series will use this table to create a script that will add the country borders overlay to Google Maps.

So after learning that Google Maps API doesn’t give programatic access to the political boundaries I moved on to doing it myself. I read about the encoded polylines and figured that I only missed the country boundaries as encoded polygons so I started searching for readily-available resources. Sure enough there were nowhere to be found.

The next best thing was to find the country boundaries in some GIS format and generate the polyligons myself. The information seems to be sparse but after a bit of research I found was the CIA World Databank but the format is incomplete, it only defines the polygons with no country names or any other administrative hook that I would be able to use. I then spent hours trying to find a better resource until I remembered that Mapping Hacks (an excellent book by the way) has data set for the examples in the book. I had the first breakthrough, the website makes available updated world borders files here, they are mirrored from thematicmapping.org.

We’ll use a utility called shp2mysql to export the boundaries to SQL. I’m on a MacBook and no shp2mysql binaries for my platform were available in the original package so we’ll have to compile it first then run it to be able to export the shapefile to SQL.

Below are detailed all steps needed to to produce the borders SQL file on Leopard 10.5.x (note that you have to have the Developer Tools installed). In short, copy and run the following script on your mac in a file called process_boundaries.sh. The script creates a file called wb_dump.sql which is ready to be imported into database:

mkdir borders
cd borders
wget http://mappinghacks.com/data/TM_WORLD_BORDERS-0.2.zip
mkdir boundaries_shp
unzip TM_WORLD_BORDERS-0.2.zip -d boundaries_shp
wget http://kartoweb.itc.nl/RIMapper/code/shp2mysql_0_4.zip
unzip shp2mysql_0_4.zip -d shp2mysql_04
wget http://dl.maptools.org/dl/shapelib/shapelib_1_2_10.zip
unzip shapelib_1_2_10.zip
cd shapelib-1.2.10
make
cd ..
rm ./shp2mysql_04/src/*.o
sed 's/^OBJS.*$/OBJS = \.\.\/\.\.\/shapelib-1\.2\.10\/shpopen\.o \.\.\/\.\.\/shapelib-1\.2\.10\/dbfopen\.o/' shp2mysql_04/src/Makefile > shp2mysql_04/src/Makefile2
mv shp2mysql_04/src/Makefile2 shp2mysql_04/src/Makefile
cd shp2mysql_04/src
make
cd ../../
cp shp2mysql_04/src/shp2mysql .
./shp2mysql -d boundaries_shp/TM_WORLD_BORDERS-0.2.shp world_boundaries test_db > wb_dump.sql
rm *.zip
rm -rf shp2mysql_04
rm -rf shapelib-1.2.10
rm -rf boundaries_shp

Make the above script executable then run it:

chmod +x process_boundaries.sh
./process_boundaries.sh

If you’re interested in the details about this script then read on otherwise happily skip it. The final SQL file is linked at the end of this post.

We’ll start by bringing the shape files:

mkdir borders
cd borders
wget http://mappinghacks.com/data/TM_WORLD_BORDERS-0.2.zip
mkdir boundaries_shp
unzip TM_WORLD_BORDERS-0.2.zip -d boundaries_shp

The files are in ESRI’s shp format which wasn’t exactly what I needed. I had to export them in a more usable format, SQL in my case. For this I used the shp2mysql utility for which I didn’t find the MacOS executable so we’ll compile it from source. Since it has a dependency to shapelib we have to download and compile it along:

mkdir shp2mysql
wget http://kartoweb.itc.nl/RIMapper/code/shp2mysql_0_4.zip
unzip shp2mysql_0_4.zip -d shp2mysql
wget http://dl.maptools.org/dl/shapelib/shapelib_1_2_10.zip
unzip shapelib_1_2_10.zip
cd shapelib-1.2.10
make
cd ..

Now you should have both shapelib and shp2mysql downloaded with shapelib succesfully compiled.  Now we’ll compile shp2mysql: we have to delete all .o files (they have been compiled for a different platform), fix the Makefile then issue the make command

rm shp2mysql/src/*.o
sed 's/^OBJS.*$/OBJS = ../../shapelib-1.2.10/shpopen.o ../../shapelib-1.2.10/dbfopen.o/' shp2mysql/src/Makefile > shp2mysql/src/Makefile2
mv shp2mysql/src/Makefile2 shp2mysql/src/Makefile
cd shp2mysql/src
make
cp shp2mysql/src/shp2mysql .

which should execute successfully.
We’re ready to export the world boundaries into SQL format. Switch to the boundaries folder then issue the export command as follows:

cd ..
./shp2mysql -d boundaries_shp/TM_WORLD_BORDERS-0.2.shp world_boundaries test_db > wb_dump.sql

this will create a SQL file called wb_dump.sql ready to be loaded into your database.

Running the script will generate a file called wb_dump.sql which is ready to be imported on your spatially enabled database:

All we have to do now is to import the sql file into the database. Note that the database has to be spatially enabled (MySQL is by default, I’m using 5.0). Issue the following command:

mysql -uroot -p my_database < wb_dump.sql

Now we should have country borders for the whole world imported in the database.
For convenience, I have made the world boundaries SQL file available for download (4.3 MB).

In the second part of this how-to, we’ll use this table to create the country borders overlay for Google Maps.

I’m a bit of a news junkie and when I heard that a twitter stream called @BreakingNewsOn existed I did the natural geekish thing, mash it up. Yahoo pipes was my friend here. I haven’t used the Pipes before but that didn’t quite matter, Yahoo did a great job keeping this playgroundsy product in the child realm so even a caveman could do it. The mashup result -two pipe connections later- is beyond the link:

http://pipes.yahoo.com/newsplore/breakingnewson

Today was my lucky day according to least one popular blog. I got the Themes tab enabled in my gmail account, wohoo! I only wish they could change the font sizes too, I’m up for a microscopic look’n feel, really small and cozy-cute.

My current favorite is the Shiny theme:

Now, if only googlers would add a Cover Flow option instead of that dull message list. Oh, and it should come gratis as usual.

I learned something today too. To take a screenshot on a MacBook, press Shift-Command-4 then select the portion of the screen you want to capture, then find the pic saved conveniently on your Desktop in png format. The karate stance is only required if you want to execute the entire screen capture kata.

[my gmail inbox via Techcrunch]

I just bought my first MacBook yesterday. My current laptop (a beaten down Toshiba Satellite) was long overdue and I realized that I didn’t want yet another boring Win notebook, it just doesn’t do it for me anymore. I’m in front of a PC every day at

work and I surely don’t want to see the same things while at home. And so I was determined to buy a Mac. My first Mac. I didn’t know that much about Apple and still don’t (imagine that)…
I waited until the new MacBooks were announced (no rush, I don’t know much about these machines anyway) and I started to do my research. A $1500 laptop seems a reasonable price for a good laptop and I had two requirements: not that big and not that heavy. MBPs seemed too expensive so my choices were the 2.0 or 2.4 MacBook. I eventually choose the 2.4 version mostly because of the backlit keyboard.

I ordered it online at the Apple Store and received the box it two days later. I didn’t set my expectations too high although I hoped that it’ll suck less than my boring Toshiba.

The first impression was quite pleasant, the machine looks astonishing. It’s finished to perfection. And pretty light too. OK, first impresion was good. I understood why I keep seeing them in the movies: they’re damn sexy. Still, skin-deep doesn’t do it for me. Gotta be more under the hood to really like it.

I’m a minimalist. I can say that the installation was the shortest I’ve ever seen. I especially enjoyed the snapshot that is taken when you create the username, nice touch.

Then I saw the interface. I’ve been around UNIX/Linux since the early ’90s, I developed an attachment to it since my student years. I still remember the HPUX machine we had in the lab, what a gorgeous GUI it had (ah Motif, still miss you). My first contact with Linux was through Slackware, a remarcable distro. But the Leopard OS is a beauty. It resembles the Linux’s own Gnome (now I know where the inspiration came from) but it has a window manager done right. And again, I realized one more thing: the legions of Apple followers started to look rational. And so do the lineups in the Apple Stores or around wireless carrier’s outlets that sell iPhones. Revelation time.

As I explored the OS more and more, I noticed that didn’t miss much from my old PC world. Moreover, I feel more productive than before. The multi-touch trackpad is just good. I’m a long-tie fan of browser mouse gestures (since Opera) and moving the concept one step ahead is quite refreshing. The Safari browser is much more responsive than Firefox and I’ve switched to it. There are  less and less differences between browsers these days anyway and I can live without the firefox plugins for a while until I feel that I really need them (especially Ubiquity and delicious).

As a linux guy, I feel at home with the goods that the UNIX OS has to offer. Moving to UNIX was a bright move by Apple: they can access the technological advancements and ideas contributed by the powerful Linux/UNIX communities. A brilliant gambit.

As a Java&web developer I was a bit concerned by the suitability of this notebook as a development machine but I can report that Eclipse, MySQL, Subversion and all the others work just fine. I’m having a problem with the Maven and SpringIDE plugins but I’m blaming the Eclipse (it’s a 3.4 version) at this point. The 13″ screen is just fine for coding, no complaints.

I actually feel more inspired working on this machine. Now I only need the Mirra and my dream home office will be complete.

Enough already. I switched. There’s no looking back.