projects | Tales From The Cloud

Spincloud, now with worldwide forecast

In my constant search for free weather data for Spincloud, a short while ago I have found a gem: free forecast data offered by the progressive Norwegian Meteorologic Institute. The long range forecast coverage is fairly thorough and covers most more than 2700 locations worldwide. I am happy to announce that I have extended spincloud.com to include it.

The data is refreshed every hour and the forecast range is available for the next seven days. You can bookmark any location or subscribe to weather reports via RSS.
On a different but related note, I can only be thankful for the free data offered by various meteorological organizations that allows Spincloud to exist and I believe in freeing public data (weather related and otherwise) as it belongs to the public that finances government and inter-government agencies in the first place. The Norwegian Met Institute is a great example for freeing its data and I am saluting them for making the right decision.
Now if only all such progressive Meteorological institutes around the world would agree on a common format for disseminating their data, it would make developers like me a tad happier…

New Spincloud feature: heat map overlay

heatmap-overlay It took a while since the previous feature update to Spincloud. I have done a number of upgrades to the underlying tech and some intensive code refactoring but nothing visible. The time has come for another eye candy: heat maps. It is a map overlay that shows a color-translated temperature layer based on interpolated values of the current weather conditions. It gives a quick indication of the average temperature across all land masses where data is available.
Naturally in areas where data is more dense, the visual representation is better. The toughest job was to figure-out the interpolation math. I still have to refine the algorithm currently based on the inverse distance weighting algorithm which interpolates scattered points on a surface. Apparently a better one to use is kriging but I have yet to implement it.
The temperature map is generated from current temperatures and is updated once every hour. You can toggle the overlay by clicking the “Temp” button found on the top of the active map area.

A project triumvirate

There are three forces that shape a project: domain, process and technology. Add the “-driven” suffix to any of them and you’ll perhaps recognize some of the methods used in projects you’ve been involved in and yet, as soon as one takes too much of a lead against the other two, failure will follow almost inevitably. At the intersection of these three forces we find familiar terms and concepts but first a word or two about each of them:

Technology
Back in the days of the tech bubble, tech was allmighty. Buzzwords like Java, EJBs, PHP defined entire projects. It was the time where software became accessible to a much larger audience. The new wave of enthusiastic geeks embraced everything from new languages to professional certifications to the then-nascent open source. I admit having my share of technology-driven projects back in the day…

Process
Process brings structure and pace to a project. The two complementary components of a project process are methodology and integration. We are all too familiar with methodology: waterfall, RUP or agile methods of software development are vastly documented and practiced; integration largely is defect management, testing, build, deployment and documentation. Today they all come together in what is called continuous integration where all these concepts become interrelated in a repetitive process that produces accountability and visibility into the progress of a project. Process is also the one force that tends to disappear first after a project is finished.

Domain
The domain captures entities and business logic, is driven by the business requirements and is in no way influenced by the two other forces. The most successful way to employ the domain is through Domain Driven Design where the main focus of software development is neither technology nor process but the business requirements.

Continue reading “A project triumvirate”

Reviewing Google AppEngine for Java (Part 1)

appengine_lowres

When Google announced that Java is the second language that the Appengine will support I almost didn’t believe it given the surge of the new languages and the perception that Java entered legacy but the JVM is a powerful tried-and-true environment and Google saw the potential of using it for what it is bound to

become: a runtime environment for the new and exciting languages (see JRuby and Grails). The JVM is the new gateway drug in the world of languages.

Note: I’ll break down this review into two posts as it’s too extensive to cover everything at once. This first part is about initial setup, JPA and some local webapp deployment issues. In the second part I’ll describe how to load data into the datastore, data indexing, how I reached some of the limitations of the AppEngine and how to get around some of them.

I managed to snatch an evaluation account and last few days I have been playing with it. (as of today Google opened the registrations to all). My goal was (again) simple: to port Spincloud to the AppEngine. The prize: free hosting which amounts to $20/month in my case (see my post about Java hosting providers), some cool monitoring tools that I can access on the web and of course the transparent super-scalability that Google is touting, in case I get techcrunched, slashdotted or whatever (slim chances but still…).
I am a couple of weeks into the migration effort and I can conclude that the current state of the AppEngine is not quite ready for Spincloud as I’ll detail below but it comes quite close. The glaring omission was scheduling but Google announced today that the AppEngine will support it (awesome!) and I plan to use it from day one.

My stated goal is running Spincloud on AppEngine. Here are the technologies I use in Spincloud:
– Spring 3.0 (details here)
– SpringAOP
– SpringSecurity with OpenId
– Webapp management with JMX (I know, it won’t fly)
– Job scheduling using Spring’s Timer Task abstraction (not optimistic about this one either)
– SiteMesh (upgraded to 2.4.2 as indicated)
– Spatial database support including spatial indexing
– JPA 2.0 (back story here)
– Level 2 JPA cache, query cache.
– Native SQL query support including native SQL spatial queries.
– Image processing (spincloud scraps images and processes pixel-level information to get some weather data)
Continue reading “Reviewing Google AppEngine for Java (Part 1)”

Spincloud, a month old: looking good

I launched Spincloud on New Year’s day. I didn’t expect a flood of traffic to hit the site since -let’s face it- there was nothing groundbreaking. Spincloud was born from my idea of having access to the world weather in one step or less and I’m pleased with the results so far. Spincloud gains adoption every day and this is only due to the people that saw something new and exciting in it. There are blog posts, traffic graphs and other goodies to talk about so let’s begin.

I announced spincloud’s launch on my personal blog then Keir from googlemapsmania broke the news to the community. And so the first spike of traffic was logged on Jan 2nd.

jan_traffic

Then followed a month of relative silence during which I took a much needed break from the project. I couldn’t help but refactoring and cleaning-up the code a bit. I had two minor releases deployed in prod in the mean time and everything went smooth.

I had a new feature I wanted to have online before the initial release, the meteoalarm mashup that displays weather warnings across Europe but chose not to delay it (a Good Decision in hindsight) . I started working on it by mid January , did a lot of digging for administrative borders data then getting the meteoalarm warning data right (those guys don’t have any integration point). I was done two weeks later.

In the mean time I was trying to raise the awareness about spincloud. I submitted the site so some mapping/mashup themed websites, improved the indexing (still working on it). Then Wednesday last week the European bloggers finally found it. UK’s Mapperz ran a nice little story that seems to have set the things in motion. A number of other bloggers picked-up the story and a surprisingly good feedback came from Italy: Matteo and italiasw.com did a great job while gisdk.blogspot.com of Denmark noticed the RSS facilities. Then the referral machine kicked in, killerstartups.com, programmableweb.com amongst the sites linking. And thus the second spike registered on the traffic meter:
Continue reading “Spincloud, a month old: looking good”

@Meteo_alarm twitter stream now available

	=

Update Nov.25.2009: The twitter name has changed to @meteo_alarm. Please update your bookmark or follow list.

Between code refactoring and cooking new ideas, I took some time to add a new feature to Spincloud. Inspired by the crowd-sourced weather updates triggered by really bad snow storms in the UK, I (finally) started to see the advantages to using Twitter. So I took the Meteoalarm data that I mashed-up on Spincloud’s map and streamed it to Twitter.

So here it is: Meteoalarm on Twitter via Spincloud = ~~@Meteoalarm~~ @Meteo_alarm. The stream is updated twice a day and it presents a concise text-only representation of the color encoded weather warnings in Europe for today and tomorrow.

Spincloud now integrates with Europe’s weather warning service Meteoalarm

A short month after the initial public release of Spincloud, I am pleased to announce the addition of a new feature: integration with the European weather warning service Meteoalarm which provides the most relevant information needed to prepare for extreme weather, expected to occur over Europe. The colors indicate the severity of the danger and its possible impact: Red: very dangerous, Orange: dangerous, Yelllow: potentially dangerous, Green: no warnings.

Currently only the country-level warnings are displayed. You can access the new feature by clicking the “Meteoalarm” button located on the top-left corner of the map. Note that the map has to be positioned over Europe for this buttton to be enabled.
More information about Meteoalarm here.

Important note: When using this new feature, the best experience is with Google’s Chrome, Mozilla or Safari browsers. Enabling the Meteoalarm overlay is much slower on IE browser which sometimes results in the browser reporting a slow-running script on slower hardware. Wait a few more seconds, the new overlay will render eventually.

Announcing a new weahter mashup: Spincloud

I started working on Spincloud back in 2003 when I got interested in regular expressions. I liked them so much I started looking for large sets of freely available data that I could crunch. Weather data was quite appealing; not only I’d parse it but I could get instant feedback by just looking at the parsed data, after all we understand temperatures and wind speeds and the like. I’d get to use other technologies I couldn’t experiment at work where I was doing the regular J2EE stuff and I kept feeling the itch to do new things.

And so Spincloud was born; it’s mission: to track the weather worldwide. It didn’t have a name back then but as things started to shape-up, I realized that the experiment was slowly turning into a real project. Then Google published its Maps API and I just knew that my small little pet project just found the perfect playground. It’s been a while since I started with this and I had no intention of publishing it initially. It has grown beyond its initial purpose and I feel it’s time to set it free.

This is the third version of Spincloud, the first two iterations were not publicly released. This is not an open source project; however, I’m planning on blogging about the technical challenges I’m encountering and are interesting enough to share.

Here are the main features of Spincloud:

– Instant weather search

– Forecast displayed for all available locations in surveyed area (US only); radar mashup.

– Upon free registration, one can tag favorite locations then track the weather either through the website or via RSS; ability to set a home location

– Integration with Flickr

– Ability to add comments for individual locations through Google Friend Connect

– Easy registration that uses OpenId and Facebook Connect.

– Both metric and imperial units of measures supported.

Update: googlemapsmania.com just ran a story about Spincloud here. Thanks guys!