Five years of weather in a timelapse

This has been a long time coming, five years actually. Spincloud went live in January 2009 and I’ve added the temperature heatmap overlay a few months later.
The heatmap and the corresponding map overlay is generated once an hour and it’s been faithfully doing so for six years. It does it by first generating an global temperature heatmap then cutting tiles (up to zoom level 6). Every hour.
But what I have also done back then was to also save one heat map image once a day. The idea was to generate a timelapse at some point, showing a visualization of the global temperatures over a longer span of time.
Now it’s time to show the result: below are 5 years of global temperatures in a one minute timelapse (make sure to switch to the HD version if not enabled by default):

Color-temperature

* I started collecting this data in Aug.2009 but the lapse above starts in in July 2010 as I have a data gap between Dec.2009 and Jul.2010. Frankly I can’t remember why.

The timelapse neatly shows the SYNOP and METAR coverage globally. The weather stations behind these data sources report at various times of day and I’ve figured that 19:00 GMT is the hour in the day with the most reporting stations. Looks like this has changed about a year ago if you look at Russia’s remote locations that are not covered well anymore at this time of day. The rest of the landmass is reasonably well covered.

I think it’s interesting to explain how I have actually generated the temperature heat map. The code you’ll see is pre Java 8 since I haven’t upgraded the code much in the past years.

Spincloud uses several sources of weather data and it’s quite remarkable how little they have changed over the past 6 years. I witnessed some data sources going offline but the core of the data still comes from the same sources. For the heatmap I use the METAR and SYNOP global data, mostly coming from the NWS servers who are providing clean and reliable data for many years. The data is stored in the local database and it essentially contains a map point, a temperature and a timestamp.
With these sources at hand, the logic to generate a heatmap image is as follows:
1. Iterate over each pixel on the global map and get the respective temperature. Only include land masses.
2. Interpolate that temperature with the temperatures of the nearest locations where there are temperature readings
3. Generate an 2×2 pixel rectangular area with a color that corresponds with the interpolated temperature. Only fill land masses in order to make the overlay look realistic.
4. Append that image in memory to the global heat map image in the correct location
5. Repeat until complete then dump the generated image to disk

Turns out that at any the temperature data points are less than 20,000 globally and so I can add all of them in a list in memory. In step 1, when iterating over each map pixel this list is looked-up for temperature points in vicinity.
The map mask referenced in the code below, the worldmap-mask.png looks like this:

Here’s the code (warning: not executable):

public void generateHeatMap() {
  //Collect all temperature points
  List metaPoints = getMetaPoints();
  BufferedImage heatMap;

   //Load the world map mask in memory; all black pixels belong to landmasses.
   //We'll use it to filter-in only land masses.
  try {
    heatMap = ImageIO.read(this.getClass().getClassLoader().getResourceAsStream("worldmap-mask.png"));
  } catch(Exception e) {
    logger.error("Exception: " + e.getMessage(), e);
    throw new RuntimeException("worldmap-mask.png not found");
  }
  Graphics2D g = (Graphics2D) heatMap.getGraphics();
  
  int blackRGB = Color.black.getRGB();
  //1. take every second pixel then scan the entire map surface
  int step = 2;
  for (int x = 0; x < MapUtil.MAP_WIDTH; x += step) {
    for (int y = 0; y < MapUtil.MAP_HEIGHT; y += step) {
      //only draw over land mass (masked with black).
      //seas/oceans will be left white.
      if(heatMap.getRGB(x, y) == blackRGB) {
      //Get the nearest temperature points along with their relative distance to the current map point
      SortedMap nearestPts = findNearest(metaPoints, x, y);
      //2. Compute the interpolated temperature using an inverse distance algorithm:
      // https://en.wikipedia.org/wiki/Inverse_distance_weighting
      int interpolatedT = (int)getInterpolatedTemperature(x, y, nearestPts);
      try {
          //identify the pixel color for this temperature
          g.setColor(getTempColor(interpolatedT));
      } catch(Exception e) { //skip exception, at worst there'll be gaps in th heatmap
          logger.info("Exception: " + e.getMessage(), e);
          logger.info("The temperature was: " + interpolatedT);
      }
        //3,4. Generate a step x step rectangle at current coordinates
        g.fillRect(x, y, step, step);
      }
    }
  }

  //5. Save image to disk. Done every day at 19:00 GMT only.
  saveLapseImageToDisk();
  //For Google maps overlay
  generateTiles(heatMap);

}

The second resource used and buried behind the call to getTempColor is the color temperature scale.
Color-temperature
The code that gets a temperature and figures its color is this:

//init code:
  private int[] colorPixels = null;
  colorPixels = new int[colorMapW * colorMapH];
  PixelGrabber pg = new PixelGrabber(heatColorImage, 0, 0, colorMapW, colorMapH, colorPixels, 0, colorMapW);
  pg.grabPixels();
..

private Color getTempColor(int temp) {
  if (temp == -9999) {
    return Color.white;
  }
  // 273 px, 39 gradations each 7 px. starting -60 ends +60
  int pos = (273 / 120) * (temp + 60);

  int c = colorPixels[(colorMapH / 2) * colorMapW + pos];
  int red = (c & 0x00ff0000) >> 16;
  int green = (c & 0x0000ff00) >> 8;
  int blue = c & 0x000000ff;
  // and the Java Color is ...
  Color cl = new Color(red, green, blue);
  return cl;
}

Currently there are 1774 useful images collected so far and still going. Two years back I have moved Spincloud to DigitalOcean (note: referral link) and kept collecting this historical data without a hitch.

To geneate the timelapse I used ffmpeg and this tutorial. You’ll notice the month-year embedded in the video, I have used used this howto to get them in and this to figure the subtitle format. There’s some code I wrote to generate the subtitles file to be in sync with the timelapse but it’s too boring to include.

As a side note, Spincloud runs a total of 8 background jobs collecting temperature and forecast data, generating temperature and radar tiles, and weather warning data, all on the cheapest DataOcean plan.

Spincloud, now with worldwide forecast

In my constant search for free weather data for Spincloud, a short while ago I have found a gem: free forecast data offered by the progressive Norwegian Meteorologic Institute. The long range forecast coverage is fairly thorough and covers most more than 2700 locations worldwide. I am happy to announce that I have extended spincloud.com to include it.

worldwide_fcast    

The data is refreshed every hour and the forecast range is available for the next seven days. You can bookmark any location or subscribe to weather reports via RSS.
On a different but related note, I can only be thankful for the free data offered by various meteorological organizations that allows Spincloud to exist and I believe in freeing public data (weather related and otherwise) as it belongs to the public that finances government and inter-government agencies in the first place. The Norwegian Met Institute is a great example for freeing its data and I am saluting them for making the right decision.
Now if only all such progressive Meteorological institutes around the world would agree on a common format for disseminating their data, it would make developers like me a tad happier…

New Spincloud feature: heat map overlay

heatmap-overlayIt took a while since the previous feature update to Spincloud. I have done a number of upgrades to the underlying tech and some intensive code refactoring but nothing visible. The time has come for another eye candy: heat maps. It is a map overlay that shows a color-translated temperature layer based on interpolated values of the current weather conditions. It gives a quick indication of the average temperature across all land masses where data is available.
Naturally in areas where data is more dense, the visual representation is better. The toughest job was to figure-out the interpolation math. I still have to refine the algorithm currently based on the inverse distance weighting algorithm which interpolates scattered points on a surface. Apparently a better one to use is kriging but I have yet to implement it.
The temperature map is generated from current temperatures and is updated once every hour. You can toggle the overlay by clicking the “Temp” button found on the top of the active map area.

Spincloud, a month old: looking good

I launched Spincloud on New Year’s day. I didn’t expect a flood of traffic to hit the site since -let’s face it- there was nothing groundbreaking. Spincloud was born from my idea of having access to the world weather in one step or less and I’m pleased with the results so far. Spincloud gains adoption every day and this is only due to the people that saw something new and exciting in it. There are blog posts, traffic graphs and other goodies to talk about so let’s begin.

I announced spincloud’s launch on my personal blog then Keir from googlemapsmania broke the news to the community. And so the first spike of traffic was logged on Jan 2nd. 

jan_traffic

Then followed a month of relative silence during which I took a much needed break from the project. I couldn’t help but refactoring and cleaning-up the code a bit. I had two minor releases deployed in prod in the mean time and everything went smooth.

I had a new feature I wanted to have online before the initial release, the meteoalarm mashup that displays weather warnings across Europe but chose not to delay it (a Good Decision in hindsight) . I started working on it by mid January , did a lot of digging for administrative borders data then getting the meteoalarm warning data right (those guys don’t have any integration point). I was done two weeks later.

In the mean time I was trying to raise the awareness about spincloud. I submitted the site so some mapping/mashup themed websites, improved the indexing (still working on it). Then Wednesday last week the European bloggers finally found it. UK’s Mapperz ran a nice little story that seems to have set the things in motion. A number of other bloggers picked-up the story and a surprisingly good feedback came from Italy: Matteo and italiasw.com did a great job while  gisdk.blogspot.com of Denmark  noticed the RSS facilities. Then the referral machine kicked in,  killerstartups.com, programmableweb.com amongst the sites linking. And thus the second spike registered on the traffic meter:
Continue reading “Spincloud, a month old: looking good”

@Meteo_alarm twitter stream now available

ma_logo   =   ma_sc_logo
toplogo
twitter_logo

Update Nov.25.2009: The twitter name has changed to @meteo_alarm. Please update your bookmark or follow list.

Between code refactoring and cooking new ideas, I took some time to add a new feature to Spincloud. Inspired by the crowd-sourced weather updates triggered by really bad snow storms in the UK, I (finally) started to see the advantages to using Twitter. So I took the Meteoalarm data that I mashed-up on Spincloud’s map and streamed it to Twitter.

So here it is: Meteoalarm on Twitter via Spincloud = @Meteoalarm @Meteo_alarm. The stream is updated twice a day and it presents a concise text-only representation of the color encoded weather warnings  in Europe for today and tomorrow.