Tag: weather

This has been a long time coming, five years actually. Spincloud went live in January 2009 and I’ve added the temperature heatmap overlay a few months later.
The heatmap and the corresponding map overlay is generated once an hour and it’s been faithfully doing so for six years. It does it by first generating an global temperature heatmap then cutting tiles (up to zoom level 6). Every hour.
But what I have also done back then was to also save one heat map image once a day. The idea was to generate a timelapse at some point, showing a visualization of the global temperatures over a longer span of time.
Now it’s time to show the result: below are 5 years of global temperatures in a one minute timelapse (make sure to switch to the HD version if not enabled by default):

_{* I started collecting this data in Aug.2009 but the lapse above starts in in July 2010 as I have a data gap between Dec.2009 and Jul.2010. Frankly I can’t remember why.}

The timelapse neatly shows the SYNOP and METAR coverage globally. The weather stations behind these data sources report at various times of day and I’ve figured that 19:00 GMT is the hour in the day with the most reporting stations. Looks like this has changed about a year ago if you look at Russia’s remote locations that are not covered well anymore at this time of day. The rest of the landmass is reasonably well covered.

I think it’s interesting to explain how I have actually generated the temperature heat map. The code you’ll see is pre Java 8 since I haven’t upgraded the code much in the past years.

Spincloud uses several sources of weather data and it’s quite remarkable how little they have changed over the past 6 years. I witnessed some data sources going offline but the core of the data still comes from the same sources. For the heatmap I use the METAR and SYNOP global data, mostly coming from the NWS servers who are providing clean and reliable data for many years. The data is stored in the local database and it essentially contains a map point, a temperature and a timestamp.
With these sources at hand, the logic to generate a heatmap image is as follows:
1. Iterate over each pixel on the global map and get the respective temperature. Only include land masses.
2. Interpolate that temperature with the temperatures of the nearest locations where there are temperature readings
3. Generate an 2×2 pixel rectangular area with a color that corresponds with the interpolated temperature. Only fill land masses in order to make the overlay look realistic.
4. Append that image in memory to the global heat map image in the correct location
5. Repeat until complete then dump the generated image to disk

Turns out that at any the temperature data points are less than 20,000 globally and so I can add all of them in a list in memory. In step 1, when iterating over each map pixel this list is looked-up for temperature points in vicinity.
The map mask referenced in the code below, the worldmap-mask.png looks like this:

Here’s the code (warning: not executable):

public void generateHeatMap() {
  //Collect all temperature points
  List metaPoints = getMetaPoints();
  BufferedImage heatMap;

   //Load the world map mask in memory; all black pixels belong to landmasses.
   //We'll use it to filter-in only land masses.
  try {
    heatMap = ImageIO.read(this.getClass().getClassLoader().getResourceAsStream("worldmap-mask.png"));
  } catch(Exception e) {
    logger.error("Exception: " + e.getMessage(), e);
    throw new RuntimeException("worldmap-mask.png not found");
  }
  Graphics2D g = (Graphics2D) heatMap.getGraphics();
  
  int blackRGB = Color.black.getRGB();
  //1. take every second pixel then scan the entire map surface
  int step = 2;
  for (int x = 0; x < MapUtil.MAP_WIDTH; x += step) {
    for (int y = 0; y < MapUtil.MAP_HEIGHT; y += step) {
      //only draw over land mass (masked with black).
      //seas/oceans will be left white.
      if(heatMap.getRGB(x, y) == blackRGB) {
      //Get the nearest temperature points along with their relative distance to the current map point
      SortedMap nearestPts = findNearest(metaPoints, x, y);
      //2. Compute the interpolated temperature using an inverse distance algorithm:
      // https://en.wikipedia.org/wiki/Inverse_distance_weighting
      int interpolatedT = (int)getInterpolatedTemperature(x, y, nearestPts);
      try {
          //identify the pixel color for this temperature
          g.setColor(getTempColor(interpolatedT));
      } catch(Exception e) { //skip exception, at worst there'll be gaps in th heatmap
          logger.info("Exception: " + e.getMessage(), e);
          logger.info("The temperature was: " + interpolatedT);
      }
        //3,4. Generate a step x step rectangle at current coordinates
        g.fillRect(x, y, step, step);
      }
    }
  }

  //5. Save image to disk. Done every day at 19:00 GMT only.
  saveLapseImageToDisk();
  //For Google maps overlay
  generateTiles(heatMap);

}

The second resource used and buried behind the call to getTempColor is the color temperature scale.

The code that gets a temperature and figures its color is this:

//init code:
  private int[] colorPixels = null;
  colorPixels = new int[colorMapW * colorMapH];
  PixelGrabber pg = new PixelGrabber(heatColorImage, 0, 0, colorMapW, colorMapH, colorPixels, 0, colorMapW);
  pg.grabPixels();
..

private Color getTempColor(int temp) {
  if (temp == -9999) {
    return Color.white;
  }
  // 273 px, 39 gradations each 7 px. starting -60 ends +60
  int pos = (273 / 120) * (temp + 60);

  int c = colorPixels[(colorMapH / 2) * colorMapW + pos];
  int red = (c & 0x00ff0000) >> 16;
  int green = (c & 0x0000ff00) >> 8;
  int blue = c & 0x000000ff;
  // and the Java Color is ...
  Color cl = new Color(red, green, blue);
  return cl;
}

Currently there are 1774 useful images collected so far and still going. Two years back I have moved Spincloud to DigitalOcean (note: referral link) and kept collecting this historical data without a hitch.

To geneate the timelapse I used ffmpeg and this tutorial. You’ll notice the month-year embedded in the video, I have used used this howto to get them in and this to figure the subtitle format. There’s some code I wrote to generate the subtitles file to be in sync with the timelapse but it’s too boring to include.

As a side note, Spincloud runs a total of 8 background jobs collecting temperature and forecast data, generating temperature and radar tiles, and weather warning data, all on the cheapest DataOcean plan.