From Cloud to GIS: Getting Weather Data.

Introduction

In this post we are going to look at how to get weather forecast data and display it as a layer in a desktop geographical information system (GIS). The tools that you need are:

• A desktop GIS. I am going to use QGIS (free software) but I have done this with ESRI ArcMap as well
• A spreadsheet application. I will be using MS Excel but I have also tried this with Softmaker Office and I am sure LibreOffice (free software) will work or indeed any other spreadsheet application
• The Natural Earth Quick Start data set. This is a collection of public domain map data that describe the planet, (http://www.naturalearthdata.com/downloads/).   You only need this if you don’t already have some data describing coastlines, countries etc.

We are going to use weather forecast data from the Global Forecast System (GFS) of the USA’s National Weather Service (NWS).

GFS is a global numerical weather prediction system that is run 4 times a day on a powerful super computer.  The forecast data is released under an OpenData licence and is accessible directly at the following address

http://nomads.ncep.noaa.gov/

However if you navigate to the above link you will be confronted with data files in some difficult to use scientific formats such as GRIB (Gridded Binary). Fortunately the GFS forecast is also available on an ERDDAP server. ERDDAP is Cloud based data broker designed to act as the intermediary between developers/users and scientific data formats (particularly gridded data). Please see my previous post for an introduction to ERDDAP.

Getting the Data

Navigate one of the following ERDDAP servers:

• https://bluehub.jrc.ec.europa.eu/erddap/index.html
• http://oos.soest.hawaii.edu/erddap/index.html

On the landing page enter ‘gfs’ into the search box’.

This will return two data sets, select the GFS one by clicking on the  ‘Graph’, this will bring you  to the data query form (see figure) which we can use to get data and to construct a URL query string that can be reused at later date.

On the query (graph) form web page make following changes:

• Graph type to Vector
• Vector X to ugrd10m
• Vector Y to vgrd10m
• Click on the map to get the data for area of the Earth that you are interested in (try the North Atlantic for this example).
• Optional: change the ‘Stop’ time. This is time of forecast and it is default at last time of the forecast, click on the ’-’ to move the time backwards
• File type to CSV (comma separated values file)

Once you have made the necessary changes click on the ‘Download Data or Image’ button to get your data. Copy the data from where it downloaded to a working directory. Below is a link to the query  I used when I was writing this post.

https://bluehub.jrc.ec.europa.eu/erddap/griddap/NCEP_Global_Best.htmlTable?ugrd10m[(2016-04-25T18:00:00Z)][(34.0):(70.5)][(323.0):(359.5)],vgrd10m[(2016-04-25T18:00:00Z)][(34.0):(70.5)][(323.0):(359.5)]&.draw=vectors&.vars=longitude|latitude|ugrd10m|vgrd10m&.color=0x000000

Processing the Data

The next stage is to convert the data into a format that we can use in the GIS. Firstly we need to do the following:

• Import the CSV into your spreadsheet of choice. The delimiter is Comma
• Delete the second row of data
• Delete the time column, we won’t be using it
• Save the file in a native format of the spreadsheet (csv is text format)

You should now have the following four columns of data:

• latitude: This data is fine and won’t need processing
• longitude: This data needs to be converted to a format usable by the GIS
• ugrd10m & vgrd10m: These two columns represent the U and V component of the wind vector and we need to use these to calculate the wind speed and wind direction

Conversion of the longitude parameter

GIS programs all want data with longitude values within -180 to 180, but many data sets have longitude values within 0 to 360. So if the data set already has longitude -180 to 180 values you do not need to do this conversion. However in this example we do need to convert.

Longitude in the GFS model output starts at the Greenwich meridian 0 degrees and moves east at regular increments up to 359.5 degrees. This representation needs to be converted to more usual:

• -180 to 0 to 180

Add the following equation to an empty column, in the first cell enter New_Long

= IF(MOD(B2,360) < 180, MOD(B2,360),MOD(B2-180,360)-180)

Where B2 is the original longitude column

Now calculate the new longitude by dragging the equation cell down the spread sheet to last row of longitude data.  A quick way of doing this is to select the cell with the equation, then Shift + Left Click the last ell in the column that you want to populate to and finally Ctrl D (this is the process works in Excel).

Calculating the Wind Direction

To calculate the wind direction we need to apply the arctangent (atan2) with the arguments U & V.

• https://www.eol.ucar.edu/content/wind-direction-quick-reference

While that atan2 function is available in most spreadsheets and programming languages it is not ubiquitous

• (https://en.wikipedia.org/wiki/Atan2).

The format of atan2 in spreadsheets is different from that used in programming languages and some GIS application in that it has the parameters reversed.

• Atan2 (vgrd10m, ugrd10m)

So to calculate wind direction enter the following equation into an empty cell of row 2:

=Atan2 (D2, C2)

where D2 is column vgrd10m and C2 is ugrd10m
Populate the column by dragging the equation down to the last row with data. Enter the name Wind_Dir into the first cell of the column.

Calculating the Wind Speed

The steps for calculating wind speed should be a familiar process now.  This time enter the following equation into an empty cell of row 2:

=SQRT(C2 * C2 + D2 * D2)

where C2 is ugrd10m and D2 is vgrd10m
Populate the column down to the last data row and put the name Wind_Speed into the first cell of the column.

Collating the Data

Open a new spreadsheet in you worksheet and copy the following columns to the spreadsheet:

• latitude
• longitude, use the new longitude data, make sure to paste just the values instead of the equation and values
• Wind_Dir, make sure to paste just the values instead of the equation and values
• Wind_Speed, make sure to paste just the values instead of the equation and values

Give the spreadsheet a name and save it as CSV file.

Importing the Data into a GIS

Finally we come to section where we import the wind data in a GIS and display it as a layer. In many ways this step is the simplest since all the difficult data wrangling has already been done.

To start open a new blank project in QGIS (or ArcMap, the process is very similar) and click on the ‘Add Delimited Text Layer’ button which is usually found on the left hand side of the screen. This will trigger ‘Create a Layer from a Delimited Text File’ form.  Click on the ‘Browse’ button and navigate to, and select, the CSV file that contains the processed data. Check everything on the ‘Create a Layer…’ form making sure that the CSV option is selected and that the x field is longitude and the y field is latitude. Click on the OK button select ‘WGS 84’ from the coordinate reference systems of the world.

Displaying the Data

You should now see the data displayed as points on the GIS layer.  On the layers panel right click on the layer and select the properties option. Do the following steps:

• Change the ‘Single Symbol’ option to ‘Graduated’
• In the Column option choose Wind_Speed (or whatever you named you wind speed column)
• Click on the Symbol and choose the simple marker of and arrow pointing up. Now change the symbol layer type to ‘Font marker’ scroll down to the bottom of the font grid and select the arrow pointing upwards.
• Now select a field for the rotation of the arrow, this is to the right hand side of the rotation text box, the field you need to activate should be Wind_Dir, unless you have used a different name.
• Click ‘OK’
• Back at the properties section choose:
  • A colour ramp
  • Equal Interval as the mode in the Classes section
  • Click on the ‘Classify’ button
  • Click on the ‘OK’ button

When you are returned to the GIS the layer should be populated by arrows which are pointing in the direction of wind flow, the arrows will also be coloured based on wind speed.

Adding Some Geographical Context

To put the wind data into geographical context we can now add a layer of global coastline data. This is contained in the Natural Earth data folder that you download earlier. Click on the ‘Add Vector Layer’ button in the left hand side of QGIS and navigate to the Natural Earth Data folder and select the file ‘ne_10m_coastline.shp’.

Conclusion and Next Steps

A small bit of effort will greatly improve the presentation of the wind data. In the figure below I added some data layers from the Natural Earth data resource and tweaked the colours for wind speed.

The work flow involved in adding other GFS parameters is very similar to what I have shown here, so why not try adding more forecast parameters? The available parameters are here:

• https://bluehub.jrc.ec.europa.eu/erddap/info/NCEP_Global_Best/index.html

Another option is to add data from other models, once again so long as you are using ERRDAP the work flow is very similar to what I have explained here. As a suggestion you can try the following:

• Wave forecast from the WaveWatchIII model
  •  https://bluehub.jrc.ec.europa.eu/erddap/griddap/NWW3_Global_Best.graph
• Sea Surface Temperature from the Real Time Ocean Forecast System
  •  https://bluehub.jrc.ec.europa.eu/erddap/griddap/noaa_pfeg_aa87_63b5_f12d.graph
• Chlorophyll from the MODIS satellite
  • http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdVH3chlamday.graph

So long as you pay attention to dates and projections you too can be using GIS to make ‘mashups with heterogeneous data’! Enjoy!