From Cloud to GIS: Getting Weather Data.

Introduction

In this post we are going to look at how to get weather forecast data and display it as a layer in a desktop geographical information system (GIS). The tools that you need are:

  • A desktop GIS. I am going to use QGIS (free software) but I have done this with ESRI ArcMap as well
  • A spreadsheet application. I will be using MS Excel but I have also tried this with Softmaker Office and I am sure LibreOffice (free software) will work or indeed any other spreadsheet application
  • The Natural Earth Quick Start data set. This is a collection of public domain map data that describe the planet, (http://www.naturalearthdata.com/downloads/).   You only need this if you don’t already have some data describing coastlines, countries etc.

We are going to use weather forecast data from the Global Forecast System (GFS) of the USA’s National Weather Service (NWS).

GFS is a global numerical weather prediction system that is run 4 times a day on a powerful super computer.  The forecast data is released under an OpenData licence and is accessible directly at the following address

http://nomads.ncep.noaa.gov/

However if you navigate to the above link you will be confronted with data files in some difficult to use scientific formats such as GRIB (Gridded Binary). Fortunately the GFS forecast is also available on an ERDDAP server. ERDDAP is Cloud based data broker designed to act as the intermediary between developers/users and scientific data formats (particularly gridded data). Please see my previous post for an introduction to ERDDAP.

Getting the Data

Navigate one of the following ERDDAP servers:

On the landing page enter ‘gfs’ into the search box’.

This will return two data sets, select the GFS one by clicking on the  ‘Graph’, this will bring you  to the data query form (see figure) which we can use to get data and to construct a URL query string that can be reused at later date.

The query form for the GFS data
The query form for the GFS data, parameters to change are highlighted with green.

On the query (graph) form web page make following changes:

  • Graph type to Vector
  • Vector X to ugrd10m
  • Vector Y to vgrd10m
  • Click on the map to get the data for area of the Earth that you are interested in (try the North Atlantic for this example).
  • Optional: change the ‘Stop’ time. This is time of forecast and it is default at last time of the forecast, click on the ’-’ to move the time backwards
  • File type to CSV (comma separated values file)

Once you have made the necessary changes click on the ‘Download Data or Image’ button to get your data. Copy the data from where it downloaded to a working directory. Below is a link to the query  I used when I was writing this post.

https://bluehub.jrc.ec.europa.eu/erddap/griddap/NCEP_Global_Best.htmlTable?ugrd10m[(2016-04-25T18:00:00Z)][(34.0):(70.5)][(323.0):(359.5)],vgrd10m[(2016-04-25T18:00:00Z)][(34.0):(70.5)][(323.0):(359.5)]&.draw=vectors&.vars=longitude|latitude|ugrd10m|vgrd10m&.color=0x000000

Processing the Data

The next stage is to convert the data into a format that we can use in the GIS. Firstly we need to do the following:

  • Import the CSV into your spreadsheet of choice. The delimiter is Comma
  • Delete the second row of data
  • Delete the time column, we won’t be using it
  • Save the file in a native format of the spreadsheet (csv is text format)

You should now have the following four columns of data:

  • latitude: This data is fine and won’t need processing
  • longitude: This data needs to be converted to a format usable by the GIS
  • ugrd10m & vgrd10m: These two columns represent the U and V component of the wind vector and we need to use these to calculate the wind speed and wind direction

Conversion of the longitude parameter

GIS programs all want data with longitude values within -180 to 180, but many data sets have longitude values within 0 to 360. So if the data set already has longitude -180 to 180 values you do not need to do this conversion. However in this example we do need to convert.

Longitude in the GFS model output starts at the Greenwich meridian 0 degrees and moves east at regular increments up to 359.5 degrees. This representation needs to be converted to more usual:

  • -180 to 0 to 180

Add the following equation to an empty column, in the first cell enter New_Long

= IF(MOD(B2,360) < 180, MOD(B2,360),MOD(B2-180,360)-180)

Where B2 is the original longitude column

Now calculate the new longitude by dragging the equation cell down the spread sheet to last row of longitude data.  A quick way of doing this is to select the cell with the equation, then Shift + Left Click the last ell in the column that you want to populate to and finally Ctrl D (this is the process works in Excel).

Calculating the Wind Direction

To calculate the wind direction we need to apply the arctangent (atan2) with the arguments U & V.

While that atan2 function is available in most spreadsheets and programming languages it is not ubiquitous

The format of atan2 in spreadsheets is different from that used in programming languages and some GIS application in that it has the parameters reversed.

  • Atan2 (vgrd10m, ugrd10m)

So to calculate wind direction enter the following equation into an empty cell of row 2:

=Atan2 (D2, C2)

where D2 is column vgrd10m and C2 is ugrd10m
Populate the column by dragging the equation down to the last row with data. Enter the name Wind_Dir into the first cell of the column.

Calculating the Wind Speed

The steps for calculating wind speed should be a familiar process now.  This time enter the following equation into an empty cell of row 2:

=SQRT(C2 * C2 + D2 * D2)

where C2 is ugrd10m and D2 is vgrd10m
Populate the column down to the last data row and put the name Wind_Speed into the first cell of the column.

Collating the Data

Open a new spreadsheet in you worksheet and copy the following columns to the spreadsheet:

  • latitude
  • longitude, use the new longitude data, make sure to paste just the values instead of the equation and values
  • Wind_Dir, make sure to paste just the values instead of the equation and values
  • Wind_Speed, make sure to paste just the values instead of the equation and values

Give the spreadsheet a name and save it as CSV file.

Importing the Data into a GIS

Finally we come to section where we import the wind data in a GIS and display it as a layer. In many ways this step is the simplest since all the difficult data wrangling has already been done.

To start open a new blank project in QGIS (or ArcMap, the process is very similar) and click on the ‘Add Delimited Text Layer’ button which is usually found on the left hand side of the screen. This will trigger ‘Create a Layer from a Delimited Text File’ form.  Click on the ‘Browse’ button and navigate to, and select, the CSV file that contains the processed data. Check everything on the ‘Create a Layer…’ form making sure that the CSV option is selected and that the x field is longitude and the y field is latitude. Click on the OK button select ‘WGS 84’ from the coordinate reference systems of the world.

Displaying the Data

You should now see the data displayed as points on the GIS layer.  On the layers panel right click on the layer and select the properties option. Do the following steps:

  • Change the ‘Single Symbol’ option to ‘Graduated’
  • In the Column option choose Wind_Speed (or whatever you named you wind speed column)
  • Click on the Symbol and choose the simple marker of and arrow pointing up. Now change the symbol layer type to ‘Font marker’ scroll down to the bottom of the font grid and select the arrow pointing upwards.
  • Now select a field for the rotation of the arrow, this is to the right hand side of the rotation text box, the field you need to activate should be Wind_Dir, unless you have used a different name.
  • Click ‘OK’
  • Back at the properties section choose:
    • A colour ramp
    • Equal Interval as the mode in the Classes section
    • Click on the ‘Classify’ button
    • Click on the ‘OK’ button

When you are returned to the GIS the layer should be populated by arrows which are pointing in the direction of wind flow, the arrows will also be coloured based on wind speed.

Adding Some Geographical Context

To put the wind data into geographical context we can now add a layer of global coastline data. This is contained in the Natural Earth data folder that you download earlier. Click on the ‘Add Vector Layer’ button in the left hand side of QGIS and navigate to the Natural Earth Data folder and select the file ‘ne_10m_coastline.shp’.

The final result
The final result

Conclusion and Next Steps

A small bit of effort will greatly improve the presentation of the wind data. In the figure below I added some data layers from the Natural Earth data resource and tweaked the colours for wind speed.

A small bit of effort to improve presentation
A small bit of effort to improve presentation

The work flow involved in adding other GFS parameters is very similar to what I have shown here, so why not try adding more forecast parameters? The available parameters are here:

Another option is to add data from other models, once again so long as you are using ERRDAP the work flow is very similar to what I have explained here. As a suggestion you can try the following:

So long as you pay attention to dates and projections you too can be using GIS to make ‘mashups with heterogeneous data’! Enjoy!

0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

34 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Paolo Cavallini
Paolo Cavallini
7 years ago

Thanks. Why not consolidating all this into a QGIS plugin?

Conor Delaney
Conor Delaney
7 years ago

That is a great idea, I will see what I can do.

Paolo Cavallini
Paolo Cavallini
7 years ago
Reply to  Conor Delaney

Glad you like it. If you need help, just let me know.

Lutra
Lutra
7 years ago

In the section for downloading it as CSV, you can choose NetCDF format. Crayfish plugin for QGIS will allow you to display and animate the format without any conversion:

https://github.com/lutraconsulting/qgis-crayfish-plugin
http://www.lutraconsulting.co.uk/products/crayfish/

Conor Delaney
Conor Delaney
7 years ago
Reply to  Lutra

Very nice. I thought there must be a suitable plugin somewhere. I hope that you are finding the ERDDAP technology useful?

Lutra
Lutra
7 years ago
Reply to  Conor Delaney

We haven’t used it yet but good to know the service.

MH Ali
MH Ali
7 years ago

Thank u
how can we relate different parameters(wind
pressure,humidity,presipitation,rain falling) with temperature in GIS
(Arcmap). I mean i want to find the trend of change in temperature of a region
w.r.t. above given parameters.

Conor Delaney
Conor Delaney
7 years ago
Reply to  MH Ali

Hi, if you follow link below you find the reanalysis data. This is project where the current weather models are run using historical data, in effect running the model backwards. It is done to fill in the geographical gaps. This data set goes back to 1948, since it available via ERDDAP the techniques I have explained in both my articles here will be enough to get started with finding the data you want, remember the metadata will give you a description of the parameters. http://coastwatch.pfeg.noaa.gov/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=reanalysis The ECMWF also have a reanalysis project: http://www.ecmwf.int/en/research/climate-reanalysis However the data is not on ERDDAP… Read more »

MH Ali
MH Ali
7 years ago
Reply to  Conor Delaney

thank u

Felipe
Felipe
7 years ago

In the next International QGIS Conference, that will take place in Girona (Spain), my collegue Mauricia will explain a new plug in that will be realesed soon, the title of the talk is “THREDDS Explorer plug-in, bringing meteoceanographic operational data into QGIS” you can hace a look to the abstract here.. http://www.sigte.udg.edu/jornadassiglibre/international-qgis-user-and-developer-conference/conferencia-qgis/

Conor Delaney
Conor Delaney
7 years ago
Reply to  Felipe

Great stuff! I look forward to seeing that.

Felipe
Felipe
7 years ago
Reply to  Conor Delaney
Conor Delaney
Conor Delaney
7 years ago
Reply to  Felipe

Great stuff I look forward to experimenting with this.

Ilya Khamushkin
Ilya Khamushkin
7 years ago

That’s a pretty detailed and low-level tutorial! Love it! Excuse me for bringing my own stuff inside this topic 🙂 I’m eager to learn about struggles that data engineers and scientists are having with data processing and generally data access. I’m working at PlanetOS, where we created a simple point based (probe) API to access similar climate/weather datasets (we have GFS, NWW3, HYCOM and bunch of others). If anybody is up to share some feedback, please, let me know. You can learn more about available datasets at http://data.planetos.com Links to API documentation, examples (Python, JS) are shared in this blog… Read more »

Conor Delaney
Conor Delaney
7 years ago

Hi Ilya, thanks for youy comment. I have only been able to have brief look at the PlanetOS api. I was looking at GFS (NCEP) and the first thing that struck me was how do I get two parameters at the same time? I was trying to recreate the workflow I described above but using PlanetOS instead of ERDDAP. I am sure it can be done, I just haven’t had enough free time to look at it.

Hewerton da silva
Hewerton da silva
7 years ago

Dear Conor Delaney,

I would like to congratulate you for the explanation. I’m finding it difficult to generate the wind field direction in Qgis, I can not identify whether the error is in the formula. I am a doctoral student at the Federal University of Pernambuco in Brazil and my research is to identify natural seeps of oil in the Plateau of Pernambuco.

I thank you.

Conor Delaney
Conor Delaney
7 years ago

Hi Hewerton, I only just saw your question. Are are you still having the problem?

Hewerton da silva
Hewerton da silva
7 years ago
Reply to  Conor Delaney

Hi Conor Delaney,

I’m not able to rotate the vectors to give the wind direction, as tenten several times and it still fails. I’ve redid the calculation several times and the results are the same and also I do not know if I’m erring on Qgis, no longer know what to do. I send you herewith a part of my spreadsheet.

I plan to use this data for my doctoral research.

Thank you for helping me.

Conor Delaney
Conor Delaney
7 years ago

I haven’t have a chance to look at this. Can you send me the are of interest you are looking at so that I can see if I can reproduce the problem.

Conor Delaney
Conor Delaney
7 years ago

Hi Hewerton, sorry for the delay. I think I know what the problem is. First of all I haven’t been able to work with GFS data as the servers in Hawaii seem to be slow today. So I worked with RTOFS data instead (it is hosted on a different server) and I noticed that QGIS will not allow you to set a wind direction field if #Value! appears anywhere in the column, I did a CTRL-H on the spreadsheet and replaced them all with 0. Once I did this I could set the wind_dir (and wind_speed) fields in the layer.… Read more »

Javier
Javier
7 years ago

Hi Conor,

Congrats for this wonderful post! I’m a PhD student from Madrid, Spain, and I’m interested in fungal spores dispersion through wind, animals, etc. I was using the information of your post some weeks ago to perform an R script to automatize all the tasks that you describe here, but today, suddenly, the gfs wind database has disappeared! It is only the sea wave data… do you have some information? Thanks in advance anyway! And congrats again or this great job!
Cheers
Javi

Javi Fernández-López
Javi Fernández-López
7 years ago
Reply to  Javier

The server is up again, thanks! 🙂
Cheers,
Javi

Conor Delaney
Conor Delaney
7 years ago

Sorry in the delay in getting back to you. With ERDDAP when a remote data server goes down it is automatically removed from the catalog. ERDDAP keeps contacting the remote server until it is ready again and then adds the data back into the catalog.

You might find this data useful as well : http://coastwatch.pfeg.noaa.gov/erddap/griddap/esrlNcepRe.graph

It is the global reanalysis weather forecast data back to 1948.

Cheers

Conor

Javi Fernández-López
Javi Fernández-López
7 years ago

Dear Community! I was following this great post and I was developing several functions in R language to automatize this tasks. I put all of them in an R package called rWind, hosted in CRAN repository. Here you have a brief tutorial to use it!

http://allthiswasfield.blogspot.com.es/2016/12/rwind-r-package-released.html

Cheers, Javi

Conor Delaney
Conor Delaney
7 years ago

Fantastic work Javi, I just tweeted the link

Javi Fernández-López
Javi Fernández-López
7 years ago
Reply to  Conor Delaney

Thanks for the feedback Conor! And thanks for the global reanalysis weather forecast data, I’m thinking to change my R functions to obtain the data directly from this database… are there many differences between them, apart dates?

I’m interested also in global sea currents… do you know any general database from ERDDAP to get this data? I’m working to implement also some functions to compute least cost paths having into account this kind of data to perform anisotropic analysis, I’ll add to my package as soon as possible… I’ll inform you!

Conor Delaney
Conor Delaney
7 years ago

Hi, the GFS data in ERDDAP is only updated daily, so if you go straight to the source (http://nomads.ncep.noaa.gov) You will get data at higher resolutions. Any some stage this data will be on ERDDAP. As to sea currents, the two modelling sources that are available to you are the Real Time Ocean Forecast System (RTOFS) from NOAA, which is on ERDDAP or the EU equivalent which can be found here http://marine.copernicus.eu/services-portfolio/access-to-products/. Looking forward to seeing your new functions. By the way if you start using RTOFS please be aware that the the geographic grid in the data is different… Read more »

Javi Fernández-López
Javi Fernández-López
6 years ago
Reply to  Conor Delaney

Hi again Conor! I’m back! Finally I put the new function in the rWind package to find the shortest path having into account wind speed and direction! Here is the github repository, the function is “flow.dispersion”. I have not posted anything about it yet, but I will in the next months! In the other hand, I didn’t find the Real Time Ocean Forecast System (RTOFS) data base that you told me… I was making some work with this one: “Currents, Geostrophic, Aviso, 0.25 degrees, Global, 1992-2012 (1 Day Composite” but it goes only until 2012… do you know what is… Read more »

Conor Delaney
Conor Delaney
6 years ago

Great Stuff! The RTOFS data is on this ERDDAP server http://coastwatch.pfeg.noaa.gov/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=rtofs There are forecast, nowcast, prognostic and diagnostic flavours (I can’t remember what they all mean, sorry). Not all the models have the same parameters. Also data near the poles use a different grid. All the information that you need is in the meta data. There are other models available, just enter the parameter that you are interested in into the search box on the ERDDAP landing page. http://coastwatch.pfeg.noaa.gov/erddap/index.html ERDDAP is getting used by a lot of National Ocean Data Centres, if you use Google to search for ERDDAP and… Read more »

Lucie Fialová
Lucie Fialová
6 years ago

Hello,

Perfect contribution. It was very useful for me. But I need your help. I do not know how to solve the following problem. I want to create a wind map and I have these data (attachment: comment image )

The data is from one meteorological station and therefore I have only one longitude and latitude value (48.59278 °, 20.09361 °). Is it possible to create a wind map from these data and display it in QGIS, similarly like in your post?

Thank you for answer.

Lucka.

Parwati Sofan
Parwati Sofan
3 years ago

Thank you so much for sharing this info. It’s really helpful. I can plot the wind direction successfully using your guidance. I did it in ArcMap.

Josh Carter
Josh Carter
2 years ago

I wanted to review your steps of selecting an up arrow and formatting, so many options are not sure what the instructions should be. I start with selecting a symbol, then selecting Simple Marker (note tried to attach image but failed). Then the instructions say “Now change the symbol layer type to ‘Font marker’ scroll down to the bottom of the font grid and select the arrow pointing upwards” But don’t we need to know what font to use, I tried wingding3? What font are we selecting an up arrow from? Thanks P.S., I am using data from https://nomads.ncep.noaa.gov/ and… Read more »

Last edited 2 years ago by Josh Carter
Josh Carter
Josh Carter
2 years ago

Note, I have been trying to edit my own question. I did get quivers or wind vectors to work.

I did select simple marker -> Font Marker -> select Font Type “wingding3” and select wind_dir as my rotation variable and wind_speed in my classes to do color-ramp.

note only difference was getting grib2 data from NOAA’s GFS dataset, using 700mb and using pygrib to extract U,V components which then transformed to wind_speed, wind_dir.

thanks for getting me going. all