In this post we are going to look at how to get weather forecast data and display it as a layer in a desktop geographical information system (GIS). The tools that you need are:
- A desktop GIS. I am going to use QGIS (free software) but I have done this with ESRI ArcMap as well
- A spreadsheet application. I will be using MS Excel but I have also tried this with Softmaker Office and I am sure LibreOffice (free software) will work or indeed any other spreadsheet application
- The Natural Earth Quick Start data set. This is a collection of public domain map data that describe the planet, (http://www.naturalearthdata.com/downloads/). You only need this if you don’t already have some data describing coastlines, countries etc.
We are going to use weather forecast data from the Global Forecast System (GFS) of the USA’s National Weather Service (NWS).
GFS is a global numerical weather prediction system that is run 4 times a day on a powerful super computer. The forecast data is released under an OpenData licence and is accessible directly at the following address
However if you navigate to the above link you will be confronted with data files in some difficult to use scientific formats such as GRIB (Gridded Binary). Fortunately the GFS forecast is also available on an ERDDAP server. ERDDAP is Cloud based data broker designed to act as the intermediary between developers/users and scientific data formats (particularly gridded data). Please see my previous post for an introduction to ERDDAP.
Getting the Data
Navigate one of the following ERDDAP servers:
On the landing page enter ‘gfs’ into the search box’.
This will return two data sets, select the GFS one by clicking on the ‘Graph’, this will bring you to the data query form (see figure) which we can use to get data and to construct a URL query string that can be reused at later date.
On the query (graph) form web page make following changes:
- Graph type to Vector
- Vector X to ugrd10m
- Vector Y to vgrd10m
- Click on the map to get the data for area of the Earth that you are interested in (try the North Atlantic for this example).
- Optional: change the ‘Stop’ time. This is time of forecast and it is default at last time of the forecast, click on the ’-’ to move the time backwards
- File type to CSV (comma separated values file)
Once you have made the necessary changes click on the ‘Download Data or Image’ button to get your data. Copy the data from where it downloaded to a working directory. Below is a link to the query I used when I was writing this post.
Processing the Data
The next stage is to convert the data into a format that we can use in the GIS. Firstly we need to do the following:
- Import the CSV into your spreadsheet of choice. The delimiter is Comma
- Delete the second row of data
- Delete the time column, we won’t be using it
- Save the file in a native format of the spreadsheet (csv is text format)
You should now have the following four columns of data:
- latitude: This data is fine and won’t need processing
- longitude: This data needs to be converted to a format usable by the GIS
- ugrd10m & vgrd10m: These two columns represent the U and V component of the wind vector and we need to use these to calculate the wind speed and wind direction
Conversion of the longitude parameter
GIS programs all want data with longitude values within -180 to 180, but many data sets have longitude values within 0 to 360. So if the data set already has longitude -180 to 180 values you do not need to do this conversion. However in this example we do need to convert.
Longitude in the GFS model output starts at the Greenwich meridian 0 degrees and moves east at regular increments up to 359.5 degrees. This representation needs to be converted to more usual:
- -180 to 0 to 180
Add the following equation to an empty column, in the first cell enter New_Long
= IF(MOD(B2,360) < 180, MOD(B2,360),MOD(B2-180,360)-180)
Where B2 is the original longitude column
Now calculate the new longitude by dragging the equation cell down the spread sheet to last row of longitude data. A quick way of doing this is to select the cell with the equation, then Shift + Left Click the last ell in the column that you want to populate to and finally Ctrl D (this is the process works in Excel).
Calculating the Wind Direction
To calculate the wind direction we need to apply the arctangent (atan2) with the arguments U & V.
While that atan2 function is available in most spreadsheets and programming languages it is not ubiquitous
The format of atan2 in spreadsheets is different from that used in programming languages and some GIS application in that it has the parameters reversed.
- Atan2 (vgrd10m, ugrd10m)
So to calculate wind direction enter the following equation into an empty cell of row 2:
=Atan2 (D2, C2)
where D2 is column vgrd10m and C2 is ugrd10m
Populate the column by dragging the equation down to the last row with data. Enter the name Wind_Dir into the first cell of the column.
Calculating the Wind Speed
The steps for calculating wind speed should be a familiar process now. This time enter the following equation into an empty cell of row 2:
=SQRT(C2 * C2 + D2 * D2)
where C2 is ugrd10m and D2 is vgrd10m
Populate the column down to the last data row and put the name Wind_Speed into the first cell of the column.
Collating the Data
Open a new spreadsheet in you worksheet and copy the following columns to the spreadsheet:
- longitude, use the new longitude data, make sure to paste just the values instead of the equation and values
- Wind_Dir, make sure to paste just the values instead of the equation and values
- Wind_Speed, make sure to paste just the values instead of the equation and values
Give the spreadsheet a name and save it as CSV file.
Importing the Data into a GIS
Finally we come to section where we import the wind data in a GIS and display it as a layer. In many ways this step is the simplest since all the difficult data wrangling has already been done.
To start open a new blank project in QGIS (or ArcMap, the process is very similar) and click on the ‘Add Delimited Text Layer’ button which is usually found on the left hand side of the screen. This will trigger ‘Create a Layer from a Delimited Text File’ form. Click on the ‘Browse’ button and navigate to, and select, the CSV file that contains the processed data. Check everything on the ‘Create a Layer…’ form making sure that the CSV option is selected and that the x field is longitude and the y field is latitude. Click on the OK button select ‘WGS 84’ from the coordinate reference systems of the world.
Displaying the Data
You should now see the data displayed as points on the GIS layer. On the layers panel right click on the layer and select the properties option. Do the following steps:
- Change the ‘Single Symbol’ option to ‘Graduated’
- In the Column option choose Wind_Speed (or whatever you named you wind speed column)
- Click on the Symbol and choose the simple marker of and arrow pointing up. Now change the symbol layer type to ‘Font marker’ scroll down to the bottom of the font grid and select the arrow pointing upwards.
- Now select a field for the rotation of the arrow, this is to the right hand side of the rotation text box, the field you need to activate should be Wind_Dir, unless you have used a different name.
- Click ‘OK’
- Back at the properties section choose:
- A colour ramp
- Equal Interval as the mode in the Classes section
- Click on the ‘Classify’ button
- Click on the ‘OK’ button
When you are returned to the GIS the layer should be populated by arrows which are pointing in the direction of wind flow, the arrows will also be coloured based on wind speed.
Adding Some Geographical Context
To put the wind data into geographical context we can now add a layer of global coastline data. This is contained in the Natural Earth data folder that you download earlier. Click on the ‘Add Vector Layer’ button in the left hand side of QGIS and navigate to the Natural Earth Data folder and select the file ‘ne_10m_coastline.shp’.
Conclusion and Next Steps
A small bit of effort will greatly improve the presentation of the wind data. In the figure below I added some data layers from the Natural Earth data resource and tweaked the colours for wind speed.
The work flow involved in adding other GFS parameters is very similar to what I have shown here, so why not try adding more forecast parameters? The available parameters are here:
Another option is to add data from other models, once again so long as you are using ERRDAP the work flow is very similar to what I have explained here. As a suggestion you can try the following:
- Wave forecast from the WaveWatchIII model
- Sea Surface Temperature from the Real Time Ocean Forecast System
- Chlorophyll from the MODIS satellite
So long as you pay attention to dates and projections you too can be using GIS to make ‘mashups with heterogeneous data’! Enjoy!
Thanks. Why not consolidating all this into a QGIS plugin?
That is a great idea, I will see what I can do, but don’t hold your breath I am pretty busy 🙂
That is a great idea, I will see what I can do.
Glad you like it. If you need help, just let me know.
In the section for downloading it as CSV, you can choose NetCDF format. Crayfish plugin for QGIS will allow you to display and animate the format without any conversion:
Very nice. I thought there must be a suitable plugin somewhere. I hope that you are finding the ERDDAP technology useful?
We haven’t used it yet but good to know the service.
how can we relate different parameters(wind
pressure,humidity,presipitation,rain falling) with temperature in GIS
(Arcmap). I mean i want to find the trend of change in temperature of a region
w.r.t. above given parameters.
Hi, if you follow link below you find the reanalysis data. This is project where the current weather models are run using historical data, in effect running the model backwards. It is done to fill in the geographical gaps. This data set goes back to 1948, since it available via ERDDAP the techniques I have explained in both my articles here will be enough to get started with finding the data you want, remember the metadata will give you a description of the parameters. http://coastwatch.pfeg.noaa.gov/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=reanalysis The ECMWF also have a reanalysis project: http://www.ecmwf.int/en/research/climate-reanalysis However the data is not on ERDDAP… Read more »
In the next International QGIS Conference, that will take place in Girona (Spain), my collegue Mauricia will explain a new plug in that will be realesed soon, the title of the talk is “THREDDS Explorer plug-in, bringing meteoceanographic operational data into QGIS” you can hace a look to the abstract here.. http://www.sigte.udg.edu/jornadassiglibre/international-qgis-user-and-developer-conference/conferencia-qgis/
Great stuff! I look forward to seeing that.
Here it is: video.. https://vimeo.com/ihcantabria/threddsexplorer ) and access to the code…. https://github.com/IHCantabria/THREDDSExplorer
Great stuff I look forward to experimenting with this.
That’s a pretty detailed and low-level tutorial! Love it! Excuse me for bringing my own stuff inside this topic 🙂 I’m eager to learn about struggles that data engineers and scientists are having with data processing and generally data access. I’m working at PlanetOS, where we created a simple point based (probe) API to access similar climate/weather datasets (we have GFS, NWW3, HYCOM and bunch of others). If anybody is up to share some feedback, please, let me know. You can learn more about available datasets at http://data.planetos.com Links to API documentation, examples (Python, JS) are shared in this blog… Read more »
Hi Ilya, thanks for youy comment. I have only been able to have brief look at the PlanetOS api. I was looking at GFS (NCEP) and the first thing that struck me was how do I get two parameters at the same time? I was trying to recreate the workflow I described above but using PlanetOS instead of ERDDAP. I am sure it can be done, I just haven’t had enough free time to look at it.
Dear Conor Delaney,
I would like to congratulate you for the explanation. I’m finding it difficult to generate the wind field direction in Qgis, I can not identify whether the error is in the formula. I am a doctoral student at the Federal University of Pernambuco in Brazil and my research is to identify natural seeps of oil in the Plateau of Pernambuco.
I thank you.
Hi Hewerton, I only just saw your question. Are are you still having the problem?
Hi Conor Delaney,
I’m not able to rotate the vectors to give the wind direction, as tenten several times and it still fails. I’ve redid the calculation several times and the results are the same and also I do not know if I’m erring on Qgis, no longer know what to do. I send you herewith a part of my spreadsheet.
I plan to use this data for my doctoral research.
Thank you for helping me.
I haven’t have a chance to look at this. Can you send me the are of interest you are looking at so that I can see if I can reproduce the problem.
Hi Hewerton, sorry for the delay. I think I know what the problem is. First of all I haven’t been able to work with GFS data as the servers in Hawaii seem to be slow today. So I worked with RTOFS data instead (it is hosted on a different server) and I noticed that QGIS will not allow you to set a wind direction field if #Value! appears anywhere in the column, I did a CTRL-H on the spreadsheet and replaced them all with 0. Once I did this I could set the wind_dir (and wind_speed) fields in the layer.… Read more »
Congrats for this wonderful post! I’m a PhD student from Madrid, Spain, and I’m interested in fungal spores dispersion through wind, animals, etc. I was using the information of your post some weeks ago to perform an R script to automatize all the tasks that you describe here, but today, suddenly, the gfs wind database has disappeared! It is only the sea wave data… do you have some information? Thanks in advance anyway! And congrats again or this great job!
The server is up again, thanks! 🙂
Sorry in the delay in getting back to you. With ERDDAP when a remote data server goes down it is automatically removed from the catalog. ERDDAP keeps contacting the remote server until it is ready again and then adds the data back into the catalog.
You might find this data useful as well : http://coastwatch.pfeg.noaa.gov/erddap/griddap/esrlNcepRe.graph
It is the global reanalysis weather forecast data back to 1948.
Dear Community! I was following this great post and I was developing several functions in R language to automatize this tasks. I put all of them in an R package called rWind, hosted in CRAN repository. Here you have a brief tutorial to use it!
Fantastic work Javi, I just tweeted the link
Thanks for the feedback Conor! And thanks for the global reanalysis weather forecast data, I’m thinking to change my R functions to obtain the data directly from this database… are there many differences between them, apart dates?
I’m interested also in global sea currents… do you know any general database from ERDDAP to get this data? I’m working to implement also some functions to compute least cost paths having into account this kind of data to perform anisotropic analysis, I’ll add to my package as soon as possible… I’ll inform you!
Hi, the GFS data in ERDDAP is only updated daily, so if you go straight to the source (http://nomads.ncep.noaa.gov) You will get data at higher resolutions. Any some stage this data will be on ERDDAP. As to sea currents, the two modelling sources that are available to you are the Real Time Ocean Forecast System (RTOFS) from NOAA, which is on ERDDAP or the EU equivalent which can be found here http://marine.copernicus.eu/services-portfolio/access-to-products/. Looking forward to seeing your new functions. By the way if you start using RTOFS please be aware that the the geographic grid in the data is different… Read more »
Hi again Conor! I’m back! Finally I put the new function in the rWind package to find the shortest path having into account wind speed and direction! Here is the github repository, the function is “flow.dispersion”. I have not posted anything about it yet, but I will in the next months! In the other hand, I didn’t find the Real Time Ocean Forecast System (RTOFS) data base that you told me… I was making some work with this one: “Currents, Geostrophic, Aviso, 0.25 degrees, Global, 1992-2012 (1 Day Composite” but it goes only until 2012… do you know what is… Read more »
Great Stuff! The RTOFS data is on this ERDDAP server http://coastwatch.pfeg.noaa.gov/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=rtofs There are forecast, nowcast, prognostic and diagnostic flavours (I can’t remember what they all mean, sorry). Not all the models have the same parameters. Also data near the poles use a different grid. All the information that you need is in the meta data. There are other models available, just enter the parameter that you are interested in into the search box on the ERDDAP landing page. http://coastwatch.pfeg.noaa.gov/erddap/index.html ERDDAP is getting used by a lot of National Ocean Data Centres, if you use Google to search for ERDDAP and… Read more »
Perfect contribution. It was very useful for me. But I need your help. I do not know how to solve the following problem. I want to create a wind map and I have these data (attachment: )
The data is from one meteorological station and therefore I have only one longitude and latitude value (48.59278 °, 20.09361 °). Is it possible to create a wind map from these data and display it in QGIS, similarly like in your post?
Thank you for answer.
Thank you so much for sharing this info. It’s really helpful. I can plot the wind direction successfully using your guidance. I did it in ArcMap.
I wanted to review your steps of selecting an up arrow and formatting, so many options are not sure what the instructions should be. I start with selecting a symbol, then selecting Simple Marker (note tried to attach image but failed). Then the instructions say “Now change the symbol layer type to ‘Font marker’ scroll down to the bottom of the font grid and select the arrow pointing upwards” But don’t we need to know what font to use, I tried wingding3? What font are we selecting an up arrow from? Thanks P.S., I am using data from https://nomads.ncep.noaa.gov/ and… Read more »
Note, I have been trying to edit my own question. I did get quivers or wind vectors to work.
I did select simple marker -> Font Marker -> select Font Type “wingding3” and select wind_dir as my rotation variable and wind_speed in my classes to do color-ramp.
note only difference was getting grib2 data from NOAA’s GFS dataset, using 700mb and using pygrib to extract U,V components which then transformed to wind_speed, wind_dir.
thanks for getting me going. all