metaf2xml: parse and decode METAR, TAF, SYNOP and BUOY and write data as XML

(deutsche Version, versión en Español)

metaf2xml parses and decodes aviation routine weather reports (METAR, SPECI), aerodrome forecasts (TAF), synoptic observations (SYNOP) and observations from buoys (BUOY). The extracted data can be written as XML or passed to a user-defined function (all done in Perl).
It also provides XSLT style sheets to convert the XML to plain language (text, HTML), or XML with different schemas. A web-based (CGI) user interface can be used to fetch and display current weather data from NOAA, Ogimet, and other sources.

Content Latest version Background Why another METAR/TAF parser? Features of metaf2xml Demo version on SourceForge Useful links Examples for messages in plain language

TAF SBGL (HTML, español)

METAR KJFK (HTML, english)

METAR UUEE (HTML, Русский)

Latest version

The latest version (2012-01-04) is 1.47 (TAR, RPM, ZIP), and includes also README, change log, ToDo, distinfo, and DTD.

Background

METAR and SPECI (weather observation) and TAF (weather forecast) information is essential for flight planning and in-flight decisions (real flight and simulation). It is regularily compiled by meteorologists at about 3500 airports worldwide and distributed via reliable, non-public networks. Information for SYNOP is collected by more than 11 000 manned and unmanned meteorological stations around the world and is used for weather forecasting and climatic statistics. The format of the original messages is abbreviated, some items are coded. Here are some METAR, TAF, and SYNOP messages from Rio (SBGL, 83746), New York (KJFK, 74486), and Tokyo (RJTT, 47662) and BUOY messages from a buoy nearby (31550, 41997, 53596):

SBGL 040600Z 30003KT 9999 FEW030 22/19 Q1011
KJFK 040551Z 32019G25KT 10SM CLR M09/M20 A3015
KJFK 040837Z 0409/0512 31012KT P6SM SKC FM041500 28011KT P6SM SCT150 BKN250 FM041800
 25011KT P6SM SCT070 BKN150 FM042200 24014KT P6SM BKN070 FM050600 25009KT P6SM FEW050 SCT070 BKN150
RJTT 040630Z 34016KT 9999 FEW030 SCT080 09/M04 Q1001 TEMPO 34018G28KT RMK 1CU030
 3AC080 A2958 VIRGA ABT 20KM NE AND SE-SW MOV SE
RJTT 040846Z 0409/0512 33016KT 9999 FEW030 TEMPO 0409/0412 34020G30KT TEMPO 0506/0512 34020G30KT
AAXX 04064 74486 NIL
AAXX 04064 47662 12/80 63409 10085 21068 39982 40026 55000 60001 82202
ZZYY 41595 04012 0540/ 731764 040662 6111/ 11119 30293 40293 52247 22219 00205 444
 20110 04012 0522/ 50101 80120 80013 9/015
ZZYY 53524 04012 0640/ 130012 144649 6111/ 22219 00207 444 20110 04012 0628/ 50101 80072 80027 9/015

The format of the messages (and a lot more about observing and reporting weather) is specified in WMO Manual No. 306 or in Annex 3 to the Convention on International Civil Aviation; some countries deviate from that standard or extend it, some have published documentation (e.g. Federal Meteorological Handbook No. 1 and No. 2 for the USA). Especially the format of the remarks for METAR and TAF messages, or the coding of sections 3 and 5 for SYNOP messages depend on the country or even the observing station.

metaf2xml can be used for flight simulations or by private pilot (students) who want to extend their knowledge about the message formats, and everyone who just wants to know the current or forecast weather at some station in plain language (see examples below).

Why another METAR/TAF parser?

There are other projects that deal with METAR and TAF messages, e.g.:

• the program "metar" of the FlightGear project,
• the Perl module Geo::METAR,
• the Java library jweather,
• the C library MDSP from the NOAA/NWS (link to download seems to be broken, on SourceForge: mdsplib),
• an MS Windows binary (runs with Wine) in a ZIP archive from MetarWeather,
• the ADDS department of the NOAA translates METAR and TAF messages (most elements) from their database,
• the web site Ogimet provides graphical and text-based interfaces to up-to-date and older METAR, TAF, SYNOP and BUOY data (collected from NOAA),
• the web site mundomanz provides graphical and text-based interfaces to up-to-date and older SYNOP data; most parts of the messages are also translated to plain language (English, Spanish),
• the web site Landings: Aviation's Databases fetches current weather messages (source: NOAA/IWS) and can display them in plain language (enter the airport code or name and follow the link for the decoded METAR or TAF),
• Allmetsat provides a graphical interface to METAR and TAF data (source: NOAA/IWS); the most relevant parts of the messages are also translated to plain language.

But somehow I always missed a feature I liked to have.

Features of metaf2xml

processing of messages

The parser module (man page) accepts (= processes) or tolerates (= marks as error) many extensions to the standard; they are marked with "EXTENSION" in the source code. Many common minor errors in the format are corrected before parsing, more combinations of weather phenomenons are allowed, missing information is recognised in many cases, too.
All remarks are processed, some of them are interpreted depending on the issuing country or station, e.g. 'MON INVIS VAL FOSCHIA' and QUL/QUK (Italy), 'JP' (some carribean countries), 'YLO1' (military), everything from the Federal Meteorological Handbook No. 1 and the Air Force Manual 15-111 (19.12.2003), 'RSNK' and 'LAG PK' (data from other observing stations), and many more. Currently, more than 80 types of METAR remarks are recognised, excluding trivial keywords (see also the DTD).
The intention is to gain as much information as possible from a message without interpreting parts of it wrongly.

output of message data

The module XML.pm (man page) can write the data as XML. Alternatively, this module can be used to register a callback function which is passed step by step the same data in the same order as for the XML output.
The data contains solely information from the original message, with 2 exceptions: the relative humidity (actually 4 different values from 4 different formulas) and the ceiling (if there is one); these 2 items are cumbersome to compute with XSLT. For each data element, the corresponding text from the message is provided.
The intention is that no parsing of the output data should be necessary.

source of messages

Messages can be specified simply as text. Additionally, the script metaf.pl (man page) can be used from the command line or as CGI script to fetch up-to-date messages for airports (METAR/TAF) and stations (SYNOP, BUOY) from a public server or read messages from a local file.
metaf.pl uses the module src2raw.pm (man page) to extract correctly formatted messages from the files provided by public servers.

display of XML file

The project also provides XSLT style sheets to convert the XML to plain language as text or HTML in English, German, Spanish, and Russian (corrections and additions are welcome). The output in the text format is similar to the one of the program "metar" from the FlightGear project. The HTML output displays each message in a separate table. The leftmost column contains the original text for each element of the message, the top row contains the original message (see also project screenshots or examples below). To convert the content to text or HTML, the program "xsltproc" is used, but others that work similar shouldn't be a problem.
There are also XSLT style sheets to create a summary or overview: only some items of each message (currently no TAF) are shown in an abbreviated format, one line per message.

command line interface

The script metaf2xml.pl (man page) reads messages, processes them, and writes the XML data to standard output. It can serve as input for "xsltproc" or similar programs to get HTML or text.
The script metaf.pl (man page) can be used to fetch up-to-date messages from other sources.

web interface

Ideally, any XML capable browser should be able to process the XML with the XSLT style sheet specified in the file, and display the formatted result, but not all browsers are XML capable. The script metaf.pl (man page) provides an interface using either XML/XSLT or pure HTML. The XML data can be seen by viewing the source of the web page if "XML" was selected as output format.

source code is available and GPL

Everybody who wants to contribute is invited to do so. If you want to help you might pick a task from the ToDo.

Demo version on SourceForge

With the demo version with graphics (uses Ajax), the stations are marked with a dot on a map. The map can be viewed in different enlargements and can be moved by click and drag. The processed messages are displayed beneath the map, with the latest request on top. The list of stations is fixed but it also could be created dynamically, e.g. from a database.

The messages for the stations are fetched from files on the SourceForge server. These are messages from the past, the files are updated manually from time to time, only. SourceForge does not allow outgoing connections, so messages cannot be fetched directly from the internet.

It is also possible to enter weather messages as text. This requires the selection of the message type (METAR, TAF, SYNOP, BUOY).

With the text-based demo version, weather messages can be entered. If the option "ICAO id(s)" or "WMO id(s)" is selected, station id's can be entered, too. The messages for the stations will be fetched from files on the SourceForge server, as for the graphical demo version.

Several organisations and providers offer public access to up-to-date and older weather messages, e.g.:

• the IWS department of the NOAA (24 hours of METAR and TAF),
• the ADDS department of the NOAA (36 hours of METAR, latest TAF),
• their experimental dataserver (72 hours of METAR and TAF, as XML or CSV),
• Nav Canada (Canada, only),
• Ogimet (METAR since 2005, TAF since 2006, SYNOP and BUOY since 1999),
• the NWS department of the NOAA (24 hours of METAR, TAF, SYNOP and BUOY),
• the College of DuPage (5 days of SYNOP but for fewer stations).

Useful links

Demo version

(see also Demo version on SourceForge)

• demo version with graphics
• text-based demo version

Files from the latest version

• the README
• the ChangeLog
• the document type definition (DTD) for the XML data

Man pages (generated from the perl sources)

• for the parser module parser.pm
• for the module XML.pm to write the XML
• for the module src2raw.pm to extract messages from the files provided by public servers
• for the script metaf2xml.pl
• for the script metafsrc2raw.pl
• for the (CGI) script metaf.pl

Project links

• the project at SourceForge: summary, packages for releases, news, web access to the Git repository
• the project at freshmeat

Examples for messages in plain language

The result of the parsing of the above example messages can be found in example.xml (XML without XSLT style sheet reference). Using the XSLT style sheets from the project, the output in plain language (here: as HTML, summary first, then detailed) looks as follows:

Warning: The information below may be out-dated, inaccurate, or both. It is not suited for use in aviation.

Warning: The information below may be out-dated, inaccurate, or both. It is not suited for use in aviation.

Values of entries marked with * do not originate directly from the message, but are derived from its data.

warning: not processed: 20110 04012 0522/ 50101 80120 80013 9/015

warning: not processed: 20110 04012 0628/ 50101 80072 80027 9/015

copyright (c) 2006-2011 metaf2xml @