metaf2xml: parse and decode METAR, TAF, and SYNOP and store as XML

(deutsche Version, versión en Español)

metaf2xml parses and decodes aviation routine weather reports (METAR, SPECI), aerodrome forecasts (TAF), and synoptic observations (SYNOP) and stores the components as XML (all done in Perl). There are also XSLT style sheets to convert the XML to plain language (text, HTML), or XML with different schemas, and a command line and a web-based (CGI) user interface.

Content Latest version Background Why another METAR/TAF parser? Features of metaf2xml Resources, demo version Examples for messages in plain language

TAF SBGL (HTML, español)

METAR KJFK (HTML, english)

METAR UUEE (HTML, Русский)

Latest version

The latest version (2009-08-09) is 1.37 (TAR, ZIP), and includes also README, change log, ToDo, distinfo, and DTD.

Background

METAR and SPECI (weather observation) and TAF (weather forecast) information is essential for flight planning and in-flight decisions (real flight and simulation). It is regularily compiled by meteorologists at about 3,500 airports worldwide and distributed via reliable, non-public networks. Information for SYNOP is collected by more than 11,000 manned and unmanned meteorological stations around the world and is used for weather forecasting and climatic statistics. The format of the original messages is abbreviated, some items are coded. Here are some METAR, TAF, and SYNOP messages from Rio (SBGL, 83746), New York (KJFK, 74486), and Tokyo (RJTT, 47662):

SBGL 090700Z 00000KT CAVOK 18/15 Q1016
SBGL 090330Z 0906/1006 07005KT 8000 NSC TN20/0908Z TX31/0917Z PROB30 0908/0912 5000 BR BECMG 0911/0913
 02005KT BECMG 0915/0917 17010KT CAVOK BECMG 0921/0923 07005KT BECMG 1001/1003 02005KT 8000 NSC
KJFK 090751Z 17006KT 10SM SCT028 BKN080 OVC120 22/16 A3016 RMK AO2 SLP213 T02220161
KJFK 302055Z 3021/3124 21012KT P6SM FEW050 SCT250 FM310200 21007KT P6SM SCT070 BKN250 FM310900 19005KT
 P6SM BKN070 OVC150 FM311200 18010KT 5SM -SHRA BR OVC040 FM311500 19012KT 3SM RA BR OVC015CB FM311800 19015KT 5SM -SHRA BR SCT015CB OVC025
SPECI RJTT 090730Z 12008KT 080V160 9999 VCSH FEW025 FEW030CB BKN035 BKN/// 30/25 Q1010
RJTT 090700Z 15009KT 9999 VCSH FEW025 FEW030CB SCT050 BKN/// 31/24 Q1010 RMK 1CU025 1CB030 4SC050 A2983
 CB 30KM E AND 15KM NW MOV NE
RJTT 310026Z 3100/3124 03010KT 9000 FEW010 SCT020 BKN030 TEMPO 3100/3103 4000 -RA BR FEW004 BKN007 BECMG
 3104/3106 07010KT BECMG 3109/3112 04012KT TEMPO 3118/3121 FEW005 BKN009
AAXX 09064 83746 31962 01201 10186 20159 40169 57012 71000
AAXX 09064 74486 12766 81706 10222 20156 30212 40222 58002 69901 90551 333 10222 20172 555 90906
AAXX 09064 47662 12/65 81603 10308 20242 30059 40100 58005 60001 8327/

The format of the messages (and a lot more about observing and reporting weather) is specified in WMO Manual No. 306 or in Annex 3 to the Convention on International Civil Aviation; some countries deviate from that standard or extend it, some have published documentation (e.g. Federal Meteorological Handbook No. 1 and No. 2 for the USA). Especially the format of the remarks for METAR and TAF messages, or the coding of sections 3 and 5 for SYNOP messages is highly dependent on the country or even the observing station.

metaf2xml can be used for flight simulations or by private pilot (students) who want to extend their knowledge about the message formats, and everyone who just wants to know the current or forecast weather at some station in plain language (see examples below).

Why another METAR/TAF parser?

There are other projects that deal with METAR and TAF messages, e.g.:

• the program "metar" of the FlightGear project,
• the Perl module Geo::METAR,
• the Java library jweather,
• the C library MDSP from the NOAA/NWS (link to download seems to be broken),
• an MS Windows binary (runs with Wine) in a ZIP archive from MetarWeather,
• the ADDS department of the NOAA translates METAR and TAF messages (most elements) from their database,
• the web site OGIMET provides graphical and text-based interfaces to up-to-date and older METAR, TAF, and SYNOP data (collected from NOAA),
• the web site mundomanz provides graphical and text-based interfaces to up-to-date and older SYNOP data; most parts of the messages are also translated to plain language (English, Spanish),
• the web site Landings: Aviation's Databases fetches current weather messages (source: NOAA/IWS) and can display them in plain language (enter the airport code or name and follow the link for the decoded METAR or TAF),
• Allmetsat provides a graphical interface to METAR and TAF data (source: NOAA/IWS); the most relevant parts of the messages are also translated to plain language.

But somehow I always missed a feature I liked to have.

Features of metaf2xml

processing of messages

The parser module (man page) accepts (= processes) or tolerates (= marks as error) many extensions to the standard; they are marked with "EXTENSION" in the source code. Many common minor errors in the format are corrected before parsing, more combinations of weather phenomenons are allowed, missing information is recognised in many cases, too.
All remarks are processed, some of them are interpreted depending on the issuing country or station, e.g. 'MON INVIS VAL FOSCHIA' and QUL/QUK (Italy), 'JP' (some carribean countries), 'YLO1' (military), everything from the Federal Meteorological Handbook No. 1 and the Air Force Manual 15-111 (19.12.2003), 'RSNK' and 'LAG PK' (data from other observing stations), and many more. Currently, more than 80 types of METAR remarks are recognised, excluding trivial keywords (see also DTD).
The intention is to gain as much information as possible from a message without interpreting parts of it wrongly.

storing message contents as XML

The module XML.pm (man page) generates the XML data. This data only contains information from the original message with 2 exceptions: the relative humidity (actually 4 different values from 4 different formulas) and the ceiling (if there is one); these 2 items are cumbersome to compute with XSLT. Each XML top node for a message component has the attribute "s" containing the corresponding part from the original message.
The intention is that no parsing of items in the XML file should be necessary, so every information item gets a separate XML node.

source of messages

Messages can be specified simply as text. The CGI script "metaf.pl" can also fetch them from a public server (currently nearly all NOAA) or read from a local file. For that, the parameters "type_metaf" and "type_synop" must be "ICAO" and "WMO", respectively, and the entry fields need to contain the airport or station code(s).

display of XML file

The project includes XSLT style sheets to convert the XML to plain language as text or HTML in English, German, Spanish, and Russian (corrections and additions are welcome). The output in the text format is similar to the one of the program "metar" from the FlightGear project. The HTML output displays each message in a separate table. The leftmost column contains the original text for each element of the message, the top row contains the original message (see also project screenshots or examples below). To convert the content to text or HTML, the program "xsltproc" is used, but others that work similar shouldn't be a problem.
There are also XSLT style sheets to create a summary or overview: only some items of each message (currently METAR and SYNOP, only) are shown in an abbreviated format, one line per message.

command line interface

The script metaf2xml.pl (man page) reads messages from standard input, processes them, and writes the XML data to standard output. This can then be fed to "xsltproc" or similar programs to get HTML or text. The script metaf.pl (man page), though intended as a CGI script, can be used to fetch messages from other sources.

web interface

Ideally, any XML capable browser should be able to process the XML with the XSLT style sheet specified in the file, and display the formatted result, but there are some that could not process some of the XSL directives in them. Also, not all browsers are XML capable. The CGI script metaf.pl (man page) provides an interface in pure HTML. The XML data can be seen by viewing the source of the web page if "XML" was selected as output format.

sources are available and GPL

Everybody who wants to contribute is invited to do so. If you want to help you might pick a task from the ToDo.

Resources, demo version

The project's home page is at SourceForge.net, releases are also announced at freshmeat.net. The latest version includes the parser, the DTD schema, XSLT style sheets to convert the XML to plain language, the Perl scripts for the user interfaces, and everything to install all of them.

With the demo version with graphics (uses Ajax), the stations are marked with a dot on a map. The map can be viewed in different enlargements and can be moved by click and drag. The processed messages are displayed beneath the map, with the latest request on top. The list of stations is fixed but it also could be created dynamically, e.g. from a database.

The messages for the stations are fetched from files on the sourceforge-server. These are messages from the past and are updated only from time to time. sourceforge.net does not allow outgoing connections, so they cannot be fetched directly from the internet.

It is also possible to enter weather messages, which requires the selection of the message type (METAR, TAF, SYNOP).

With the text-based demo version, weather messages can be entered. If the option "ICAO id(s)" or "WMO id(s)" is selected, station id's can be entered, too; the messages for the stations will be fetched from files on the server, as for the graphical demo version.

Several organisations and providers offer public access to up-to-date and older weather messages, e.g.:

• the IWS department of the NOAA (24 hours of METAR and TAF),
• the ADDS department of the NOAA (36 hours of METAR, latest TAF),
• their experimental dataserver (72 hours of METAR and TAF, as XML or CSV),
• the Swedish Civil Aviation Administration (Europe, only),
• Nav Canada (Canada, only),
• OGIMET (more than 180 days of METAR, TAF, and SYNOP), or
• the NWS department of the NOAA (24 hours of SYNOP), or
• the College of DuPage (5 days of SYNOP but for fewer stations).

Examples for messages in plain language

The result of the parsing of the above example messages can be found in metar-taf-synop.xml (XML without XSLT style sheet reference). Using the XSLT style sheets from the project, the output in plain language (here: english, HTML) looks as follows:

warning: station pressure missing

copyright (c) 2006-2009 metaf2xml @