HPWREN is Making a Large Data Set of its Met Data Available

22 July 2022

A request from an outside researcher looking for access to 1 Hz wind data from HPWREN meteorological sensors led to the creation of a large downloadable data set across many years. For about two decades HPWREN had already made its sensor data from cameras and met stations publicly available, but the met data was not easily accessible in bulk on the web side. That data, as collected from HPWREN's Vaisala WXT5xx weather transmitters, includes second-by-second wind speed and direction, 10 second temperature, relative humidity and atmospheric pressure, as well as up to 10 second rain and hail information, and in some cases extends back to 2008. The sensor manual, which is needed to understand and interpret the data, is currently available at

https://www.vaisala.com/sites/default/files/documents/M210906EN-C.pdf

A Vaisala WXT520 met sensor, displayed on the right, is often deployed on mountain tops collocated with HPWREN nodes, shown here in an ad-hoc setting which includes a camera alongside wireless communication. A setup like this could be deployed quickly to provide real-time information, e.g., during a wildland fire.

Vaisala WXT520 met sensor

How to get the data

The uncompressed more than 100 gigabyte data set extends from 2008 to
5 July 2022 and is structured into annual files in https://localhost/hpwren-site/WXTdata as compressed tar archives. See table on the right.

Extracting files from a tar archive (e.g.: "tar xf 2008.tgz") creates a top level directory for the year (e.g., "2010"), followed by directories named as the day of the year as YYYYMMDD (e.g., "20101002"). The raw data files are then in those daily directories, such as:

ls 2010/20101002
....
198.202.124.3-HPWREN:LP-WXT520:0R0:1:0
198.202.124.3-HPWREN:LP-WXT520:0R1:1:0
198.202.124.3-HPWREN:LP-WXT520:0R2:1:0
198.202.124.3-HPWREN:LP-WXT520:0R5:1:0
....
198.202.124.3-HPWREN:TP-WXT520:0R0:1:0
198.202.124.3-HPWREN:TP-WXT520:0R1:1:0
198.202.124.3-HPWREN:TP-WXT520:0R2:1:0
198.202.124.3-HPWREN:TP-WXT520:0R3:1:0
198.202.124.3-HPWREN:TP-WXT520:0R5:1:0
....

Those file names are HPWREN-internal sensor identifiers. The data in the file is obtained from typically mountain-top sensors via a serial line connected to a serial/Ethernet converter which receives the continuous stream. A centralized computer at UCSD then connects to the converter, adds an origin IP, sensor ID and a timestamp followed by the raw data, and writes it to the network via multicast.

198.202.124.3   HPWREN:TP-WXT520:0R1:1:0        1286002800      0R1,Dn=205D,Dm=214D,Dx=219D,Sn=5.3M,Sm=6.4M,Sx=7.2M
198.202.124.3   HPWREN:TP-WXT520:0R1:1:0        1286002801      0R1,Dn=205D,Dm=213D,Dx=219D,Sn=4.6M,Sm=6.2M,Sx=7.2M
198.202.124.3   HPWREN:TP-WXT520:0R1:1:0        1286002802      0R1,Dn=205D,Dm=212D,Dx=219D,Sn=4.3M,Sm=6.0M,Sx=7.2M

The data itself is detailed in the manual. The sensor name should be consistent with https://localhost/hpwren-site/Sensors/.

Multiple computers at UCSD listen to the multicast streams, among them one who just writes the data from those streams to disk for safe keeping, which is where the met data described in this article is from. Other machines use the multicast data for the real-time HPWREN web interface at https://localhost/hpwren-site/Sensors/, make data available in different ways to other parties such as NOAA/NWS and MesoWest, or to issue real-time Santa Ana alerts, including to firefighters (see https://localhost/hpwren-site/news/041106.html and https://localhost/hpwren-site/news/20180415/).

We hope that these data sets will be useful to more researchers and students, and we encourage you to let us know if you use it and what you use it for.