The History of HPWREN: A Retrospective from 2023

31 December 2023


Table of contents

  1. Introduction
  2. Project Foundation and Support
  3. Evolution and Milestones
    1. In the Beginning ...
    2. Stakeholders and Support
    3. Network Infrastructure and Growth
    4. Sensor Deployments
    5. Data Access and Management Infrastructure
    6. Evolving Information Technology Infrastructure over Time
    7. People and Teamwork
    8. The Impact of HPWREN on the Demand for HPWREN
  4. Enabling R&D and Technology Demonstrations
  5. Outreach
    1. Native American Connections and Outreach
    2. Remote education
    3. User Workshops
    4. Social Media
  6. HPWREN in 2023
  7. Future Directions
  8. Options to Replicate
  9. Additional Photos
  10. Acknowledgments
  11. References

1. Introduction

This overview describes the evolution of the University of California San Diego (UCSD) High Performance Wireless Research and Education Network (HPWREN) Project. HPWREN originated as a wide-area wireless communications research project at the dawn of this millennium. Initially limited to San Diego County in support of Internet data applications in the research and education domains, it evolved over time. A substantial collaboration with first responder agencies emerged from at first very informal discussions, when those agencies assisted with HPWREN's need to deploy routers and wireless links on mountain tops.

Examples of early connections include Native American tribes via their education centers, the SDSU Santa Margarita Ecological Reserve as a biological field station, the Mount Laguna and Palomar Observatories for astronomy, the California Wolf Center as another biological observatory, as well as many earthquake and other geophysical sensors. The initial first responder agencies with which HPWREN collaborated were the California Department of Forestry and Fire Protection (CDF, now CAL FIRE) via their Emergency Command Center, as well as the Regional Communications System staff at the San Diego Sheriff's Department. In retrospect, without this assistance at their mountaintop locations, it would have been unlikely that HPWREN could have expanded to the scope it is today, almost a quarter century later. Over time, collaborations also expanded to include San Diego Gas and Electric, as well as various research groups at multiple academic institutions.

Currently, in addition to the communications infrastructure system for research and education, HPWREN also provides wireless communications support for cameras and sensors, often in remote areas, through a radio based wireless networking system with components that are typically off-the-shelf equipment. This wide-area network now spans San Diego County and has been extended into multiple other Southern California counties. With its radios often located at high elevations, many of the sites also support collocated cameras, weather stations with a high temporal resolution, and other sensors. All of the regional ANZA seismic sensors are also supported. HPWREN camera and other sensor data is openly published in near-real time at https://www.hpwren.ucsd.edu. Seismic data is available at https://doi.org/10.7914/SN/AZ. For more than a decade HPWREN has also provided Internet access to first responders in support of 60+ backcountry federal (US Forest Service), State (CAL FIRE) and San Diego County fire stations. The fire station network access and near-real time camera image availability across San Diego County provide a significant public safety component to Emergency Responders as well as the larger community.


a
b
c
1
2
3

1a: Bud Hale transporting an 8 foot antenna. 1b: Glen Offield planning his next steps. 1c: Todd Hansen experimenting with a passive relay. 2a: Todd Hansen watches Kimberly Mann Bruch hammer in a grounding rod for lightning protection at one of the rural sites that received connectivity via HPWREN. Both were HPWREN team members for many years. 2b: James Hale and Pablo Bryant attending to the arrival of an 8 foot antenna on Monument Peak, next to a substantial dropoff towards the desert thousands of feet lower. 2c: Hans-Werner Braun, Michael Peralta (TDVNet) and Pablo Bryant at the end of a day of work at a TDVNet tower site. 3a: Pablo Bryant checking the electronics package to be installed on an SDSU Sky Arrow high-wing pusher light aircraft for ground-air-ground communications tests. 3b: James Hale installing equipment on North Peak as part of the preparation for a tracking antenna to undertake communications experiments with the AA330, in collaboration with MIT Lincoln Labs and CAL FIRE. 3c: James Hale is participating in the planning and preparations for the HPWREN exhibit within the National Science Foundation's booth at the 2007 American Association for the Advancement of Science (AAAS) annual meeting in San Francisco.


2. Project Foundation and Support

Headquartered at the University of California, HPWREN currently provides high-speed, low-latency networking support to research, education, and public safety users across various Southern California counties.

HPWREN was initially funded by the National Science Foundation (in 2000: ANI-0087344 "An Interdisciplinary Collaboration on Performance Aspects of a High Performance Wireless Research and Education Network" and in 2004: SCI-0426879 "Integration and Analysis of Reliable Networking for Remote Science, Education, and First Responders" with an Accomplishments Based Renewal for another two years in 2009). With NSF funding ending around 2011, HPWREN became a subscriber service, and as a result self-sustaining, while operating on a cost recovery basis.

Although the original concept was for the creation of a wireless communications research project, HPWREN early on found itself involved in discussions and activities related to public safety. In its first years, in 2003 HPWREN assisted firefighting operations by providing its first ad-hoc connection to a wildfire in San Diego County, named Coyote Fire, operations Incident Command Post (ICP). This was followed by connections to fire stations, camps, and air bases, as well as several more connections to ICPs for active fires across several years. In addition, HPWREN increased situational awareness with its real-time environment observing cameras and sensors. Some of the meteorological sensor data started being used to generate automated alerts of critical conditions, such as Santa Ana winds reaching one or more critical thresholds, to various public safety officials.

HPWREN operates as a partnership project, with access predominantly provided in rural and remote environments. In addition, HPWREN staff seek to educate its users about the underlying technologies, thereby creating a collaborative technology transfer platform in which its users can actively participate in the network's evolution to ensure it meets their own programatic requirements.

NSF provided some financial support prior to the August 2000 award for a proof of concept effort. With the award, the prototype was to be extended as cyberinfrastructure to various research and education projects for the purpose of communications and back-hauling data, and accessing remote sensors, which then included earthquake and various other types of sensors. As the project progressed, HPWREN transitioned to receiving support from its major customers and stakeholders. In 2010, the County of San Diego provided funding to augment HPWREN's existing camera and sensor infrastructure. At that time, HPWREN already had a significant number of fire stations connected. This led to San Diego Gas and Electric (SDG&E) supporting a substantial buildout of connectivity, cameras, and weather sensors in the San Diego County area, while also adding connectivity for as many local, state, and federal fire stations as possible. The continuation of this buildout is now directly supported by San Diego County, SDG&E, and various fire agencies.

3. Evolution and Milestones

A good source of interesting imagery from HPWREN activities over time is found at the video Looking back at more than 20 years of the High Performance Wireless Research and Education Network.

3.a In the beginning ...

In 1999, geophysicist Frank Vernon of the UCSD Scripps Institution of Oceanography approached Hans-Werner Braun of the UCSD San Diego Supercomputer Center to request assistance in transitioning his wireless earthquake sensor network to the Internet Protocol suite. The confluence of capability with Frank Vernon's seismologist work on the ANZA seismic network and Hans-Werner Braun's research (e.g., NLANR) and infrastructure (including NSFNET) endeavors on IP networks gave HPWREN a strong starting point for an interesting project.


`

The left photo shows Frank Vernon in 2007, while explaining the benefits of high performance wireless data networks to seismic sensor infrastructures that reach remote areas, ranging from high mountain tops to below sea level desert locations. In the right photo Hans-Werner Braun (seen on the left) prepares for his HPWREN presentation for a congressional hearing before the U.S. House of Representatives Subcommittee on Research in July 2001.


Prior low-power wireless systems were based on proprietary asynchronous serial protocols, and this transition was necessary to ensure that the seismic stations in the ANZA seismic network could utilize the reliable state-of-the-art TCP/IP protocol suite. The other primary driver was the parallel transition of seismic data-loggers from serial data streams to IP based communications. Neither Frank Vernon nor Hans-Werner Braun could have anticipated at the time that their collaboration would give rise to the HPWREN project, which would ultimately evolve into a large scale cyberinfrastructure in Southern California.

In 2000, Hans-Werner Braun and his team began to investigate the feasibility of supporting a regional wireless network. They needed data to support their analysis, and they were particularly interested in assessing the impact of weather conditions on microwave signals, such as cloud cover and other atmospheric phenomena. Meteorological sensors and cameras were initially considered as a means of observing weather patterns and correlating microwave signal viability.

The first HPWREN field sensor was a weather station at Hans-Werner Braun's residence in Ramona (which initially served as an HPWREN Lab), a few months prior to the original NSF award. Cameras were subsequently added, before being moved to HPWREN mountain top locations, to observe cloud activity with the initial goal of better understanding and improving the performance of wide area wireless networks. At the time, there were instabilities, especially with one of the microwave links, and it was initially thought to be weather related. In the end, it turned out that inversion layer conditions were the real threat to these kinds of long haul wireless links while using large directional antennas.

These early efforts included the deployment of multiple weather stations and, subsequently, cameras at HPWREN network backbone locations. Thanks to NSF support, the network expanded, and additional sensors were added over time.

3.b Stakeholders and support

For more than its first decade, HPWREN was funded by the National Science Foundation. Hans-Werner Braun served as Principal Investigator on those grants, with Co-Principal Investigator Frank Vernon. Around 2012 HPWREN became self sustaining, supported by contributions from its various stakeholders, including San Diego County, San Diego Gas and Electric, and various constituents, including major research centers such as the California Institute of Technology's Palomar Observatory and the UC San Diego Scripps Institution of Oceanography. In 2016 Frank Vernon stepped into the role of Project Director (and PI) when Hans-Werner Braun retired from UCSD for medical reasons, while continuing to stay involved part time.

In recent years, HPWREN has significantly deepened its collaboration with the ALERTCalifornia Program, which is a UC San Diego (UCSD) initiative under the leadership of Prof. Neal Driscoll. The origin for this partnership has its roots in the creation of the ALERTWildfire consortium, co-initiated by Prof. Driscoll at UCSD and Prof. Graham Kent at the University of Nevada, Reno. Starting in 2016, ALERTWildfire began the deployment of pan-tilt-zoom (PTZ) cameras across California, Nevada, and Oregon. This period marked a phase of rapid expansion for ALERTWildfire, which now boasts over 1000 PTZ cameras equipped with near-infrared capabilities for enhanced wildfire detection and monitoring. Building on the success of ALERTWildfire, ALERTCalifornia was introduced in 2022, representing a significant evolution of the original program and aiming to further leverage technology in wildfire management and research.

As a state-specific program, ALERTCalifornia is focused on wildfire monitoring and related research efforts in California. The state has more than 163,000 square miles of diverse terrain, which requires adaptive fire management across government agencies and novel research. ALERTCalifornia is positioned to work with government agencies, utilities, and universities to meet state-specific needs and provide the data required to fill knowledge gaps and inform management decisions. HPWREN has two roles in support of ALERTCalifornia. The first role was hosting 95 ALERTCalifornia PTZ cameras, and the second was managing a subset of the ALERTCalifornia networking system. Both roles are expanding as new sites are added to ALERTCalifornia.

Throughout its history, HPWREN has placed a high level of importance on its partnership with the Center for Wireless Communications at UCSD and, subsequently, the California Institute for Telecommunications & Information Technology, also at UCSD, in particular via Ramesh Rao and Larry Smarr. Tom James, then CWC Associate Director, provided expertise earlyon for the initial proof-of-concept links. Following the August 2000 NSF award for HPWREN, the proposal for the establishment of Calit2, was submitted on October 5th, 2000 and described plans to leverage the HPWREN networking infrastructure to address critical applications in Environment and Civil infrastructure monitoring. The relationship between HPWREN and Calit2 evolved over the last two decades enabling new scientific applications, such as hyperspectral environmental sensing and the testing of new wireless technologies such as CBRS (Citizen's Broadband Radio System). These collaborations have also helped establish a cloud based archival storage environment and access to tools that enable students to develop advanced ML based algorithms to infer the effect of the environment on lives, livelihoods, communities and industry. In addition to project discussions, Calit2 supported HPWREN's data server infrastructure, particularly when the number of cameras was rapidly increasing in the early 2010s. This collaboration has since resulted in staffing support for HPWREN (from Calit2's Pacific Research Platform project), equipment (multiple storage servers and services from both Calit2@UCSD and Calit2@UCI), and funding support for an AWS proof of concept experiment (for a variety of cloud services, including S3 and Glacier storage, image workflow management, and dynamic web hosting).


These photos show Cuyamaca Peak, Toro Peak, and the lower and upper parts of the tower at the Mount Laguna Observatory. At the Toro Peak tower are Jaime Bourdon, Jacob Keeton, Diarmuid O'Neill and Adam Brust, probably all humming to the tune of "Hi Ho, Hi Ho."


3.c Network Infrastructure and Growth

In 2000, the High Performance Wireless Research and Education Network was established to explore wide-area wireless network connectivity. The initial setup involved a point-to-point mountain-top backbone network comprised of IP routers interconnected via wireless links. This network connected the San Diego Supercomputer Center (SDSC) to several key backbone locations: Mount Woodson, North Peak, and Stephenson Peak, while providing Internet access to the HPWREN sites. The technology used was Ethernet over 45 Megabits per second wireless links operating in the public 5.8 GHz spectrum, which was state-of-the-art at that time. Based on HPWREN's requirements, Tom Hutton of the SDSC networking group planned, configured and bench tested the initially deployed routers and switches. This also allowed for a smooth and quick integration into SDSC's operational network, which then provided the HPWREN addition with its internet connectivity via SDSC. As the project evolved, additional backbone nodes were strategically added to enhance coverage and connectivity. These included Mount Soledad, Red Mountain, and eventually Toro Peak following a shift from Stephenson Peak to Monument Peak. These mountain-top locations were critical in creating wireless access links that connected various remote locations directly to the Internet via SDSC. The network's expansion continued with the addition of nodes to connect to several significant sites. These included the Palomar Observatory, San Diego State University, Lyons Peak, and Boucher Hill. Currently, the HPWREN network encompasses a vast area, with access capabilities in San Diego, Orange, Imperial, and Riverside County, showcasing a significant advancement in wireless network technology and infrastructure development since its inception.

Building on the extensive HPWREN network established since 2000, significant advancements have been made to the infrastructure. As of now, all backbone links in the network boast a minimum bandwidth of 200 Mbit/sec, utilizing licensed 6 GHz and 11 GHz links. These frequencies are particularly suited for long-range point-to-point connections, effectively covering distances up to 70 km (43.5 miles). This enhancement ensures more robust and reliable connectivity across the wide geographic spread of the network. In 2023, HPWREN took a leap forward in wireless technology by installing its first millimeter wave link. This cutting edge addition is designed for shorter distances, up to 5 km, but offers a substantial increase in capacity, reaching speeds of 10 Gbit/sec. This significant upgrade underscores HPWREN's commitment to staying at the forefront of wireless networking technology, continuously expanding its capacity and reach to meet growing demands and technological advancements.

Given the capabilities of the early HPWREN network, discussions about ad-hoc connectivity for firefighting incidents resulted in a Mobile Rapid Response Demonstration between Mount Woodson and Dos Picos Park in early 2001, with CDF Emergency Command Center officers on site. The objective was to prototype the deployment of connectivity to Incident Command Posts (ICP) and Base Camps in real time during active fires. This resulted in at least nine deployments of ICP connectivity to the Internet via HPWREN, starting with the 2003 Coyote Fire in San Diego County, and followed by the 2004 Eagle Fire connection in Riverside County. Often the ICPs were in difficult to reach locations, requiring the construction and deployment of one or more relays to support the path. The 2004 Mataguay Fire extended HPWREN's ability into nighttime deployments of ICP connections for active fires. 2005 saw two events with HPWREN ICP connectivity: the Volcan and Border50 fires. Some of these activities also resulted in the creation of persistent relays, such as at La Cima, Puerta La Cruz and towards the Dos Picos County Park, to be readily available for ad-hoc ICP needs in case of active emergency conditions. For a more detailed description of HPWREN deployments supporting fire fighting and emergency response activities, as well as network extensions to fire station and Emergency Operation Center sites, see this article.

Direct networking support for San Diego county fire stations evolved as part of the San Diego County and SDG&E supported ASAPNet from 2013 onward. The Advanced Situational Awareness for Public Safety Network (ASAPNet) became an HPWREN wireless network extension primarily serving fire stations in remote areas of San Diego County. ASAPNet is mapped into HPWREN, while using it as their "backbone" wide area network. Currently ASAPNet comprises 66 sites, and 12 relay sites originating from 16 mountaintops in San Diego County.

Several years ago, at the request of the Forest Fire Lookout Association, San Diego Riverside Chapter (FFLA-SDRC) staff, two fire lookout towers in San Diego County were connected to HPWREN. As these tower sites at Boucher Hill and High Point on Palomar Mountain were already functioning as HPWREN relays, the connections were relatively straightforward. The benefits to FFLA staff include improved access to weather and mapping data. The combination of such data, along with camera images and human expertise, facilitates the generation of more precise smoke reports. In addition, the Los Pinos tower, run by the US Forest Service, was connected as well.

In 2019, new backcountry community resource centers were established by SDG&E in partnership with local communities, which were later upgraded by HPWREN in 2021 to offer resilient Internet access. These community centers are activated as part of the Public Safety Program when power is dropped in the area during red flag warnings. These community centers include:

  • Dulzura Community Center - Internet access provided via Lyons Peak
  • Lake Morena Community Church -Internet access added via Los Pinos
  • Potrero Community Center -Internet access added via Los Pinos
  • Pine Valley Improvement Club - Internet access added via Cuyamaca Peak


    This table shows nine snapshots of the HPWREN network. Many more are at this web page. It starts with very modest connectivity in 2000. By 2012 it became too unwieldy to maintain the topology with a graphing program, and the maps were moved to Google Earth, as can be seen in the example in the lower right.


    3.d Sensor Deployments

    In December 2000, the HPWREN project initiated its first deployment of a Pan/Tilt/Zoom camera, manufactured by Axis, on Mount Woodson. This deployment occurred approximately four months after NSF had sent the grant award letter. Subsequently, an additional PTZ camera was deployed in May 2001 at the Mount Laguna Observatory, operated by San Diego State University (SDSU).

    About the same month (May 2001), camera related experiments were conducted at the Santa Margarita Ecological Reserve, also managed by SDSU. These experiments involved the introduction of a new type of camera, featuring higher resolution and a predictable fixed field of view. Initially, the Ricoh RDC-i700 Image Capturing Device was utilized, which retained the capability for zooming. However, this advanced camera, even by contemporary standards, incorporated an Ethernet attachment capability, enabling remote control, image capture, and image acquisition through that interface. The sole issue encountered was the camera's inability to automatically power up after a loss of input power. Ricoh demonstrated a cooperative spirit by providing HPWREN with a firmware update that rectified this problem. Subsequently, an i700 model was installed on Mount Soledad in early 2002. Later in September that year, a second PTZ camera was deployed on the network, this time at Monument Peak.


    These images show the first HPWREN-deployed sensor: an inexpensive network-connected weather station at Mount Woodson (photo from July 2000), followed by HPWREN's first deployed PTZ camera, which was also installed on Mount Woodson (December 2000 photo), and the PTZ camera at the Mount Laguna Observatory (photographed in May 2001).


    During this period, Pablo Bryant, our primary collaborator from SMER, introduced the team to IQinVision's IQeye cameras. These devices were distinguished by their lack of moving parts and their network-driven operation via Ethernet, rapidly becoming a preferred choice for deployment. IQinVision also provided valuable technical advice and facilitated the omission of the IR blocking filter during the manufacturing process for one of the cameras. Approximately mid-2003, HPWREN's initial set of four IQeye cameras, each offering a 90-degree field of view, was deployed on Monument Peak. Additionally, a single IQeye camera was installed at CAL FIRE's Ramona Air Attack Base in August of that year, shortly before the disastrous Cedar Fire affected San Diego County. After the fire, in December 2003, CDF Division Chief Randy Lyle expressed interest in getting their La Cima fire camp in the San Diego mountains connected via HPWREN, not the least since the phone lines had burned, and given the resulting profound lack of choices at that time. With the Cedar fire having burned next to and around La Cima, the surrounding area can probably be best described as resembling a moonscape. To create a wireless path between the main building and North Peak, an (initially solar-powered) relay had to be created atop a nearby hill. For HPWREN, this was another opportunity to deploy a fixed FoV camera, with this one pointing into the burned wilderness. As of 2023, this is HPWREN's sole remaining IQeye camera, which, with some interruptions, is now at about 20 years of collecting post-burn vegetation recovery data.

    Early 2004 witnessed further PTZ deployments on Lyons Peak and Red Mountain. In August of the same year, another set of four IQeye cameras was deployed, this time on Lyons Peak. By 2004, HPWREN had established two locations that facilitated side-by-side comparisons of PTZ cameras and fixed FoV cameras. Subsequently, the CAL FIRE ECC Chief and one of the Captains, who, as part of this collaboration, had been given video and control access by HPWREN for the PTZ cameras and had already used and controlled them during the 2003 Cedar Fire, were consulted regarding their camera preferences. Both individuals expressed a preference for the fixed cameras, citing their higher resolution and predictable view.

    Later, the project began to use Mobotix cameras instead, only leaving one iQeye camera remaining in active use today. The newer cameras have six megapixels per chip and separate color and monochrome imagers, for a total of two imagers per camera enclosure. Two chips times four cameras per site amounted to 48 megapixels being collected at a baseline rate of initially once per two minutes and, later, with more storage space becoming available, every minute. This image acquisition rate can be substantially increased as needed, such as during fires. Images are archived and converted into time-lapse videos every three hours. The benefit of the separate monochrome chip is higher camera light and near-infrared sensitivities. CDF quickly began using the cameras for observational, operational, and logistics support. In consequence, HPWREN moved away from deploying PTZ cameras. However, many years later, a separate UNR ALERTTahoe project reintroduced PTZ cameras to mountaintop wildfire observational systems around 2014.

    The ALERTTahoe project at the University of Nevada, Reno, installed Axis pan/tilt/zoom cameras at multiple locations around Lake Tahoe. Remarkable improvements to the Axis cameras in the decade since they were used in the HPWREN network included much higher environmental resiliency with their now 2 megapixel image resolution. Around the end of 2016 HPWREN partnered with ALERTTahoe to deploy their technology on Santa Ynez Peak in Santa Barbara County, alongside an HPWREN fixed 360 degree camera system and a weather station. These cameras proved to be invaluable together during the July 2017 Whittier Fire. In the Fall of 2017, in partnership with the newly created ALERTSDGE project, HPWREN deployed PTZ cameras at 11 existing HPWREN camera sites in San Diego County along with four new camera sites, with funding support from San Diego Gas and Electric.

    Starting in 2015, the County of Orange Area Safety Task Force (COAST) group in partnership with the Orange County Fire Authority (OCFA) embarked on an ambitious project to bring HPWREN capability into Orange County. This initiative involved identifying multiple sites for additional network links and cameras across the county. Brian Norton, a Battalion Chief for the OCFA, alongside Mike O'Connell from COAST, led this effort. In April 2018, the OCFA significantly upgraded its wildfire monitoring capabilities with the installation of six cameras - comprising two Pan-Tilt-Zoom (PTZ) and four Fixed Field of View (FFoV) units - along with a weather station atop a Southern California Edison (SCE) telecommunications tower situated on Santiago Peak, the highest point in Orange County. This strategic placement was made possible through a collaborative effort between Brian Norton and Troy Whitman, Senior Fire Management Advisor for SCE. The cameras, monitored by SCE and fire watch volunteers, were effective for early wildfire detection. Brian Norton, who had been promoted to OCFA Division Chief, highlighted the technology's importance in validating smoke reports and aiding fire watch patrols to respond effectively. The pan and tilt features of two PTZ cameras allowed for a more dynamic and strategic approach in observing potential wildfire situations. The Santiago cameras were in place for only four months when the Holy Jim fire broke out on August 6, 2018. The value of having two PTZ cameras on the opposite sides of the tower along with four FFoV cameras providing unobstructed 360 degree panoramic views was demonstrated as Santiago Peak burned on all sides of the peak over the next several days. This prototype of two PTZ and four FFoV cameras is now the standard throughout the HPWREN system.

    Starting in 2018 the Axis pan/tilt/zoom cameras supported by HPWREN were integrated into the newly formed ALERTWildfire project which was an outgrowth and merger of the ALERTTahoe and ALERTSDGE programs.

    Weather sensors were similarly deployed on HPWREN, starting in 2002, eventually alongside the deployed cameras, initially as a direct consequence of some initial weather-related wireless challenges. For example, HPWREN's original sensor suite on Monument Peak included a solid-state 3D ultrasonic anemometer, a more conventional anemometer, temperature, relative humidity, solar radiation, fuel temperature and fuel moisture probes, as well as a tipping rain-bucket and a Pan/Tilt/Zoom camera. See an example illustration.

    Based on more than two decades of field experience, multiple sensor types have been shown to be valuable for providing real-time situational awareness from a public safety perspective. It has also become evident that a one-size-fits-all approach is not optimal. At mountaintop sites with substantial infrastructure and unobstructed views, a combination of two pan-tilt-zoom (PTZ) cameras (for a consistent view without obstructions from the tower) collocated with four fixed field-of-view (FoV) cameras provides the most comprehensive coverage. These sites can be augmented with meteorological, seismic, and/or GPS sensors as needed to fill in gaps in coverage. Other sites may only require a subset of these or other sensors. New types of sensors, such as air and water quality sensors, will provide additional options in the future. The information received by the sensors can often be augmented by NOAA's GOES16/GOES18 hot spot alerts, which show the approximate location of a fire.

    More than 1,000 distinct public IP addresses access HPWREN camera data on a typical day, and far more can be observed during a significant fire or hazardous weather event. The U.S. National Weather Service is a significant user of HPWREN data and uses it on its forecast pages, such as those for Toro Peak and Mount Woodson. The majority of HPWREN meteorological sensors collect and archive real-time data once per ten seconds, with wind data stored at a once per second rate.


    a
    b
    c
    1
    2

    1a: Earthquake sensor on Toro Peak. 1b: Meteorological sensors on Monument Peak, with the highest instrument being a 3D anemometer which can help with insights at the edge between about 6300 ft elevation to below sea level desert. 1c: Fixed FoV camera on Monument Peak with ice shield protection. 2a: A common occurrence in San Diego mountains is rime ice in winter, shown here on a sonic anemometer at Monument Peak. 2b: Water sensor at the Santa Margarita Ecological Reserve. 2c: GPS sensor near the Ramona Airport measuring the movement of tectonic plates.


    3.e Data Access and Management Infrastructure

    Images from HPWREN cameras have always been publicly accessible via the HPWREN web pages and are also archived. Images are acquired from each camera every minute, with some exceptions for 10-second intervals, enabling the creation of time-lapse videos. One of the first major uses of post-processed time-lapse videos was a DVD documenting the progression of the 2003 Cedar Fire, available on YouTube, narrated by CDF Fire Captain Ron Serabia who flew on the Air Attack OV-10 during the fire. Both real-time and time-lapse videos are available on the HPWREN website and in the archives.

    Images (jpg) and time-lapse videos (mp4) were made accessible via a variety of tools integrated with the HPWREN website. A summary can be found at HPWREN Camera Image Download Overview. These tools allowed users to manually select images and videos for download, as well as to search for specific ranges of images based on various criteria, such as camera, date, and time. Also supported were NextCloud sites for a more graphical, point-and-click ("Google Drive"-like) experience for locating and bulk downloading HPWREN data. These tools remained available through 2022, when HPWREN began the transition to using object-oriented storage and cloud-based AWS services in response to weaknesses with the mass storage used at that time. Currently, only web-based manual lookup and downloads of imagery are available. Once these new storage/cloud technologies stabilize, we will re-address the need for more sophisticated and automated data access tools.

    In 2013, HPWREN began to make fire-related videos available on YouTube, beginning with the 2013 Chariot Fire, the 2007 Harris Fire, and most major fires in the HPWREN coverage area since summer of 2013. This archive also includes other non-fire-related videos, such as flooding events of the Santa Margarita River, habitat regrowth after wildfires, and interesting encounters during field work.

    In 2016, HPWREN started experimental live streaming of video data on YouTube. Prototyping was done using a controlled burn, observed from the Mount Laguna Observatory, and a flooding event in the Santa Margarita Ecological Reserve. The value of this technology really took off during the 2017 Whittier Fire observed from Santa Ynez Peak in Santa Barbara County. Feedback from various fire agency officers about the utility of HPWREN real-time streaming of camera images included:

    "Having been around this for so many years, I am always glad to see improvements (more sites, HD cameras) to the network. However, watching the live feed on the Whittier Fire last week gave all of us a chance to learn and put away something in our toolbox for later use."

    "The live feed and High Def allowed us to watch the preheating of the fuels from the fire at the bottom of canyon and then start the brush on fire farther up the hill. We were able to watch very important fire behavior without being in damage, record it and use it to app along to others. We got a up close look at how effective the helicopter water drops where, along with several air tanker drops over the two days. As I write this today, the fire last night picked up and came around the other side, we watched a hot shot crew on the hill protect it. If the IC or Operations were watching, they could see the crew was safe while working on top of the hill."

    "To be able to add live streaming to a few of the mountain tops in San Diego would be a great addition to an already fantastic network."

    Alternative storage and camera workflow experiments, supported by the Pacific Research Platform (PRP) and California Institute for Telecommunications and Information Technology (Calit2), took place from 2017 onward. Object-oriented storage systems such as EdgeFS and CEPH were prototyped. The "getcams" image fetching project concluded with multiple proof-of-concept regional real-time image processing capabilities in San Diego and Orange counties on multiple traditional and distributed storage back-ends.

    This experimental system was implemented as a service, spawning and maintaining a single, long-running process per camera. We were able to handle concurrent fetching from 150 cameras once per minute using a single 4 Core 4 GB VM. The system allowed image load to be spread across multiple servers as needed to support flexible scaling. The prototype system was tested at both UCSD and UCI. In parallel, a container-based version of the getcams system was also developed. This system was originally anticipated to present one option for upgrading legacy systems.

    The "getcams" system served as the foundation for a Calit2@UCI repository of Orange County HPWREN camera images. The Calit2/HPWREN collaboration was initially implemented in a virtual machine environment during 2020 and expanded and integrated into a container based application in 2021. The container based and VM based applications share a common source code, configuration files, and development environment. The UCI implementation supported Orange County camera images that were fetched, stored, and published using NextCloud@UCI. A similar proof of concept was developed at UCSD. NextCloud supported camera image exploration, access, and bulk download using a point and click web based interface.

    Around 2022, a data driven system was developed to automatically rebuild the HPWREN camera web pages from central spreadsheets which described all cameras. During this same period, we redesigned our image fetch system for improved scaling abilities. Finally, multiple storage technologies were investigated to better support our system and its ever increasing archival storage requirements. More detail on those storage efforts is offered below.

    The HPWREN camera images are currently utilized by a diverse range of entities, including government agencies, private corporations, and the general public. On an average day, these images are accessed by over 1,000 unique IP addresses. During significant events, such as large wildfires or mountain snowstorms, this number can easily increase by one or more orders of magnitude. HPWREN camera images are furthermore extensively used by the public media, such as local news networks which are using them on nearly a daily basis as background during the weather reports.

    Facilitated by HPWREN for approximately the past two decades, this data is made publicly available with minimal restrictions. The primary requirement is simply the acknowledgment of HPWREN as the source, if only as a way to document the usefulness of our data products.

    3.f Evolving Information Technology Infrastructure over Time

    The HPWREN network infrastructure has evolved over time through systematic upgrades and expansions of wireless network radios to support increased throughput and broader connectivity. Stratex, Harris, and later Exalt radios were used for backbone links during the first two decades, with Lucent, WiLAN, Trango, Radwin, and others used for the access links. Of note is the more recent migration to more modern SIAE Microwave communications equipment on over a dozen links beginning in 2019.


    a
    b
    c
    1
    2
    3

    1a: Pablo Bryant and Mark VanScoy working at HPWREN's level of the CAL FIRE tower on Red Mountain. 1b: Antennas were being exchanged on Monument Peak, due to the move from 5.8 GHz license-exempt to 6 GHz FCC-licensed spectrum. 1c: Air delivery of an antenna on a tower of SDSU's campus, which Pablo Bryant is receiving, while supported by Ron Serabia next to him on the tower. 2a: Christian Braun, Bud Hale, James Hale and (not visible in this picture) Ron Serabia assembling and moving an antenna prior to the installation. 2b: Winter at Monument Peak. 2c: This view out of one of the telescope domes at the Mount Laguna Observatory shows the HPWREN antenna in the background. 3a: In 2005 the Native American TDVNet helped HPWREN with a tower location they had built, as well as with manual labor, which then enabled HPWREN to connect CAL FIRE's Puerta La Cruz Conservation Camp. 3b: Moving from license exempt to FCC-licensed radio spectrum, some of the new antennas were temporarily stored at the Palomar Observatory inside the building for the 200" Hale Telescope dome. 3c: Helicopter-based antenna and radio replacement in 2007 on Lyons Peak.


    On the storage front, numerous storage evaluations and improvements have been made. As a significant user of large image storage, incremental additions of a few hundred terabytes were ultimately found to be insufficient for our ever-increasing image archival environment. From approximately 2014 to 2018, Calit2's Pacific Research Platform (PRP) project provided HPWREN with multiple servers, each with 100-200 terabytes of storage. Beginning in 2017, compression of JPEG images into MP4 videos has been used to reduce archival storage needs, extending our ability to keep archival imagery online for as long as possible. Around the same time, data was replicated in off-line storage using traditional external drives and some online services.

    Complementing PRP storage systems, HPWREN invested in additional local servers and investigated options for building a new distributed storage system. Ultimately, we decided to use our new server hardware to provide improved processing and storage capacity for our camera image fetch workflows (which were at the time being processed predominantly on VMs). This also addressed our shorter-term storage requirements. In approximately 2017, various object-store based distributed systems were investigated, including EdgeFS and CEPH. CEPH was ultimately selected for initial trial usage. By 2022, CEPH was being used for intermediate storage, while imagery was being slowly migrated to cloud services.

    In 2022, we began the process of adopting cloud storage for our longer-term storage requirements. This adoption of AWS cloud storage, enabled via a collaboration with Calit2@UCSD, was undertaken to provide regional storage independence and high availability to our users. A cloud-based system has been designed and initially adopted by mid-2023 for bulk storage. The new storage system will initially host archival data and eventually provide various web-accessible data services.

    Website enhancements in 2023 included upgrades in performance and navigation. The camera banner page became more responsive to screen changes, such as resizing or rotation, and supports a wider range of device types, sizes, and display capabilities. The "details" pointers now direct users to more information about specific cameras or camera collages. The cameras are now sorted alphabetically, and additional sort order options can be added over time. Each site's "details" page contains numerous archival data sources related to cameras and weather stations.

    HPWREN employs a variety of automated mechanisms to maintain server consistency and keep systems up to date. It has also incorporated Netbox among its software tools to assist in tracking various software, hardware, and networking configurations.

    3.g People and teamwork

    The foundation of HPWREN's success can be attributed to the early efforts of Hans-Werner Braun and Frank Vernon, who were garnering NSF support for the initiative. Initially, Frank Vernon and Hans-Werner Braun strategically recruited individuals with specialized skills, such as Glen Offield, whose expertise in radio and microwave tower infrastructure proved invaluable, as did the involvement of network researchers from the NLANR project. Over time, HPWREN expanded its network of collaborators through partnerships with various organizations. Notably, projects such as SDSU's Santa Margarita Ecological Reserve and collaborations with Native American reservations served as fertile ground for recruiting volunteers eager to contribute and enhance their knowledge of wireless networking technologies. Remarkably, one researcher from SMER, Pablo Bryant, demonstrated exceptional enthusiasm, eagerly ascending the towers each morning to assist the team. Furthermore, HPWREN attracted additional volunteers from the community, motivated by the inherent benefits of the program and the amiable spirit of collaboration that characterized the group.

    Two fundamental aspects of the HPWREN project resonated particularly well with participants: the potential impact of our work and the collaborative nature of our working environment. The impact of our work is significant, encompassing initiatives such as connecting research projects and communities to the internet, providing firefighters with enhanced communication technologies and real-time sensor data, and facilitating equitable access to information and data for Native American tribes.

    The collaborative work environment at HPWREN was unique in that it was structured as a leader-leader model, empowering each individual to excel in their roles while collectively driving the organization towards its objectives. Being part of an organization that not only delivers tangible outcomes but also enables its members to actively contribute to those outcomes fosters a strong sense of intrinsic motivation. This motivation often led us to work beyond the expected hours each week, driven by the significance of our mission.

    A quote from Todd Hansen: "As [now being] a Sr. Manager at a major tech company, I can trace many of the philosophies I practice in leading my team from the way the HPWREN team enabled each and everyone of us to contribute to the direction and impact of the project as a team of leaders."

    The HPWREN team currently features a group of professionals with distinct roles and expertise, each contributing to the network's success. Geoff Davis, as the Systems Architect, is responsible for the strategic architectural design of the system, ensuring its effectiveness and resilience. Adam Brust, working as a Systems Engineer, is pivotal in both engineering aspects and leading fieldwork, showcasing his multifaceted skills. Lisa Richards, another Systems Engineer, is integral to the day-to-day operations of HPWREN, ensuring smooth and efficient network functionality. Jim Hale, who has been with HPWREN since 2002, contributes his skills in technical support and wireless deployments, having started as a volunteer and now a key team member. Pablo Bryant from San Diego State University provides vital field support with his expertise in technology management, particularly in ecological reserves. Finally, Jamie Bourdon enhances the team's capabilities with additional field support, essential for successful operations in remote and challenging locations. Together, their diverse skills and experiences enable HPWREN to adeptly tackle complex challenges in wireless research and education networking.


    `
    a
    b
    c
    d
    1
    2
    3
    4
    5
    6
    7
    8
    9

    1a: May 2005 - Todd Hansen assembling communications gear during an exercise at the Coronado Bridge. 1b-1c: Pablo Bryant installing a camera on a tower at Mount Woodson. 1d: Sep 2019 - Geoff Davis multitasking at the 100' deck at Boucher Hill. 2a: Oct 2000 - Kimberly Mann Bruch during the connection of the Mount Laguna Observatory. 2b: May 2002 - Frank Vernon presenting seismic data. 2c: Dec 2006 - Jim Davidson exchanges batteries at the HPWREN Dos Picos relay. 2d: Oct 2018 - Frank Vernon riding a bucket elevator to the top of 150' solar tower at the Mount Wilson Observatory to install HPWREN Sensors. 3a: Aug 2007 - Ron Serabia at Red Mountain during a tower rescue exercise. 3b: Pablo Bryant on Mount Woodson. 3c: Jun 2007 - Teamwork made this sensor platform near the tidepools at the NPS Cabrillo National Monument possible; the photo shows Pablo Bryant, James Hale, Mike Maki (NPS), Susan Teel (NPS) and Andrea Compton (NPS). 3d: May 2011 - CAL FIRE Chief Marc Hafner scoping out the County in preparations for more firestation connections. 4a: Mar 2006 - Bud Hale napping in a comfy spot. 4b: Nov 2020 - Adam Brust trying to figure out how a screw came loose inside a PTZ dome at Toro Peak. 4c: Jun 2020 - Adam Brust driving a boom lift to Black Mountain for the repair of a failed PTZ. 4d: Mar 2008 - Hans-Werner Braun taking video clips of wolves at the CWC while inside of the enclosure with them. 5a: May 2006 - Jim Davidson watching one of the county's firefighting helicopters fly by the Dos Picos relay during an exercise. 5b: Dec 2003 - CDF Division Chief Randy Lyle in discussions with Bud Hale during the installation of the La Cima fire camp connection in an area that burned over during the Cedar Fire. 5c: Glen Offield, Frank Vernon and Tom Hutton at the Mount Laguna Observatory. 5d: Feb 2001 - Todd Hansen looking for the other end of a link. 6a: Oct 2010 - CAL FIRE Chief Marc Hafner during installation work on Mount Woodson. 6b: Aug 2010 - Larry Smarr, Calit2, presents on joint activities. 6c: Aug 2022 - Ramesh Rao experimenting with radios near the HPWREN tower at the SDSU Mount Laguna Observatory. 6d: Aug 2010 - Larry Smarr and Hans-Werner Braun presenting at a County/State meeting. 7a: May 2002 - James Hale during an exercise at the Coronado Bridge. 7b: Oct 2021 - Glen Offield and Jamie Bourdon at Toro Peak repairing multiple devices due to lighting strike. 7c: Mar 2002 - Pablo Bryant looking to find Toro Peak from a high elevation level of the Red Mountain tower. 7d: Jan 2019 - HPWREN and Alert climbers celebrate completion of tower safety and rescue training on the 160' deck at Red Mountain. From left to right: Adam Brust (HPWREN), Jamie Bourdon (HPWREN), Pablo Bryant (HPWREN), Colby Nicholson (Alert), Ernie Aaron (Alert), Geoff Davis (HPWREN) and Boe Derosier (Alert). 8a: Jamie Bourdon enhances the team's capabilities with additional field support and medical first responder training, essential for successful operations in remote and challenging locations; the picture is from the helicopter operations on SMER Highlands. 8b: May 2002 - Christian Braun helping at the Monument Peak site. 8c: Jun 2022 - Matt Nornbert, Pablo Bryant and Jamie Bourdon installing a solar powered HPWREN site in Escondido. 8d: Jan 2023 - Jamie keeping warm with a friend at the Palomar Fire station on the way to a High Point site rebuild. 9a: NPS interns Kelly Lion and Sarina Cassaro are helping with an HPWREN/NPS sensor platform near the Zone-1 tidepools at the Cabrillo National Monument. 9b: Kelly Lion, Susan Teel and Sarina Cassaro are experimenting with a prototype mobile real-time water quality measurement system in the Santa Margarita Ecological Reserve. 9c: Michael Peralta (TDVnet) at one of their sites above Pala. 9d: Mar 2005 - Bud Hale and Dan Zieber working on the link upgrade to FCC-licensed spectrum for the Palomar Observatory.


    3.h The impact of HPWREN on the demand for HPWREN

    Around the time of the inception of HPWREN, internet connectivity was beginning to proliferate within suburban communities; however, rural and remote regions remained predominantly disconnected. Furthermore, mobile phones during that era lacked the substantial network connectivity capabilities that were yet to come, and laptops had not gained widespread adoption at that time. Consequently, individuals seeking internet access were typically required to commute to their place of employment or connect from a residential location within an urban or suburban setting, often necessitating the use of a desktop computer.

    Rural internet access was not really a thing at that time and as such, many of the research programs, reservations and first responders we worked with were not ready to think about all of the advances such a connection could bring to their programs. Initially, some collaborators were apprehensive or did not seem interested in working with us.

    It was intriguing to observe the expanding range of possibilities facilitated by rural internet access, the evolving conceptualizations among collaborators, and the subsequent attraction of additional collaborators who had not previously exhibited an interest in partnering with us. One notable instance involved the integration of high speed data transfer capabilities, which enhanced world-class research endeavors focused on supernovae and asteroid tracking. This integration resulted in the establishment of globally distributed systems, encompassing research instruments situated in various parts of the world and functioning collectively as a distributed system characterized by real-time exchanges of substantial data volumes. Notably, the transformation witnessed fire stations transitioning from entities uncertain about the utility of the internet to viewing it as a critical resource in their daily operations and emergency response endeavors.

    In essence, the beneficiaries experienced an expanded understanding of the potential applications of the internet within their respective contexts, including research, community engagement, and emergency response initiatives, as a direct consequence of establishing this connectivity. This transformation elevated the internet availability from a mere amenity to an indispensable resource, ensuring its continued utilization, funding, and expansion by these beneficiaries to this day.

    4. Enabling R&D Support and Technology Demonstrations

    Since its inception, HPWREN infrastructure has supported a variety of domain research and education projects. For example, in September 2001 HPWREN and multiple agencies met at a local National Guard Armory to test, evaluate and compare satellite and terrestrial-based microwave link options for ad-hoc deployments. That same year HPWREN connected to the Santa Margarita Ecological Reserve, allowing researchers remote access to real-time field data. The Palomar and Mount Laguna Observatories were also connected, allowing the transmission of real-time images directly from the observatories to worldwide astronomy laboratories. The following list shows some example activities, while https://www.hpwren.ucsd.edu/news/ shows many more articles in the form of news updates.

    An approximately magnitude 5 earthquake event in October 2001 in southern California facilitated the demonstration of an earthquake early warning system. ANZA seismic sensors were streaming their data to SIO, where the automated real-time process triggered a warning which was multicast on HPWREN. Based on logs, a computer in Hans-Werner Braun's house "knew" that the event was coming several seconds before it hit.

    2002 saw SDSU collaborating with HPWREN to test 2.4 GHz connectivity to obtain images in real-time from an airborne platform, utilizing SDSU's Global Change Research Group (GCRG) aircraft. A follow up took the lessons learned and applied them to better radio/antenna packaging on the airplane and on the ground.

    In 2002 HPWREN also participated in a technology demonstration on the Coronado Bridge by providing wireless connectivity to sensors, some of which were also provided by HPWREN.

    Another interesting event in 2002 was when the U.S. Navy Deep Submergence Unit (USN-DSU) brought their ship, the Kellie Chouest, as well as the Remotely Operated Vehicle (ROV) Scorpio, to the waters off Scripps pier. Their mission: to locate and dive the site of SeaLab II, a USN/SIO undersea habitat that operated in August-October 1965. This would be the first return to the site since the habitat was pulled up. HPWREN was able to test new communication technologies embodied in the HPWREN link between the Kellie Chouest and Scripps pier. In addition to these tests, HPWREN and ROADNet researchers aboard the ship experimented with a web-accessible, high resolution camera.

    In 2003, a CDF captain at their Ramona Air Attack Base tested an HPWREN-provided monochrome camera on their North American Rockwell OV-10 Bronco observation aircraft. He then reported being able to see a fire through thick black smoke, due to the camera's high sensitivity in near-infrared.

    Greg Aldering (LBNL) used his slide set at the HPWREN Users meeting in October 2003 to explain that the HPWREN connection at the Palomar Observatory was essential in facilitating turning the telescope system into a real-time observatory which could trigger followup observations at other observatories elsewhere, based on detections at Palomar. With the 48 inch Samuel Oschin telescope already undertaking automated sky surveys on behalf of the Nearby Supernova Factory (SNfactory), Near Earth Asteroid Tracking (NEAT) and Quasar Equatorial Survey Team (QUEST), data was now being streamed off the mountain via HPWREN to the NERSC facilities at LBNL for real-time and non real-time analysis. At that time, SNfactory needed a day for processing the data, to find targets to focus on during the next night. With the follow on PTF program, by then having faster computers, it became possible to find anomalies in the data in near real-time to forward target information for a telescope in Hawaii for automated spectral measurement follow up before the same part of the sky that allowed for the initial observation at the Palomar Observatory was "arriving" above Hawaii. The image next to this text shows the last of those slides from 2003.

    The devastating Cedar Fire in late October 2003 created an opportunity to demonstrate use for the cameras. CDF firefighters and their San Diego Emergency Command Center. They were directly able to view and control HPWREN PTZ cameras to help them to stay on top with the fire progression. For example, Captain Ron Serabia, who was directing air drops the day after the evening fire ignition, was able to already gather information from the cameras during the night while also utilizing the PTZ controls on HPWREN's camera on Mount Woodson. He later narrated a video with images of the fire, which is available on YouTube.

    The Scripps Orbit and Permanent Array Center (SOPAC) has maintained since 2010 a network of dozens of continuously monitoring GNSS stations in southern California as part of the California Real Time Network (http://sopac-csrc.ucsd.edu/index.php/crtn/) using HPWREN to reliably transmit high-rate (1sps) real-time (<1 s latency) data back to SIO. To support research into earthquake and early warning systems, SIO MEMS accelerometers are collocated at the GNSS stations to also provide 100 Hz seismogeodetic displacements and velocities. The HPWREN-transmitted GNSS data are also rebroadcast in support of high-precision (<1 cm) real-time kinematic (RTK) surveys made by surveyors, scientific and academic researchers, geographic information system (GIS) professionals, state and local governments, and the navigation and spatial information industries.

    Due to access issues at an HPWREN backbone site, which still persists today, in 2005 the Sheriff's Department facilitated the repair of a fuel moisture sensor at that site via one of their helicopters. Moisture sensors are a critical metric for firefighters in making determinations about the degree of vegetation risk of ignition.

    In December of 2006, the National Science Foundation extended an invitation to HPWREN to partake in the NSF's exhibition booth at the annual meeting of the American Association for the Advancement of Science (AAAS) to be held in the city of San Francisco in 2007. The primary objective of this participation was to demonstrate a real-time cyberinfrastructure that facilitates research and educational endeavors through the medium of Live Interactive Virtual Explorations (LIVE) of remote scientific locations.

    In October 2007, when San Diego County was hit again by massive wildfires, HPWREN cameras provided valuable information to first responders, as well as to the public at large. By then the HPWREN cameras were a well known and used asset across firefighting agencies. Some summaries regarding these fires can be found at https://www.hpwren.ucsd.edu/news/20071030/ and at https://www.hpwren.ucsd.edu/news/20071110/. In addition, four cameras, each with a 90-degree field of view, continuously monitoring all 360 degrees around Lyons Peak for the several days the Harris fir lasted. The fire even burned over the top of Lyons Peak and underneath the tower which the HPWREN equipment is on. The images were turned into time-lapse videos, and are available for the North, East, South and West cardinal directions.

    A collaboration with the National Park Service enabled HPWREN to collect large digital images from atmospheric visibility cameras at the Cabrillo National Monument. This used inexpensive consumer-grade cameras, which could be of value to ad-hoc deployments, for example in a wildland fire situation, possibly as expendable systems. Since HPWREN archives long term image data, it is possible to support post-fire research across multiple years, for example, to observe vegetation regrowth.

    The HPWREN project supported a collaboration in 2009 between CAL FIRE and the MIT Lincoln Lab on Enhanced Situational Awareness for First Responders. A pair of visible light and thermal IR cameras were mounted underneath the OV-10 Air Attack plane, and then tracked by a mountain top receiver with a steerable tracking antenna, and thus connected to the HPWREN network node on that mountain.

    Multicast distribution of HPWREN sensor data supported the ability to easily analyze the resulting sensor output from multiple machines without impacting data collection or archival. This supported the creation of a production system in 2004, which is still active today in 2023, in which a fully automated and continuously running process receives meteorological data, analyzes it in real-time for Santa Ana conditions, and immediately alerts the CAL FIRE Emergency Command Center and others of such events via paging or email.

    More recent collaborations include:

    Since HPWREN started to deploy cameras at its backbone sites in 2002 for persistent environmental observations, it had been approached numerous times about concepts and technologies for automated fire plume detection. For varieties of reasons this has not yet resulted in a broad technology adaptation, with one of the main reasons often being an expectation of continuous and inefficient video streams from the cameras being sent across a multi-purpose network to centralized servers.
    [https://www.hpwren.ucsd.edu/news/20190823/]

    HPWREN imagery was also used by SDSU astronomy researcher Robert Quimby to demonstrate the effects of light pollution in San Diego
    [https://www.hpwren.ucsd.edu/news/20210928/]

    The HPWREN cameras often capture images of meteors streaking through the sky. A particularly good example was identified by HPWREN user Ross Seibert in one image from the west-facing camera at Lyons Peak the night of November 9, 2021.
    [https://www.hpwren.ucsd.edu/news/20211201/]

    HPWREN imagery was also used by SDSU astronomy researcher Robert Quimby to demonstrate the effects of light pollution in San Diego
    [https://www.hpwren.ucsd.edu/news/20210928/]

    High school student Colin Wessels used air traffic data delivered from an ADS-B receiver on Mount Woodson by HPWREN in his study on how the Coronavirus affected air traffic in Southern California
    [https://www.hpwren.ucsd.edu/news/20210826]

    Open Climate Tech, an all volunteer nonprofit, has built an automated early wildfire smoke detection system using images from HPWREN cameras.
    [https://www.hpwren.ucsd.edu/news/20210430/]

    Members of the Biggs and Mladenov Labs have installed water quality sondes and pressure transducers to measure stream level on Boulder Creek, San Diego River at Mission Trails, and the Tijuana Estuary. The systems report the data in real-time via HPWREN.
    [ https://www.hpwren.ucsd.edu/news/20210323/]

    A request from an outside researcher looking for access to 1 Hz wind data from HPWREN meteorological sensors led to the creation of a large downloadable data set across many years.
    [https://www.hpwren.ucsd.edu/news/20220812/]

    At the request of the San Diego River Park Foundation, HPWREN has provided cameras and instrumentation at the Boulder Creek Preserve Environmental Monitoring Research Program.
    [https://www.hpwren.ucsd.edu/news/20220315/]


    a
    b
    c


    Hit by many dislocations based on high winds or falling ice, the HPWREN team had to take a new approach to physically stabilize the cameras. top-a: In late 2018, Glen Offield designed new mounts for the Mobotix fixed FoV cameras. Two cameras share one mount which guarantees 90 degree orthogonal orientation for each of the four cardinal directions. It saves time for the field engineers when aligning the cameras as well as provides a more robust housing for the harsh elements. top-b and top-c: These photos show the dual Mobotix mount deployed in the field along with a PTZ and Met sensor, showing the North and East facing cameras at Cuyamaca Peak. The bottom-a and bottom-b photos represent what our solar powered sites look like. bottom-a: HPWREN Solar site Upper Talega, installed June 2019 bottom-b: HPWREN Solar site Chino Hills, installed December 2022


    More examples of HPWREN activities and collaborations can be found in the HPWREN News section.

    5. Outreach

    5.a Native American Connections and Outreach

    HPWREN initially connected three Native American reservations, specifically through tribal learning centers, providing them with access to high-speed Internet connectivity (see here). For the initial connection at the Pala reservation in late 2000, HPWREN performed all of the installation tasks necessary for network connectivity, and then worked with the tribal education center director to develop programming initiatives that utilized the new high-speed connectivity. For the second connection at the La Jolla reservation, five tribal members worked with the HPWREN team on the construction process; once connectivity was established, the HPWREN team worked with the La Jolla tribal education director to develop programming similar to the Pala model. In the third case, Rincon, in early 2001, members of the reservation education center's staff participated in the installation process, while well knowing what this was for, and then used the Pala and La Jolla programs as models for their own.


    a
    b
    c


    top-a: The HPWREN team (Bud Hale, Christian Braun, Dave Cheney, Todd Hansen) drilling a hole for the tower segment for the Pala antennas. top-b: Joint La Jolla and HPWREN team during the construction phase of their relay. top-c: Hunwut Turner and Michael Peralta working with HPWREN on the relay installation which then provided internet connectivity to their education facilities. bottom-a: Doretta Musick's Learning Center at the Pala Reservation. The note on the wall saying "internet access is 30 minutes per person" was removed following the installation of the HPWREN connection. bottom-b: When the Rincon relay was finished, all participants signed inside the enclosure for the radio equipment.


    This early direct Native American involvement, along with its technology and expertise transfer, increased with each HPWREN reservation installation. Eventually, the connectivity was expanded to all reservations in San Diego County through separate Native American activities that were supported by separate funding for the by then tribal-owned Tribal Digital Village Network (TDVNet) under the leadership of Matthew Rantanen, which is part of the Southern California Tribal Chairmen's Association (SCTCA), with HPWREN becoming a collaborator. During this collaboration of network construction, expansion, and distance learning programming, the HPWREN team served as unofficial technical advisors to TDVNet, with TDVNet having the sole responsibility for the architecture, construction, and maintenance of their tribal-owned cyberinfrastructure. Simultaneously, the research team mentored a collection of Native American collaborators. One of the key collaborators had been Tribal Digital Village Network Administrator Michael Peralta. In working to expand the now growing tribal infrastructure, he made significant design changes and improvements in the network architecture to better address tribal needs.

    Peralta stated that he is very excited about the continued collaboration between HPWREN and TDVNet. "The met station will provide key data on the development of the alternative energies that power our backbone. I also believe that the new cameras will help all of our Native American communities by providing us the opportunity of being better informed during emergencies like the 2003 and 2007 fires."

    Extensive collaborations between HPWREN and Native Americans later incorporated networked sensors and environmental cameras. Using the Tribal Digital Village Network (TDVNet) backbone location described in https://www.hpwren.ucsd.edu/news/050701.html, the partnership decided to augment the existing infrastructure with meteorological sensors, as well as 360-degree view high-resolution environmental camera systems for mutual use. With TDVNet staff providing access to a site, a team consisting of UCSD and SDSU staff, as well as two NPS/SDSU interns, undertook installation work.

    Significant ongoing activities include distance education as part of the Live Interactive Virtual Explorations project (e.g., https://www.hpwren.ucsd.edu/news/20080215/), TDVNet providing support to HPWREN based on connectivity via microwave towers they have built and own as part of their network, and the initiation of a new collaboration using environmental sensors and high-resolution cameras, all supported via the HPWREN and TDVNet network infrastructure.

    5.b Remote education

    In addition to supporting education centers at Pala, Rincon, and La Jolla Reservations, HPWREN participated in a variety of educational activities at the California Wolf Center, the NPS Cabrillo National Monument, the Warner Community Resource Center, the Reuben H. Fleet Science Center, and multiple California State Park and National Park Service locations.

    In Summer 2003, the HPWREN team expanded their educational outreach by collaborating with the California State Parks Distance Interpretation Program. This initiative enabled remote interactive presentations and educational experiences using broadband connections. Specifically, they worked with Anza-Borrego State Park, allowing park rangers, educators, and administrators from various locations to virtually explore the park's features, such as the fossilized oyster beds in Fish Creek, from urban conference rooms situated about 100 miles away. This innovative use of high-speed wireless technology by HPWREN was praised for providing unique educational access to remote park resources, previously inaccessible to traditional school trips. This initiative marked a significant advancement in utilizing technology for environmental education and remote natural resource management. For more details of remote education activities, see here.


    (left) Research Associate Maria Wiehe, Student Conservation Association, demonstrates how field researchers can use a real-time network-connected camera, in this case a Ricoh i700, to monitor the Santa Margarita river crossing at SDSU's Santa Margarita Ecological Reserve in 2001. (right) Melinda Booth, California Wolf Center, is interactively presenting live to a remote audience in November 2005 during an early phase of HPWREN's Live Interactive Virtual Exploration activity, which was orchestrated by Kimberly Mann Bruch.


    5.c User Workshops

    The first of many user workshops occurred in May, 2002 at SDSU's Mount Laguna Observatory. Users came to hear presentations and discuss current and planned usages of the HPWREN network. Speakers included Observatory Director Paul Etzel, HPWREN PI Hans-Werner Braun, HPWREN co-PI Frank Vernon, Greg Aldering from the Palomar Observatory, Dan Cayan from the Scripps Institution of Oceanography, Mike Peralta from the Tribal Digital Village Network, and Pablo Bryant from the SDSU Santa Margarita Ecological Reserve.

    In subsequent years, user meeting were held at Palomar Observatory, San Diego County Sheriff's Department, SDSU Santa Margarita Ecological Reserve, Ramona Air Attack Base, Cabrillo National Monument, Pala Native American Reservation, SDSC, U.S. Fish and Wildlife Service's San Diego Bay National Wildlife Refuge, and the SDG&E Mission Control Site.


    These photos show attendees at the first HPWREN User Workshop, which was held in May 2005 at SDSU's Mount Laguna Observatory. The second photo is from the fourth workshop in November 2006 at CAL FIRE's Ramona Air Attack Base. The photo on the right is from the 2009 workshop at the Palomar Observatory.


    5.d Social Media

    Read or track more about HPWREN activities on a number of social media sites: Facebook, Twitter, and YouTube. News articles are at https://www.hpwren.ucsd.edu/news/.

    6. HPWREN in 2023

    Today, the HPWREN network supports numerous applications, ranging from the transfer of high-volume astronomical data generated by the Palomar Observatory, to a steady output of continuous low-volume traffic from devices such as earthquake sensors and meteorological stations, delivering real-time telemetry data. HPWREN includes permanent sites as well as those created temporarily and on short notice, such as firefighter base camps and Incident Command Posts (ICP). HPWREN saw use at ICPs in several of the major wildfires hitting San Diego County across many years. Many meteorological stations are deployed which give real-time fine grained space and time weather data coverage, a number of which support up-to-the-second wind data. In addition to weather data (wind direction and speed, temperature, relative humidity, and fuel moisture) some of these stations also have soil moisture and rainfall sensors. In the past, a few sites experimented with acoustic sensors to observe ocean, river, and wolf sounds. They could easily be reinstalled or adapted to new requirements, and can serve as an example of how enabling cyberinfrastructure can support new and expanded sensor networks.

    The HPWREN network provides Internet access to over 60 regional fire stations and community access to both meteorological stations and 140+ cameras (color and near-infrared), often at high elevations, around Southern California. Coupled with a collection of servers at the Scripps Institution of Oceanography (SIO), the San Diego Supercomputer Center (SDSC) and Calit2 (all at UCSD), and backup CENIC gateways at San Diego State University (SDSU), University of California Irvine (UCI), and Saddleback College, the larger community can access these data and image resources via reliable public web sites and other modes of interaction.


    The above two graphics show the extent of the HPWREN backbone network, as well as the maximum link performances, as of the end of 2023.


    7. Future Directions

    Besides maintaining network stability, over the last several years the most important HPWREN objective has been to increase the network performance and reach, both to support more sites across Southern California as well as to substantially increase the speed and redundancy of the backbone network that holds the system together. Significant progress has been made in network expansion into San Diego, Orange, Los Angeles, Santa Barbara, Imperial and Riverside counties, as we continue to expand reach and fill in gaps of coverage as defined by the fire agencies.

    HPWREN produces open data, while making such data available to various common operating pictures, emergency management and response agencies, researchers, environmental groups, and the general public.

    A significant area of interest for the last few years has been to enable and facilitate activities that utilize HPWREN images for early fire detection, for example the 2019 Image Processing Experiments for Fire Plume Detection via Fixed HPWREN Cameras summary. Projects such as AI For Mankind and Open Climate Tech have already shown substantial progress, often being able to identify a fire plume within its first few image frames of visibility. A summary about image processing experiments has resulted in the HPWREN Fire Ignition images Library for neural network training (FIgLib), which provides hundreds of image sequences of fire ignitions. Another collaboration, which included WIFIRE and SAGE, in conjunction with the Argonne National Laboratory, is labeling the FIgLib data sets with both boxes and contours, so the data can be directly used for neural network training. The labeling itself was done by ANL, and a demonstration video about "How Fires Start: Fire ignition and early progression labels" is publicly available.

    Another exemplar of work in progress pertains to the HPWREN interactive images flow interface. This serves as an alternative to the three-hour videos per camera that HPWREN has made available on its website for over a decade. It transmits the original sized JPEG images to the user machine across a user-defined image-origin period, such as thirty minutes ago until the most recent image. The images are displayed as they load, after which a time series image will be displayed, reflecting the position of the mouse on the screen window. When the mouse is moved horizontally at that time, the time series image will move with it, with the left side of the window representing the location for the first image of the time series and the right side of the window showing the last. For exact frame-by-frame movement, the keyboard arrow keys can be utilized.

    The primary advantage of this, in comparison to MP4 video, is that the images are downloaded as JPEG chunks into the user machine, and as such, they can be processed much more expeditiously than by repeatedly downloading the images from the web server, based on user interactions. This is especially pertinent if a user wishes to step forward and backward in a time series very rapidly, while displaying each and every frame, as opposed to typical video applications that tend to move backward on key frames only and with the user potentially skipping numerous frames. Given the absence of interframe compression, this necessitates a substantial amount of bandwidth and significant quantities of memory in the user machine, relative to compressed video. At present, an individual time series is restricted to a six-hour period, primarily to maintain the volume of network traffic for downloaded images within a reasonable amount. The default setting is to display the last thirty minutes of a selected camera. A six-hour download could consist of hundreds of JPEG images that may exceed 100 MB.

    From a technical perspective, HPWREN is transitioning its image acquisition, storage, archival and web publication mechanisms to adopt AWS cloud services. This is a joint collaboration with Calit2 and is still evolving.


    The preceding pair of photographs illustrates the ongoing development of communications technology. The first photograph depicts the initial antenna, installed in May 2001, which established a 45 Mbps license-exempt connection between the Palomar Observatory and HPWREN. At the time, this represented cutting-edge technology, with each radio unit costing approximately $10,000.

    In 2005, a substantial upgrade was implemented, resulting in a 155 Mbps connection licensed by the FCC. Subsequently, in 2016, this connection underwent further enhancement through the integration of a new suite of radios and was also redirected from North Peak to Monument Peak, yielding a substantial increase in bandwidth to 210 Mbps. Furthermore, in the same year HPWREN successfully established a second connection to the Observatory from Boucher Hill. Presently, constructive dialogues are being conducted with the objective of elevating the connection speed to the remarkable range of 1.4 gigabits per second, which is about 30 times as fast as HPWREN's initial connection to the Palomar Observatory.


    8. Options to Replicate

    We are frequently asked how one might replicate the HPWREN networking capabilities in their own environment. When starting from scratch, homegrown, smaller-scale options are available from multiple vendors for both the network and sensor devices. Ubiquiti is currently one of the more popular choices for relatively low-cost unlicensed point-to-point links and wider area networks, and mesh network kits are readily available from multiple vendors for starting small and then building up. Internet and AI search will provide numerous options.

    The same holds true for IP cameras. Vendors can provide hardware and software for managing the network, cameras, and imagery. Multi-platform options exist as well, thanks to ONVIF standards. Many commercial cameras today offer their own software (custom NVR options), and some vendors offer video management software that works with many (ONVIF compliant) IP cameras. One thing to be mindful of is that many camera manufacturers appear to prioritize selling a service over just selling equipment, with the images often being collected at a related data storage location, with them retaining image ownership. This can make the data accessible to wherever the (authorized) user is, such as on a smartphone, while leaving the questions about privacy and who can make access decisions open.

    See this document as an example of a good overview of IP camera network design.

    9. Additional Photos

    These are a few additional photos, to show some more of the significant scope of HPWREN activities and collaborations:


    a
    b
    c
    d
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    1a: CAL FIRE's Ramona Air Attack Base as viewed from above, with the HPWREN antenna visible on the left side of and connected to the building while connecting the AAB via a link to Mount Woodson. 1b: Scenario composite of the HPWREN connection to support the base camp during the 2005 Volcan Fire near Julian. 1c: Temporary HPWREN relay installation on the north side of Palomar Mountain for HPWREN's first ICP connection, specifically in support of the Coyote Fire operations in San Diego County. 1d: Pablo Bryant and Hans-Werner Braun during a fire drill. 2a-2c: A multi-agency fire exercise in 2005 allowed HPWREN to demonstrate an airdropped network relay installation transported by a Sheriff Department's helicopter. 2d: Relay able to connect a possible CAL FIRE base camp at the Dos Picos County Park via Mount Woodson, plus two supporting cameras monitoring environment conditions. 3a: Preparations for a high-speed network connectivity event in the Anza Borrego Desert State Park, showing HPWREN access via a link to Monument Peak. 3b: Popsicles in the San Diego mountains. 3c: Tower rescue practice (indoors at UCSD). 3d: Palomar 200" telescope from below, photographed straight up, but without the 5 meter mirror in place. 4a: Wolf pups at the HPWREN-connected California Wolf Center. 4b: Test run of a LIVE event at the NPS Cabrillo National Monument. The woman standing is Susan Teel, who had a formal collaboration with HPWREN for several years on behalf of NPS. 4c: Sensor site near the tide pools at the Cabrillo National Monument. 4d: Lightning strike, emanating from a tower at Monument Peak, with the image taken by a pixel-comparing motion-detect camera. 5a: A close encounter by Pablo Bryant. 5b: Motion-detect image of an Osprey in 2008. 5c: Crow vs. Raptor - who will win? Motion-detect photo from 2004. 5d: ... but who is really chasing whom (2024)? 6a: Preparation flight for the 2007 helicopter-based replacement of an antenna on Lyons Peak, with the top of the tower in sight. 6b: Hovering right above the tower to see what the pilot will see during the actual deployment. 6c: Pablo Bryant underneath the SDSU Sky Arrow working on his radio/sensor setup in preparation for and actual flight with ground-air-ground data communication. 6d: Pablo Bryant giving a LIVE presentation, using stereo video, from the river crossing at the Santa Margarita Ecological Reserve. 7a: Todd Hansen pre-building a solar-powered relay for deployment in the La Jolla Reservation. 7b: Assembled solar-powered relay. 7c: Early prototype to test a remote-controllable Ricoh i700 camera connected via a wireless link. 7d: Enclosure, built by Pablo Bryant, to experiment with 3 consumer-grade cameras to achieve a combined 180 degree view. This was then deployed at SMER. 8a: James Hale in Copter 12 for repairs on Lyons Peak in January 2007. 8b: Hans-Werner Braun and James Hale in Copter 12 for repairs on Lyons Peak. 8c: Landing in a Sheriff's Department helicopter on Hans-Werner Braun's property in July 2005 to pick up a sensor for repair on Lyons Peak. 8d: After landing. 9a: For the return flight from Lyons Peak in January 2007, Copter 10 is arriving. 9b: Copter 12 during a wildland fire exercise, 9c: Navy helicopter picking up water during a wildland fire exercise, 9d: Antenna replacement at the Palomar Observatory 10a: Conceptual slide (the pieces implemented, but not as a system) of an initial sensor data distribution setup. 10b: Coyote Fire setup in 2003 in San Diego County. 10c: Connection of the Fallbrook CDF firestation in 2003. 10d: The extend of the Cedar Fire in 2003, and why the La Cima fire camp needed an Internet connection.


    10. References

    See news articles at https://www.hpwren.ucsd.edu/news/. See references at the end of "HPWREN Retrospective: 20 Years of HPWREN Collaborations with Firefighting and Other Public Safety Agencies".

    11. Acknowledgments

    We extend our deepest gratitude for the support and contributions that have been pivotal in the advancement of HPWREN. We are immensely grateful to San Diego Gas and Electric (SDG&E) for their Resilience Program support, with special thanks to Brian D'Agostino and Mark Mezta for their guidance and contributions.

    Further, we wish to acknowledge the County of San Diego for their support which facilitated crucial upgrades to the HPWREN communication equipment, and for their funding support for ASAPNet. This support has been instrumental in enhancing the connectivity to remote fire stations.

    HPWREN has also benefited greatly from major operational support provided by the Palomar Observatory, California State Parks, San Diego County Fire, and SDG&E. Their collaboration and assistance have been essential for our operations and in achieving our objectives.

    Moreover, we are thankful for the in-kind support received from CALFIRE, California Department of General Services, California Governor's Office of Emergency Services, the San Diego Sheriff's department, SDG&E, Southern California Tribal Chairmen's Association Tribal Digital Village Network, USMC Camp Pendleton, and the USFS Cleveland National Forest. The provision of tower access and use has been a cornerstone in our efforts to enhance communication and monitoring capabilities across the regions we serve.

    We would also like to express our gratitude to the many more individuals who, despite not being explicitly mentioned here, have nonetheless made valuable contributions to the advancement and improvement of HPWREN.

    Lastly, but certainly not least, it is important to acknowledge the crucial role played by the National Science Foundation's initial support, which commenced in 2000 and spanned over a decade. Without this support, none of the accomplishments we have described here would have been possible.

    This collective support underscores the power of partnership and collaboration in driving forward initiatives that benefit our communities and the environment. We are deeply appreciative of each organization's contribution and look forward to continuing our work together towards shared goals.

    Appendix - HPWREN camera deployments over time

    This table lists cameras that were added to HPWREN, year by year. The year shown is based on the oldest data for a camera in HPWREN's archive, and in some cases may not be the year of the actual initial deployment.


    • 2002 Santa Margarita Ecological Preserve
    • 2004 California Wolf Center
    • 2004 Lyons Peak
    • 2005 La Cima
    • 2005 Sky Oaks
    • 2005 Sweetwater Marsh
    • 2005 Toro Peak
    • 2006 Mount Laguna
    • 2007 Palomar Observatory
    • 2007 San Diego Supercomputer Center
    • 2009 Mesa Grande
    • 2009 Mount Woodson
    • 2009 North Peak
    • 2009 Spanish Landing
    • 2011 Calit2 UCSD Building
    • 2011 Castro Peak Santa Monica Mountains National Recreation Area
    • 2011 National Park Service
    • 2011 Red Mountain (Fallbrook)
    • 2011 Spanish Landing
    • 2011 Temecula
    • 2011 Volcan Mountain
    • 2012 High Point
    • 2012 Los Pinos
    • 2012 Sky Oaks
    • 2013 Big Black Mountain
    • 2014 Black Mountain
    • 2014 Cabrillo National Monument
    • 2014 Laguna Observatory
    • 2014 Volcan Mountain
    • 2015 Otay Mountain
    • 2015 San Miguel Mountain
    • 2016 Boucher Hill
    • 2016 Mount Woodson
    • 2016 San Diego Supercomputer Center
    • 2016 Santa Ynez Peak
    • 2017 Birch Hill
    • 2017 Boucher Hills
    • 2017 Coronado Hills
    • 2017 Cowles Mountain
    • 2017 High Point
    • 2017 Laguna Observatory
    • 2017 Los Pinos
    • 2017 Lyons Peak
    • 2017 Monument Peak
    • 2017 Mount Soledad
    • 2017 Near Anza
    • 2017 Otay Mountain
    • 2017 Red Mountain
    • 2017 Santa Margarita Ecological Preserve Highlands
    • 2017 Toro Peak
    • 2017 Volcan South
    • 2018 Los Angeles County Helibase 69 Bravo
    • 2018 Mesa Grande
    • 2018 Red Mountain
    • 2018 Santiago Peak
    • 2018 Wilson Observatory
    • 2019 Boucher Hill
    • 2019 Los Angeles County Helibase 69 Bravo
    • 2019 Mount Laguna Observatory
    • 2019 San Clemente
    • 2019 Santa Barbara Mesa
    • 2019 Signal Peak
    • 2019 UCSB
    • 2019 Upper Bell
    • 2019 Upper Talega
    • 2020 Cuyamaca Peak
    • 2020 Green Peak
    • 2020 Los Pinos
    • 2020 Near Anza
    • 2020 San Juan Hills
    • 2020 San Miguel
    • 2020 San Vicente
    • 2020 Toro Peak
    • 2021 Boulder Creek
    • 2021 Chino Hills
    • 2021 Felicita
    • 2021 Laguna Hills
    • 2021 Mesa Grande
    • 2021 Mount Lee
    • 2021 Perris
    • 2021 Red Mountain
    • 2021 Santiago Peak
    • 2021 Sky Oaks Jolly Roger
    • 2021 Tijuana Estuary
    • 2021 Via Verde NM
    • 2021 Volcan Tract III
    • 2021 White Star
    • 2022 Buffalo Bump
    • 2022 Coronado Hills
    • 2022 Cuyamaca Peak
    • 2022 Rincon Del Diablo
    • 2023 UC Santa Barbara