HPWREN team members have had a busy year. In addition to our normal ongoing infrastructure maintenance, and more intensive around the clock support for our emergency responders during wildfires, we have accomplished numerous upgrades to our equipment and expansions of our services. Below is a summary of these 2017 activities (from our logs, monitors, trouble tickets and 140 field service reports), categorized as:

• Maintenance efforts (both proactive and reactive)
• Network backbone enhancements and expansion
• Deployments of new cameras
• Replacement (though virtualization) of legacy servers
• Improved systems management and monitor tools
• Website enhancements
• Outreach activities
• Regular team member and stakeholder meetings
• R&D efforts

1) General maintenance activities -- keeping the HPWREN system operational

The HPWREN services, many of which are visible on http://hpwren.ucsd.edu, depend upon the correct operation of many elements, near to campus and far into the field, including IT (computing and networking) hardware, systems and application software, web and data services, wired and wireless communications and all the components that support them (towers, radios, antennas, power systems, batteries, solar panels etc). Maintenance and repair of these components become day to day priorities to keep the HPWREN system operational.

2) Network backbone enhancements and expansion

Backbone sites are those critical to the HPWREN network and its services. They include UCSD and SDSU campus connection points as well as numerous mountain top radio sites spanning San Diego and with incursions into Orange, Riverside and Imperial counties. We upgraded nine HPWREN backbone sites to accommodate 1 Gbps throughput on their Ethernet interfaces, eliminating multiple 100 Mbps choke points. We also increased the effective bandwidth on the southern portion of the network between UCSD and SDSU by lighting up a new 10 Gigabit CalREN XD fiber link.

Lyons Peak is one of our key backbone sites and it received numerous upgrades, including ugrading the radios there used to connect to Monument Peak, Los Pinos and SDSU. The new radios are Exalt Extreme Air systems. The site also received upgraded cameras (Mobotix M15s) and one of its antenna dishes got a new cover (the old one was shredded).

We also applied for and received 4.9 Ghz licensing from the FCC for 24 of our backbone sites to allow us to use the most efficient spectrum and bandwidth available.

3) Deployments of new cameras

HPWREN has clusters of (usually 4 color and 4 near infrared capable) fixed cameras around the county, pointing North, South, East and West, which require regular routine maintenance. This year, in response to the needs of the community and our stakeholders, we have added more of these clusters and later in the year many new PTZ (pan, tilt, zoom) cameras that are remotely controllable by emergency responders.

We deployed 15 ALERTSDGE PTZ cameras between Sept 1 and Dec 14. Eleven of those installations were completed by Oct 1 (these are visible at http://www.alertwildfire.org/sdge/ as well as /cameras). New sites include Toro Peak, Big Black Mountain, Birch Hill, Little Black Mountain, Boucher Hill, Coronado Hills, Cowles Mountain, High Point, Los Pinos, Lyons Peak, Mount Laguna, Mount Laguna Observatory, Mount Woodson, Anza, Otay Mountain, Red Mountain, Santa Ynez Peak, Southwest Escondido, Toro Peak and Volcan South. ALERTSDGE is a collaboration between University of Nevada, Reno Seismological Lab, San Diego Gas and Electric, and HPWREN.

We also reactivated Santa Margarita Ecological Reserve cameras and upgraded them using Mobotix M15s.

4) Replacement (though virtualization) of legacy servers

HPWREN has been in service since 2000. Many of its servers are getting dated and nearing the end of their life. We have been upgrading this "legacy" equipment slowly when addressing anticipated failures and are also working on plans for upgrading all of them. This year we virtualized servers associated with web, data and sensor collection services, which stabilized much of our web presence.

We also migrated some of the HPWREN services to new servers which are now handling the bulk of the storage of camera images, all image archival data, all of the daily animations (done on every camera every 3 hours), and some backup web services that are available even if the main HPWREN server is down. These archival data services are available at http://dl-hpwren.ucsd.edu/archive and redundant current camera images are available at http://hpwren-cams.ucsd.edu/cameras.

5) Improved systems management and monitoring tools

HPWREN automates as much as possible to improve support and maintenance efficiency and to diminish its overhead. We have implemented Puppet configuration management for all HPWREN servers, allowing us to bring up new servers (even virtual machines) faster. We use custom versions of Intermapper network monitoring tools to monitor our networks and the various networking, sensor and end user equipment attached to them. We use a customized Jira ticket tracking system to keep track of all of our activities. This year we created and handled 165 outage-related and 581 systems level jira tickets.

6) Website enhancements /

In addition to regular new content on our web site home page (such as news articles, supporting the outreach activities below as well), we have added new camera access, additional access to camera image archives, new ways to display camera image data for first responders using data that automatically updates its images every minute, new more flexibly formatted camera images (some which work well even on small mobile devices), higher resolution videos of all camera image collections, and access and control for PTZ cameras.

We are now supporting access to HPWREN data in KML format, allowing it to be downloaded and displayed using Google Earth. In our weather data section, /Sensors/ we have added a real time weather sensor map focusing on Santa Ana wind events. On /cameras/ there is now a link to "KML sensor interface" at which you can download camera site location data from which Google Earth can provide mapping.

We are providing data to other projects including the ALERTWildfire Program led by Dr. Graham Kent and Dr. Ken Smith of the University of Nevada, Reno; Dr. Neal Driscoll and Dr. Frank Vernon, UC San Diego; and Dr. Doug Toomey, University of Oregon:

• ALERTSDGE http://www.alertwildfire.org/sdge/
• ALERTSoCal http://www.alertwildfire.org/socal/

and the Wifire Program led by Dr. Ilkay Altintas, San Diego Supercomputer Center

• Wifire https://wifire.ucsd.edu/
• Wifire Firemap https://firemap.sdsc.edu/

7) Outreach activities

In addition web site articles and expanded data access, HPWREN also promotes its activities through social media using YouTube, Facebook and Twitter. You will find interesting and informative videos showing off our capabilities on YouTube at https://www.youtube.com/user/hpwren/videos. See also our Twitter feed at https://twitter.com/hpwren and our Facebook page at https://www.facebook.com/HPWREN.UCSD.EDU/.

8) Regular team member and stakeholder meetings

HPWREN holds regular in person and teleconference meetings with their support staff and stakeholders, including alternating biweekly HPWREN and ASAPNet meetings.

9) R&D efforts

HPWREN is always looking for new ways to expand the utility of their data, improve their levels of service and add redundancy of infrastructure to insure HPWREN remains a highly available system. In addition to modernizing our equipment, we are re-architecting HPWREN network flows (routing) across regions to take advantage of redundant paths between the HPWREN internal radio based network and the public Internet. We are working in conjunction with SDSC, SDSU, UCI and CENIC to coordinate route advertisements across San Diego and Orange (and later Riverside) counties to allow for reliable networking even in the face of regional disasters.

We are also investigating emerging technologies, such as container based methodologies and new distributed file system designs, to support and improve HPWREN data services. Using servers at multiple campuses, we are redesigning our data flows to insure those services and their data remain available even if some of the servers were to fail or become unreachable.

Moving in to 2018…

Having had such a busy 2017, we are now embarking on even more new activities in 2018, including:

1. Completing our Mount Soledad move
2. Moving our Boucher Hill site to new tower
3. Moving Red Mountain cameras to upper level of tower
4. Adding Sky Oaks to our High Point communications link
5. Upgrading multiple HPWREN backbone links
6. Adding 5 more new ALERTWildfire/HPWREN camera systems
7. Adding new Firestation support for ASAPNet, including the newly constructed Julian JCFPD Fire station #56 as well as other new stations
8. Expanding HPWREN to Santiago Peak and then on to UCI
9. Revamping data collection, processing and the web interface for HPWREN meteorological sensors