HPWREN Camera Image Access and Download OverviewRevised in Oct 2022 and Jan 2023 (from original of March 18, 2021)This is an evolving document, you may want to recheck occasionally. The camera/sensor pages show images from equipment that are part of various research and infrastructure projects or funding sources. The system is not a guaranteed operational service, and may even produce incorrect data. If you use the data for anything serious, at least check the time stamps (but don't even entirely rely on those!), e.g., to make sure an image is recent. Not all cameras and/or sensors are HPWREN owned, and a specific data source may belong to an HPWREN collaborator. Please note that the cameras page is accessed by many and access consumes significant bandwidth and processor resources. Excessive access, (such as high-resolution versions at more than the image update frequency), may result in exclusion. The HPWREN camera image websites underwent significant changes in early 2022. The look and feel was updated (see the May 2022 notes further down this page for details) and the image workflow was adjusted to take advantage of added storage capabilities (CEPH and AWS Glacier long-term storage in particular). Also added were capabilities requested by the public and our first-responder user communities. It is inappropriate use of materials found on this web site, including images, videos and other products, if HPWREN is not credited and/or watermark-like imprints (such as text on images or videos) are being modified or removed, for example via image cropping, if only as recognition helps the HPWREN project with continued support for the cameras and other sensors, and being able to make the data freely available. See this broader and more legalesed document for more details. "How to..." for image access Images and videos seen on the various HPWREN camera sites, such as /cameras, are available to the public for download from our archives. The crediting requirement mentioned above applies to the usage of such data products, for example if images or other work products are published. The latest HPWREN camera images and videos are available in various display formats at: The individual cameras typically show their current near real-time image, and should automatically update approximately every minute. Recent collections of images and videos are found in the directory structure described in #2 below. In addition, this YouTube site shows many time lapse videos of events such as wildland fires. HPWREN servers capture, process, publish and archive images and videos around the clock. Images are typically captured once per minute, and videos, combining sets of images, are being generated every three hours. This data is collected from a few hundred camera image sources around San Diego, Imperial and Orange County, and beyond, with new cameras being added fairly frequently. Our archives are spread across multiple servers and we provide multiple tools for their access. Images are stored as JPEG files and videos are stored as MPEG-4 files. Our Archives data back to almost 2000 with past images coalesced into (and replaced by) MPEG-4 videos to reduce storage requirements by using the video format's interframe compression. Those MPEG-4 videos can be downloaded and the near original quality jpg files can be re-extracted, if desired, using FFmpeg. This is most readily done on a Unix system, example re-extraction command line: ffmpeg -i Q4.mp4 -f image2 -qscale 0 -vcodec mjpeg Q4-%03d.jpg FFmpeg is available on Windows as well. See an example perl script for recovering JPEG images from HPWREN's MPEG-4 files in Appendix A below. HPWREN provides a number of mechanisms for users to search for, view, and if desired download single or multiple images and videos (in custom packaged zip files when larger amounts of data are requested). These tools include the following:
Frequently asked questions -- "How often are the images updated?" Most, but not necessarily all, camera images are usually updated every two minutes by two 60-second interleaved data servers, to create minute by minute images. The a2/a3/a4/a5 in the upper left corner text refer to the specific server that collected the image. -- "Are the yellow-text time stamps accurate? The yellow-text time stamps in the upper left of the images is added by a centralized image acquisition server, and they reflect the actual time of when the server requests an image from a camera. The server should have quite accurate time stamps, due to its time synchronisation via NTP with other time-accurate systems. -- "When are the time-lapse animations generated?" Typically after the end of the three-hour period at 00:00, 03:00, 06:00, 09:00, 12:00, 15:00, 18:00 and 21:00. It takes time to generate videos for 400+ cameras, and the conversion, currently done in alphabetical order by camera name, may take multiple hours. -- "How are the animations created?" The animations are automatically created every three hours from the full sized images using ffmpeg. As of 21 December 2022 it uses: $cmd="/bin/ffmpeg -framerate 8 -pattern_type glob -i \"$dirname/$CAMERA/large/$dstamp/$APTAG/*.jpg\" -vcodec libx264 -pix_fmt yuv420p -crf 21 -g 10 -threads 0 -metadata title=$dstamp-$APTAG-$CAMERA -y $dirname/$CAMERA/temp.mp4"; -- "What is the black dot that appears in the middle of the sun?" Photon overload on the CMOS imager chip. -- "What do the camera error images mean"
-- "Some of the ptz cameras are labeled "'place name' south", but the image does not appear to be a view to the south. Why is that?" A 'ptz' camera is not designed to point in a single direction, but rather can "pan, tilt, and zoom" to show different areas of interest. A camera with a label of "south" indicates that it is mounted on the south side of the tower. Nominally, the camera will be pointed in the direction that corresponds to its label, but when there is an event of interest, such as a wildfire, you may see the 'south' camera pointed in an alternate direction. Some cameras are also set up in a "patrol" mode, where they take images pointed in a different direction during a defined interval. Some sites have co-located fixed and ptz cameras. For cameras that are set up in a fixed position, see /cameras/ rather than /cameras/ptz.html. You should be able to read the current direction of a ptz camera by looking at the status line at the bottom of the image where the orientation is reported as the x offset in degrees. An x value of 0 indicates the camera is pointed north. If you see a negative value, that would be the direction measured counterclockwise from north, i.e. -90 degrees would be due west. A positive value is the direction measures clockwise from north, i.e. +90 would indicate the camera is pointed due east. The z value indicates if the camera is zoomed in, if you see a number larger than 1. We hope you find these tools useful and encourage any feedback you may have for us at contact infomation. Even your stories about what you use the images for would be appreciated. Appendix A Perl script example for recovering JPEG images from MPEG-4 files The following perl script shows how to re-extract the original jpg files to their original (unix epoch data string) filenames. Assumes you downloaded an MPEG-4 file from https://c1.hpwren.ucsd.edu/archive/sp-s-mobo-c/large/2018/20180806/MP4/Q4.mp4 and the corresponding text file from https://c1.hpwren.ucsd.edu/archive/sp-s-mobo-c/large/2018/20180806/MP4/Q4.txt you should be able to cd to the directory where the two files are and use the following Perl script, using the Q4 prefix as the only input parameter:
#!/usr/local/bin/perl
$vname=$ARGV[0]; # e.g.: Q4 -- just the file name prefix
open(I,"$vname.txt") || die("Cannot open input file $vname");
system("ffmpeg -i $vname.mp4 -f image2 -qscale 0 -vcodec mjpeg %05d.jpg");
$lcnt=0;
while(<I>){
($t,$t,$t,$t,$t,$t,$t,$t,$string,@r)=split(" ",$_);
($t,$t,$t,$site,$t,$date,$Q,$time)=split("\/",$string);
($time,@r)=split(".jpg",$time);
if($lcnt++ == 0){
$dir=sprintf"$site-$date-$Q";
if(-e $dir){die("directory name already in use");}
mkdir($dir,0755);
}
$if=sprintf"%.5d",$lcnt;
printf"$if.jpg to $dir/$time.jpg\n";
rename("$if.jpg","$dir/$time.jpg");
}
close(I);
Some of the .txt files may use a different format, so your mileage may vary. Also, a way to download an entire directory would be to use the command: wget -e robots=off -r --no-parent -R "index.html*" https://www.hpwren.ucsd.edu/cameras/LTA/bm-n-mobo-c/large/20221017/Q4/ Some notes related to the new version of the webpage as of May 2022. The primary reason for creating this new version was that the old version became difficult to manage as more and more cameras got installed. Specifically in the old version changes to its many files had to be done by hand, while the new version uses a few simple (currently spreadsheet) configuration files from which its thousands of HTML/web pages get generated or installed. A currently incomplete set of KML files is also being generated from those spreadsheets to show HPWREN sensor and network attributes in Google Earth. This allows real-time data to be displayed in Google Earth. The columns sticking up, visible when rotating the image, show the wind speed (height) and wind direction (the more towards red, the closer to NE wind, to get a quick view of Santa Ana-ish conditions). Those towers are clickable for all the met data from that site. Other changes include:
The old cameras web page is still around but not linked in and can be restored if need be. The new one may also change over time. You are welcome to submit comments and suggestions for improvements via the HPWREN feedback mechanism referenced on the main HPWREN web page. |
