Hydroclimate Data Retriever

HyRiver (formerly named hydrodata) is a suite of Python packages that provides a unified API for retrieving geospatial/temporal data from various web services. HyRiver includes two categories of packages:

• Low-level APIs for accessing any of the supported web services, i.e., ArcGIS RESTful, WMS, and WFS.

• High-level APIs for accessing some of the most commonly used datasets in hydrology and climatology studies. Currently, this project only includes hydrology and climatology data within the US.

Video Gallery

Pangeo Showcase

Taher Chegini

ESIP IT&I

Taher Chegini

WaterHackWeek 2020

Taher Chegini

UH Seminar

Taher Chegini

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

High-level APIs for accessing some pre-configured web services

PyNHD

Navigate and subset mid- and high-res NHD, NHDPlus, and NHDPlus VAA using WaterData, NLDI, ScienceBase, and The National Map web services.

PyGeoHydro

Access NWIS, NID, HCDN 2009, NLCD, and SSEBop databases.

Py3DEP

Access topographic data through The National Map’s 3DEP web service.

PyDaymet

Access Daymet for daily, monthly and annual summaries of climate data at 1-km scale for both single pixels and gridded, over the North America, Hawaii, and Puerto Rico.

PyGridMET

Access GridMet for daily climate data at 4-km scale for both single pixels and gridded data, over the conterminous United States.

PyNLDAS2

Access hourly NLDAS-2 forcing data.

HydroSignatures

A collection of tools for computing hydrological signatures

Low-level APIs for connecting to supported web service protocols

PyGeoOGC

Send queries to and receive responses from any ArcGIS RESTful-, WMS-, and WFS-based services.

PyGeoUtils

Convert responses from PyGeoOGC’s supported web services protocols into geospatial and raster datasets.

AsyncRetriever

Asynchronous send/receive requests with persistent caching.

Getting Started

Why HyRiver?

Some major capabilities of HyRiver are as follows:

• Easy access to many web services for subsetting data on server-side and returning the requests as masked Datasets or GeoDataFrames.

• Splitting large requests into smaller chunks, under-the-hood, since web services often limit the number of features per request. So the only bottleneck for subsetting the data is your local machine memory.

• Navigating and subsetting NHDPlus database (both medium- and high-resolution) using web services.

• Cleaning up the vector NHDPlus data, fixing some common issues, and computing vector-based accumulation through a river network.

• A URL inventory for some popular (and tested) web services.

• Some utilities for manipulating the obtained data and their visualization.

Installation

You can install all the packages using pip:

$ pip install py3dep pynhd pygeohydro pydaymet pygridmet pynldas2 hydrosignatures pygeoogc pygeoutils async-retriever

Please note that installation with pip fails if libgdal is not installed on your system. You should install this package manually beforehand. For example, on Ubuntu-based distros the required package is libgdal-dev. If this package is installed on your system you should be able to run gdal-config --version successfully.

Alternatively, you can install them using conda:

$ conda install -c conda-forge py3dep pynhd pygeohydro pydaymet pygridmet pynldas2 hydrosignatures pygeoogc pygeoutils async-retriever

or mambaforge (recommended):

$ mamba install py3dep pynhd pygeohydro pydaymet pygridmet pynldas2 hydrosignatures pygeoogc pygeoutils async-retriever

Additionally, you can create a new environment, named hyriver with all the packages and optional dependencies installed with mambaforge using the provided environment.yml file:

$ mamba env create -f ./environment.yml

Dependencies

AsyncRetriever

• aiohttp[speedups]>=3.8.3

• aiohttp-client-cache>=0.8.1

• aiosqlite

• cytoolz

• ujson

PyGeoOGC

• async-retriever<0.17,>=0.16

• cytoolz

• defusedxml

• joblib

• multidict

• owslib>=0.27.2

• pyproj>=3.0.1

• requests

• requests-cache>=0.9.6

• shapely>=2

• typing_extensions

• ujson

• url-normalize>=1.4

• urllib3

• yarl

PyGeoUtils

• cytoolz

• geopandas>=0.10

• netcdf4

• numpy>=1.21

• pyproj>=3.0.1

• rasterio>=1.2

• rioxarray>=0.11

• scipy

• shapely>=2

• ujson

• xarray>=2023.01

PyNHD

• async-retriever<0.17,>=0.16

• cytoolz

• geopandas>=0.10

• networkx

• numpy>=1.21

• pandas>=1

• pyarrow>=1.0.1

• pygeoogc<0.17,>=0.16

• pygeoutils<0.17,>=0.16

• shapely>=2

Py3DEP

• async-retriever<0.17,>=0.16

• click>=0.7

• cytoolz

• geopandas>=0.10

• numpy>=1.17

• pygeoogc<0.17,>=0.16.1

• pygeoutils<0.17,>=0.16.1

• rasterio>=1.2

• rioxarray>=0.11

• scipy

• shapely>=2

• xarray>=2023.01

PyGeoHydro

• async-retriever<0.17,>=0.16

• cytoolz

• defusedxml

• folium

• geopandas>=0.10

• h5netcdf

• hydrosignatures<0.17,>=0.16

• matplotlib>=3.5

• numpy>=1.21

• pandas>=1

• pygeoogc<0.17,>=0.16

• pygeoutils<0.17,>=0.16

• pynhd<0.17,>=0.16

• pyproj>=3.0.1

• rioxarray>=0.11

• scipy

• shapely>=2

• ujson

• xarray>=2023.01

PyDaymet

• async-retriever<0.17,>=0.16

• click>=0.7

• geopandas>=0.10

• numpy>=1.21

• pandas>=1

• py3dep<0.17,>=0.16.1

• pygeoogc<0.17,>=0.16.1

• pygeoutils<0.17,>=0.16.1

• pyproj>=3.0.1

• scipy

• shapely>=2

• xarray>=2023.01

PyGridMET

• async-retriever<0.17,>=0.16

• click>=0.7

• geopandas>=0.10

• numpy>=1.21

• pandas>=1

• pygeoogc<0.17,>=0.16

• pygeoutils<0.17,>=0.16

• pyproj>=3.0.1

• shapely>=2

• xarray>=2023.01

PyNLDAS2

• async-retriever<0.17,>=0.16

• h5netcdf

• numpy>=1.21

• pandas>=1

• pygeoutils<0.17,>=0.16

• pyproj>=3.0.1

• rioxarray>=0.11

• xarray>=2023.01

HydroSignatures

• numpy>=1.21

• pandas>=1

• scipy

• xarray>=2023.01

Additionally, you can also install bottleneck and numba to improve the performance of some computations. Installing pyogrio is highly recommended for improving the performance of working with vector data. For NHDPlus, py7zr and pyogrio are required dependencies. For retrieving soil data, you should install planetary-computer and pystac-client.

Software Stack

A detailed description of each component of the HyRiver software stack.

PyNHD

PyGeoHydro

Py3DEP

PyDaymet

PyGridMET

PyNLDAS2

HydroSignatures

AsyncRetriever

PyGeoOGC

PyGeoUtils

PyNHD: Navigate and subset NHDPlus database

Features

PyNHD is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services.

This package provides access to several hydro-linked datasets:

• StreamCat

• WaterData

• NHDPlus MR

• NHDPlus HR

• NLDI

• PyGeoAPI

• GeoConnex

• Mainstems

• HUC12 Pour Points

These web services can be used to navigate and extract vector data from NHDPlus V2 (both mid- and high-resolution) database such as catchments, HUC8, HUC12, GagesII, flowlines, and water bodies.

Moreover, the PyGeoAPI service provides four functionalities:

1. flow_trace: Trace flow from a starting point to up/downstream direction.

2. split_catchment: Split the local catchment of a point of interest at the point’s location.

3. elevation_profile: Extract elevation profile along a flow path between two points.

4. cross_section: Extract cross-section at a point of interest along a flow line.

PyNHD also provides access to the entire NHDPlus dataset for CONUS (L48) via nhdplus_l48 function. You can get any of the 31 layers that are available in the NHDPlus dataset. You can also get NHDPlus Value Added Attributes on Hydroshare and ENHD. These datasets that do not have geometries, include slope and roughness, among other attributes, for all NHD flowlines. You can use nhdplus_vaa and enhd_attrs functions to get these datasets.

Additionally, you can get many more derived attributes at NHD catchment-level through two sources:

• Select Attributes for NHDPlus Version 2.1 Reach Catchments from an item on ScienceBase

• EPA’s StreamCat dataset.

They both include hundreds of attributes such as hydroclimate properties, water quality, urbanization, and population. In addition to NHD catchment summaries, they also have their network-accumulated values (both upstream and divergence-routed). You can use nhdplus_attrs, epa_nhd_catchments, streamcat functions to get these datasets.

Additionally, PyNHD offers some extra utilities for processing the NHD flowlines:

• flowline_xsection and network_xsection: Get cross-section lines along a flowline at a given spacing or a network of flowlines at a given spacing.

• flowline_resample and network_resample: Resample a flowline or network of flowlines based on a given spacing. This is useful for smoothing jagged flowlines similar to those in the NHDPlus database.

• prepare_nhdplus: For cleaning up the data frame by, for example, removing tiny networks, adding a to_comid column, and finding terminal flowlines if it doesn’t exist.

• topoogical_sort: For sorting the river network topologically which is useful for routing and flow accumulation.

• vector_accumulation: For computing flow accumulation in a river network. This function is generic, and any routing method can be plugged in.

These utilities are developed based on an R package called nhdplusTools and a Python package called nldi-xstool.

All functions and classes that request data from web services use async-retriever that offers response caching. By default, the expiration time is set to never expire. All these functions and classes have two optional parameters for controlling the cache: expire_after and disable_caching. You can use expire_after to set the expiration time in seconds. If expire_after is set to -1, the cache will never expire (default). You can use disable_caching if you don’t want to use the cached responses. The cached responses are stored in the ./cache/aiohttp_cache.sqlite file.

You can find some example notebooks here.

Moreover, under the hood, PyNHD uses PyGeoOGC and AsyncRetriever packages for making requests in parallel and storing responses in chunks. This improves the reliability and speed of data retrieval significantly.

You can control the request/response caching behavior and verbosity of the package by setting the following environment variables:

• HYRIVER_CACHE_NAME: Path to the caching SQLite database for asynchronous HTTP requests. It defaults to ./cache/aiohttp_cache.sqlite

• HYRIVER_CACHE_NAME_HTTP: Path to the caching SQLite database for HTTP requests. It defaults to ./cache/http_cache.sqlite

• HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to one week.

• HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.

• HYRIVER_SSL_CERT: Path to a SSL certificate file.

For example, in your code before making any requests you can do:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/aiohttp_cache.sqlite"
os.environ["HYRIVER_CACHE_NAME_HTTP"] = "path/to/http_cache.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"
os.environ["HYRIVER_SSL_CERT"] = "path/to/cert.pem"

You can also try using PyNHD without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install PyNHD using pip after installing libgdal on your system (for example, in Ubuntu run sudo apt install libgdal-dev):

$ pip install pynhd

Alternatively, PyNHD can be installed from the conda-forge repository using Conda or Mamba:

$ conda install -c conda-forge pynhd

Quick start

Let’s explore the capabilities of NLDI. We need to instantiate the class first:

from pynhd import NLDI, WaterData, NHDPlusHR
import pynhd as nhd

First, let’s get the watershed geometry of the contributing basin of a USGS station using NLDI:

nldi = NLDI()
station_id = "01031500"

basin = nldi.get_basins(station_id)

The navigate_byid class method can be used to navigate NHDPlus in both upstream and downstream of any point in the database. Let’s get the ComIDs and flowlines of the tributaries and the main river channel upstream of the station.

flw_main = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamMain",
    source="flowlines",
    distance=1000,
)

flw_trib = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="flowlines",
    distance=1000,
)

We can get other USGS stations upstream (or downstream) of the station and even set a distance limit (in km):

st_all = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="nwissite",
    distance=1000,
)

st_d20 = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="nwissite",
    distance=20,
)

We can get more information about these stations using GeoConnex:

gcx = GeoConnex("gauges")
stations = st_all.identifier.str.split("-").str[1].unique()
gauges = gpd.GeoDataFrame(
    pd.concat(gcx.query({"provider_id": sid}) for sid in stations),
    crs=4326,
)

Instead, we can carry out a spatial query within the basin of interest:

gauges = pynhd.geoconnex(
    item="gauges",
    query={"geometry": basin.geometry.iloc[0]},
)

Now, let’s get the HUC12 pour points:

pp = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="huc12pp",
    distance=1000,
)

Also, we can get the slope data for each river segment from the NHDPlus VAA database:

vaa = nhd.nhdplus_vaa("input_data/nhdplus_vaa.parquet")

flw_trib["comid"] = pd.to_numeric(flw_trib.nhdplus_comid)
slope = gpd.GeoDataFrame(
    pd.merge(flw_trib, vaa[["comid", "slope"]], left_on="comid", right_on="comid"),
    crs=flw_trib.crs,
)
slope[slope.slope < 0] = np.nan

Additionally, we can obtain cross-section lines along the main river channel with 4 km spacing and width of 2 km using network_xsection as follows:

from pynhd import NHD

distance = 4000  # in meters
width = 2000  # in meters
nhd = NHD("flowline_mr")
main_nhd = nhd.byids("COMID", flw_main.index)
main_nhd = pynhd.prepare_nhdplus(main_nhd, 0, 0, 0, purge_non_dendritic=True)
main_nhd = main_nhd.to_crs("ESRI:102003")
cs = pynhd.network_xsection(main_nhd, distance, width)

Then, we can use Py3DEP to obtain the elevation profile along the cross-section lines.

Now, let’s explore the PyGeoAPI capabilities. There are two ways that you can access PyGeoAPI: PyGeoAPI class and pygeoapi function. The PyGeoAPI class is for querying the database for a single location using tuples and list while the pygeoapi function is for querying the database for multiple locations at once and accepts a geopandas.GeoDataFrame as input. The pygeoapi function is more efficient than the PyGeoAPI class and has a simpler interface. In future versions, the PyGeoAPI class will be deprecated and the pygeoapi function will be the only way to access the database. Let’s compare the two, starting by PyGeoAPI:

pygeoapi = PyGeoAPI()

trace = pygeoapi.flow_trace((1774209.63, 856381.68), crs="ESRI:102003", direction="none")

split = pygeoapi.split_catchment((-73.82705, 43.29139), crs=4326, upstream=False)

profile = pygeoapi.elevation_profile(
    [(-103.801086, 40.26772), (-103.80097, 40.270568)],
    numpts=101,
    dem_res=1,
    crs=4326,
)

section = pygeoapi.cross_section((-103.80119, 40.2684), width=1000.0, numpts=101, crs=4326)

Now, let’s do the same operations using pygeoapi:

import geopandas as gpd
import shapely.geometry as sgeom
import pynhd as nhd

coords = gpd.GeoDataFrame(
    {
        "direction": ["up", "down"],
        "upstream": [True, False],
        "width": [1000.0, 500.0],
        "numpts": [101, 55],
    },
    geometry=[
        sgeom.Point(-73.82705, 43.29139),
        sgeom.Point(-103.801086, 40.26772),
    ],
    crs=4326,
)
trace = nhd.pygeoapi(coords, "flow_trace")
split = nhd.pygeoapi(coords, "split_catchment")
section = nhd.pygeoapi(coords, "cross_section")

coords = gpd.GeoDataFrame(
    {
        "direction": ["up", "down"],
        "upstream": [True, False],
        "width": [1000.0, 500.0],
        "numpts": [101, 55],
        "dem_res": [1, 10],
    },
    geometry=[
        sgeom.MultiPoint([(-103.801086, 40.26772), (-103.80097, 40.270568)]),
        sgeom.MultiPoint([(-102.801086, 39.26772), (-102.80097, 39.270568)]),
    ],
    crs=4326,
)
profile = nhd.pygeoapi(coords, "elevation_profile")

Next, we retrieve mid- and high-resolution flowlines within the bounding box of our watershed and compare them using WaterData for mid-resolution, NHDPlusHR for high-resolution.

mr = WaterData("nhdflowline_network")
nhdp_mr = mr.bybox(basin.geometry[0].bounds)

hr = NHDPlusHR("flowline")
nhdp_hr = hr.bygeom(basin.geometry[0].bounds)

An alternative to WaterData and NHDPlusHR is the NHD class that supports both the mid- and high-resolution NHDPlus V2 data:

mr = NHD("flowline_mr")
nhdp_mr = mr.bygeom(basin.geometry[0].bounds)

hr = NHD("flowline_hr")
nhdp_hr = hr.bygeom(basin.geometry[0].bounds)

Moreover, WaterData can find features within a given radius (in meters) of a point:

eck4 = "+proj=eck4 +lon_0=0 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"
coords = (-5727797.427596455, 5584066.49330473)
rad = 5e3
flw_rad = mr.bydistance(coords, rad, loc_crs=eck4)
flw_rad = flw_rad.to_crs(eck4)

Instead of getting all features within a radius of the coordinate, we can snap to the closest feature ID using NLDI:

comid_closest = nldi.comid_byloc((x, y), eck4)
flw_closest = nhdp_mr.byid("comid", comid_closest.comid.values[0])

Since NHDPlus HR is still at the pre-release stage let’s use the MR flowlines to demonstrate the vector-based accumulation. Based on a topological sorted river network pynhd.vector_accumulation computes flow accumulation in the network. It returns a data frame that is sorted from upstream to downstream that shows the accumulated flow in each node.

PyNHD has a utility called prepare_nhdplus that identifies such relationships among other things such as fixing some common issues with NHDPlus flowlines. But first, we need to get all the NHDPlus attributes for each ComID since NLDI only provides the flowlines’ geometries and ComIDs which is useful for navigating the vector river network data. For getting the NHDPlus database we use WaterData. Let’s use the nhdflowline_network layer to get required info.

wd = WaterData("nhdflowline_network")

comids = flw_trib.nhdplus_comid.to_list()
nhdp_trib = wd.byid("comid", comids)
flw = nhd.prepare_nhdplus(nhdp_trib, 0, 0, purge_non_dendritic=False)

To demonstrate the use of routing, let’s use nhdplus_attrs function to get a list of available NHDPlus attributes

char = "CAT_RECHG"
area = "areasqkm"

local = nldi.getcharacteristic_byid(comids, "local", char_ids=char)
flw = flw.merge(local[char], left_on="comid", right_index=True)


def runoff_acc(qin, q, a):
    return qin + q * a


flw_r = flw[["comid", "tocomid", char, area]]
runoff = nhd.vector_accumulation(flw_r, runoff_acc, char, [char, area])


def area_acc(ain, a):
    return ain + a


flw_a = flw[["comid", "tocomid", area]]
areasqkm = nhd.vector_accumulation(flw_a, area_acc, area, [area])

runoff /= areasqkm

Since these are catchment-scale characteristics, let’s get the catchments then add the accumulated characteristic as a new column and plot the results.

wd = WaterData("catchmentsp")
catchments = wd.byid("featureid", comids)

c_local = catchments.merge(local, left_on="featureid", right_index=True)
c_acc = catchments.merge(runoff, left_on="featureid", right_index=True)

More examples can be found here.

PyGeoHydro: Retrieve Geospatial Hydrology Data

Features

PyGeoHydro (formerly named hydrodata) is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services. This package provides access to some public web services that offer geospatial hydrology data. It has three main modules: pygeohydro, plot, and helpers.

PyGeoHydro supports the following datasets:

• gNATSGO for US soil properties.

• Derived Soil Properties for soil porosity, available water capacity, and field capacity across the US.

• NWIS for daily mean streamflow observations (returned as a pandas.DataFrame or xarray.Dataset with station attributes),

• SensorThings API for accessing real-time data of USGS sensors.

• CAMELS for accessing streamflow observations (1980-2014) and basin-level attributes of 671 stations within CONUS.

• Water Quality Portal for accessing current and historical water quality data from more than 1.5 million sites across the US,

• NID for accessing the National Inventory of Dams web service,

• HCDN 2009 for identifying sites where human activity affects the natural flow of the watercourse,

• NLCD 2021 for land cover/land use, imperviousness descriptor, and canopy data. You can get data using both geometries and coordinates.

• WBD for accessing Hydrologic Unit (HU) polygon boundaries within the US (all HUC levels).

• SSEBop for daily actual evapotranspiration, for both single pixel and gridded data.

• Irrigation Withdrawals for estimated monthly water use for irrigation by 12-digit hydrologic unit in the CONUS for 2015

• STN for access USGS Short-Term Network (STN)

• eHydro for accessing USACE Hydrographic Surveys that includes topobathymetry data

• NFHL for accessing FEMA’s National Flood Hazard Layer (NFHL) data.

Also, it includes several other functions:

• interactive_map: Interactive map for exploring NWIS stations within a bounding box.

• cover_statistics: Categorical statistics of land use/land cover data.

• overland_roughness: Estimate overland roughness from land use/land cover data.

• streamflow_fillna: Fill missing daily streamflow values with day-of-year averages. Streamflow observations must be at least for 10-year long.

The plot module includes two main functions:

• signatures: Hydrologic signature graphs.

• cover_legends: Official NLCD land cover legends for plotting a land cover dataset.

• descriptor_legends: Color map and legends for plotting an imperviousness descriptor dataset.

The helpers module includes:

• nlcd_helper: A roughness coefficients lookup table for each land cover and imperviousness descriptor type which is useful for overland flow routing among other applications.

• nwis_error: A dataframe for finding information about NWIS requests’ errors.

You can find some example notebooks here.

Moreover, under the hood, PyGeoHydro uses PyGeoOGC and AsyncRetriever packages for making requests in parallel and storing responses in chunks. This improves the reliability and speed of data retrieval significantly.

You can control the request/response caching behavior and verbosity of the package by setting the following environment variables:

• HYRIVER_CACHE_NAME: Path to the caching SQLite database for asynchronous HTTP requests. It defaults to ./cache/aiohttp_cache.sqlite

• HYRIVER_CACHE_NAME_HTTP: Path to the caching SQLite database for HTTP requests. It defaults to ./cache/http_cache.sqlite

• HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to one week.

• HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.

• HYRIVER_SSL_CERT: Path to a SSL certificate file.

For example, in your code before making any requests you can do:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/aiohttp_cache.sqlite"
os.environ["HYRIVER_CACHE_NAME_HTTP"] = "path/to/http_cache.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"
os.environ["HYRIVER_SSL_CERT"] = "path/to/cert.pem"

You can also try using PyGeoHydro without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install PyGeoHydro using pip after installing libgdal on your system (for example, in Ubuntu run sudo apt install libgdal-dev). Moreover, PyGeoHydro has an optional dependency for using persistent caching, requests-cache. We highly recommend installing this package as it can significantly speed up send/receive queries. You don’t have to change anything in your code, since PyGeoHydro under-the-hood looks for requests-cache and if available, it will automatically use persistent caching:

$ pip install pygeohydro

Alternatively, PyGeoHydro can be installed from the conda-forge repository using Conda:

$ conda install -c conda-forge pygeohydro

Quick start

We can obtain river topobathymetry data using the EHydro class. We can subset the dataset either using a geometry or a bounding box, based on their ID, or SQL query:

from pygeohydro import EHydro

ehydro = EHydro("points")
topobathy = ehydro.bygeom((-122.53, 45.57, -122.52, 45.59))

We can explore the available NWIS stations within a bounding box using interactive_map function. It returns an interactive map and by clicking on a station some of the most important properties of stations are shown.

import pygeohydro as gh

bbox = (-69.5, 45, -69, 45.5)
gh.interactive_map(bbox)

We can select all the stations within this boundary box that have daily mean streamflow data from 2000-01-01 to 2010-12-31:

from pygeohydro import NWIS

nwis = NWIS()
query = {
    "bBox": ",".join(f"{b:.06f}" for b in bbox),
    "hasDataTypeCd": "dv",
    "outputDataTypeCd": "dv",
}
info_box = nwis.get_info(query)
dates = ("2000-01-01", "2010-12-31")
stations = info_box[
    (info_box.begin_date <= dates[0]) & (info_box.end_date >= dates[1])
].site_no.tolist()

Then, we can get the daily streamflow data in mm/day (by default the values are in cms) and plot them:

from pygeohydro import plot

qobs = nwis.get_streamflow(stations, dates, mmd=True)
plot.signatures(qobs)

By default, get_streamflow returns a pandas.DataFrame that has a attrs method containing metadata for all the stations. You can access it like so qobs.attrs. Moreover, we can get the same data as xarray.Dataset as follows:

qobs_ds = nwis.get_streamflow(stations, dates, to_xarray=True)

This xarray.Dataset has two dimensions: time and station_id. It has 10 variables including discharge with two dimensions while other variables that are station attitudes are one dimensional.

We can also get instantaneous streamflow data using get_streamflow. This method assumes that the input dates are in UTC time zone and returns the data in UTC time zone as well.

date = ("2005-01-01 12:00", "2005-01-12 15:00")
qobs = nwis.get_streamflow("01646500", date, freq="iv")

We can query USGS stations of type “stream” in Arizona using SensorThings API as follows:

odata = {
    "filter": "properties/monitoringLocationType eq 'Stream' and properties/stateFIPS eq 'US:04'",
}
df = sensor.query_byodata(odata)

Irrigation withdrawals data can be obtained as follows:

irr = gh.irrigation_withdrawals()

We can get the CAMELS dataset as a geopandas.GeoDataFrame that includes geometry and basin-level attributes of 671 natural watersheds within CONUS and their streamflow observations between 1980-2014 as a xarray.Dataset, like so:

attrs, qobs = gh.get_camels()

The WaterQuality has a number of convenience methods to retrieve data from the web service. Since there are many parameter combinations that can be used to retrieve data, a general method is also provided to retrieve data from any of the valid endpoints. You can use get_json to retrieve stations info as a geopandas.GeoDataFrame or get_csv to retrieve stations data as a pandas.DataFrame. You can construct a dictionary of the parameters and pass it to one of these functions. For more information on the parameters, please consult the Water Quality Data documentation. For example, let’s find all the stations within a bounding box that have Caffeine data:

from pynhd import WaterQuality

bbox = (-92.8, 44.2, -88.9, 46.0)
kwds = {"characteristicName": "Caffeine"}
wq = WaterQuality()
stations = wq.station_bybbox(bbox, kwds)

Or the same criterion but within a 30-mile radius of a point:

stations = wq.station_bydistance(-92.8, 44.2, 30, kwds)

Then we can get the data for all these stations the data like this:

sids = stations.MonitoringLocationIdentifier.tolist()
caff = wq.data_bystation(sids, kwds)

Moreover, we can get land use/land cove data using nlcd_bygeom or nlcd_bycoods functions, percentages of land cover types using cover_statistics, and overland roughness using overland_roughness. The nlcd_bycoords function returns a geopandas.GeoDataFrame with the NLCD layers as columns and input coordinates as the geometry column. Moreover, the nlcd_bygeom function accepts both a single geometry or a geopandas.GeoDataFrame as the input.

from pynhd import NLDI

basins = NLDI().get_basins(["01031450", "01318500", "01031510"])
lulc = gh.nlcd_bygeom(basins, 100, years={"cover": [2016, 2019]})
stats = gh.cover_statistics(lulc["01318500"].cover_2016)
roughness = gh.overland_roughness(lulc["01318500"].cover_2019)

Next, let’s use ssebopeta_bygeom to get actual ET data for a basin. Note that there’s a ssebopeta_bycoords function that returns an ETA time series for a single coordinate.

geometry = NLDI().get_basins("01315500").geometry[0]
eta = gh.ssebopeta_bygeom(geometry, dates=("2005-10-01", "2005-10-05"))

Additionally, we can pull all the US dams data using NID. Let’s get dams that are within this bounding box and have a maximum storage larger than 200 acre-feet.

nid = NID()
dams = nid.get_bygeom((-65.77, 43.07, -69.31, 45.45), 4326)
dams = nid.inventory_byid(dams.id.to_list())
dams = dams[dams.maxStorage > 200]

We can get also all dams within CONUS with maximum storage larger than 2500 acre-feet:

conus_geom = gh.get_us_states("contiguous")

dam_list = nid.get_byfilter([{"maxStorage": ["[2500 +inf]"]}])
dams = nid.inventory_byid(dam_list[0].id.to_list(), stage_nid=True)

conus_dams = dams[dams.stateKey.isin(conus_geom.STUSPS)].reset_index(drop=True)

The WBD class allows us to get Hydrologic Unit (HU) polygon boundaries. Let’s get the two Hudson HUC4s:

from pygeohydro import WBD

wbd = WBD("huc4")
hudson = wbd.byids("huc4", ["0202", "0203"])

The NFHL class allows us to retrieve FEMA’s National Flood Hazard Layer (NFHL) data. Let’s get the cross-section data for a small region in Vermont:

from pygeohydro import NFHL

nfhl = NFHL("NFHL", "cross-sections")
gdf_xs = nfhl.bygeom((-73.42, 43.28, -72.9, 43.52), geo_crs=4269)

Py3DEP: Topographic data through 3DEP

Features

Py3DEP is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services. This package provides access to the 3DEP database which is a part of the National Map services. The 3DEP service has multi-resolution sources and depending on the user-provided resolution, the data is resampled on the server-side based on all the available data sources. Py3DEP returns the requests as xarray dataset.

The following functionalities are currently available:

• get_map: Get topographic data the dynamic 3DEP service that supports the following layers:

  • DEM

  • Hillshade Gray

  • Aspect Degrees

  • Aspect Map

  • GreyHillshade Elevation Fill

  • Hillshade Multidirectional

  • Slope Degrees

  • Slope Map

  • Hillshade Elevation Tinted

  • Height Ellipsoidal

  • Contour 25

  • Contour Smoothed 25

• static_3dep_dem: Get DEM data at 10 m, 30 m, or 60 m resolution from the staged 3DEP data. Since this function only returns DEM, for computing other terrain attributes you can use xarray-spatial. Just note that you should reproject the output DataArray to a projected CRS like 5070 before passing it to xarray-spatial like so: dem = dem.rio.reproject(5070).

• get_dem: Get DEM data from either the dynamic or static 3DEP service. Considering that the static service is much faster, if the target DEM resolution is 10 m, 30 m, or 60 m, then the static service is used (static_3dep_dem). Otherwise, the dynamic service is used (get_map using DEM layer).

• get_map_vrt: Get DEM data and store it as a GDAL VRT file from the dynamic 3DEP service. This function is mainly provided for large requests due to its low memory footprint. Moreover, due to lazy loading of the data this function can be much faster than get_map or get_dem, even for small requests at the cost of higher disk usage.

• elevation_bygrid: For retrieving elevations of all the grid points in a 2D grid.

• add_elevation: For adding elevation data as a new variable to an input xarray.DataArray or xarray.Dataset.

• elevation_bycoords: For retrieving elevation of a list of x and y coordinates.

• elevation_profile: For retrieving elevation profile along a line at a given spacing. This function converts the line to a B-spline and then calculates the elevation along the spline at a given uniform spacing.

• deg2mpm: For converting slope dataset from degree to meter per meter.

• query_3dep_sources: For querying bounds of 3DEP’s data sources within a bounding box.

• check_3dep_availability: For querying 3DEP’s resolution availability within a bounding box.

You can find some example notebooks here.

Moreover, under the hood, Py3DEP uses PyGeoOGC and AsyncRetriever packages for making requests in parallel and storing responses in chunks. This improves the reliability and speed of data retrieval significantly.

You can control the request/response caching behavior and verbosity of the package by setting the following environment variables:

• HYRIVER_CACHE_NAME: Path to the caching SQLite database for asynchronous HTTP requests. It defaults to ./cache/aiohttp_cache.sqlite

• HYRIVER_CACHE_NAME_HTTP: Path to the caching SQLite database for HTTP requests. It defaults to ./cache/http_cache.sqlite

• HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to one week.

• HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.

• HYRIVER_SSL_CERT: Path to a SSL certificate file.

For example, in your code before making any requests you can do:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/aiohttp_cache.sqlite"
os.environ["HYRIVER_CACHE_NAME_HTTP"] = "path/to/http_cache.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"
os.environ["HYRIVER_SSL_CERT"] = "path/to/cert.pem"

You can also try using Py3DEP without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install Py3DEP using pip after installing libgdal on your system (for example, in Ubuntu run sudo apt install libgdal-dev). Moreover, Py3DEP has an optional dependency for using persistent caching, requests-cache. We highly recommend installing this package as it can significantly speed up send/receive queries. You don’t have to change anything in your code, since Py3DEP under-the-hood looks for requests-cache and if available, it will automatically use persistent caching:

$ pip install py3dep

Alternatively, Py3DEP can be installed from the conda-forge repository using Conda:

$ conda install -c conda-forge py3dep

Quick start

You can use Py3DEP using command-line or as a Python library. The command-line interface provides access to two functionality:

• Getting topographic data: You must create a geopandas.GeoDataFrame that contains the geometries of the target locations. This dataframe must have at least three columns: id, res, and geometry. The id column is used as filenames for saving the obtained topographic data to a NetCDF (.nc) file. The res column must be the target resolution in meter. Then, you must save the dataframe to a file with extensions such as .shp or .gpkg (whatever that geopandas.read_file can read).

• Getting elevation: You must create a pandas.DataFrame that contains coordinates of the target locations. This dataframe must have at least two columns: x and y. The elevations are obtained using airmap service in meters. The data are saved as a csv file with the same filename as the input file with an _elevation appended, e.g., coords_elevation.csv.

$ py3dep --help
Usage: py3dep [OPTIONS] COMMAND [ARGS]...

Command-line interface for Py3DEP.

Options:
-h, --help  Show this message and exit.

Commands:
coords    Retrieve topographic data for a list of coordinates.
geometry  Retrieve topographic data within geometries.

The coords sub-command is as follows:

$ py3dep coords -h
Usage: py3dep coords [OPTIONS] FPATH

Retrieve topographic data for a list of coordinates.

FPATH: Path to a csv file with two columns named ``lon`` and ``lat``.

Examples:
    $ cat coords.csv
    lon,lat
    -122.2493328,37.8122894
    $ py3dep coords coords.csv -q airmap -s topo_dir

Options:
-q, --query_source [airmap|tnm|tep]
                                Source of the elevation data.
-s, --save_dir PATH             Path to a directory to save the requested
                                files. Extension for the outputs is either
                                `.nc` for geometry or `.csv` for coords.

-h, --help                      Show this message and exit.

And, the geometry sub-command is as follows:

$ py3dep geometry -h
Usage: py3dep geometry [OPTIONS] FPATH

Retrieve topographic data within geometries.

FPATH: Path to a shapefile (.shp) or geopackage (.gpkg) file.
This file must have three columns and contain a ``crs`` attribute:
    - ``id``: Feature identifiers that py3dep uses as the output netcdf/csv filenames.
    - ``res``: Target resolution in meters.
    - ``geometry``: A Polygon or MultiPloygon.

Examples:
    $ py3dep geometry ny_geom.gpkg -l "Slope Map" -l DEM -s topo_dir

Options:
-l, --layers [DEM|Hillshade Gray|Aspect Degrees|Aspect Map|GreyHillshade_elevationFill|Hillshade Multidirectional|Slope Map|Slope Degrees|Hillshade Elevation Tinted|Height Ellipsoidal|Contour 25|Contour Smoothed 25]
                                Target topographic data layers
-s, --save_dir PATH             Path to a directory to save the requested
                                files.Extension for the outputs is either
                                `.nc` for geometry or `.csv` for coords.

-h, --help                      Show this message and exit.

Now, let’s see how we can use Py3DEP as a library.

Py3DEP accepts Shapely’s Polygon or a bounding box (a tuple of length four) as an input geometry. We can use PyNHD to get a watershed’s geometry, then use it to get the DEM and slope in meters/meters from Py3DEP using get_map function.

The get_map has a resolution argument that sets the target resolution in meters. Note that the highest available resolution throughout the CONUS is about 10 m, though higher resolutions are available in limited parts of the US. Note that the input geometry can be in any valid spatial reference (geo_crs argument). The crs argument, however, is limited to CRS:84, EPSG:4326, and EPSG:3857 since 3DEP only supports these spatial references.

import py3dep
from pynhd import NLDI

geom = NLDI().get_basins("01031500").geometry[0]
dem = py3dep.get_map("DEM", geom, resolution=30, geo_crs=4326, crs=3857)
slope = py3dep.get_map("Slope Degrees", geom, resolution=30)
slope = py3dep.deg2mpm(slope)

We can also use static_dem function to get the same DEM:

import xrspatial

dem = py3dep.get_dem(geom, 30)
slope = xrspatial.slope(dem.rio.reproject(5070))
slope = py3dep.deg2mpm(slope)

We can use rioxarray package to save the obtained dataset as a raster file:

import rioxarray

dem.rio.to_raster("dem_01031500.tif")

Moreover, we can get the elevations of a set of x- and y- coordinates on a grid. For example, let’s get the minimum temperature data within this watershed from Daymet using PyDaymet then add the elevation as a new variable to the dataset:

import pydaymet as daymet
import xarray as xr
import numpy as np

clm = daymet.get_bygeom(geometry, ("2005-01-01", "2005-01-31"), variables="tmin")
elev = py3dep.elevation_bygrid(clm.x.values, clm.y.values, clm.crs, clm.res[0] * 1000)
attrs = clm.attrs
clm = xr.merge([clm, elev])
clm["elevation"] = clm.elevation.where(~np.isnan(clm.isel(time=0).tmin), drop=True)
clm.attrs.update(attrs)

Now, let’s get street network data using osmnx package and add elevation data for its nodes using elevation_bycoords function.

import osmnx as ox

G = ox.graph_from_place("Piedmont, California, USA", network_type="drive")
x, y = nx.get_node_attributes(G, "x").values(), nx.get_node_attributes(G, "y").values()
elevation = py3dep.elevation_bycoords(zip(x, y), crs=4326)
nx.set_node_attributes(G, dict(zip(G.nodes(), elevation)), "elevation")

We can get the elevation profile along a line at a given spacing using elevation_profile function. For example, let’s get the elevation profile at 10-m spacing along the main flowline of the upstream drainage area of a USGS station with ID 01031500:

import py3dep
from pynhd import NLDI

flw_main = NLDI().navigate_byid(
    fsource="nwissite",
    fid="USGS-01031500",
    navigation="upstreamMain",
    source="flowlines",
    distance=1000,
)
line = flw_main.geometry.unary_union
elevation = py3dep.elevation_profile(line, 10)

PyDaymet: Daily climate data through Daymet

Warning

Since the release of Daymet v4 R1 on November 2022, the URL of Daymet’s server has been changed. Therefore, only PyDaymet v0.13.7+ is going to work, and previous versions will not work anymore.

Features

PyDaymet is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services. This package provides access to climate data from Daymet V4 R1 database using NetCDF Subset Service (NCSS). Both single pixel (using get_bycoords function) and gridded data (using get_bygeom) are supported which are returned as pandas.DataFrame and xarray.Dataset, respectively. Climate data is available for North America, Hawaii from 1980, and Puerto Rico from 1950 at three time scales: daily, monthly, and annual. Additionally, PyDaymet can compute Potential EvapoTranspiration (PET) using three methods: penman_monteith, priestley_taylor, and hargreaves_samani for both single pixel and gridded data.

For PET computations, PyDaymet accepts four additional user-defined parameters:

• penman_monteith: soil_heat_flux, albedo, alpha,

  and arid_correction.

• priestley_taylor: soil_heat_flux, albedo, and arid_correction.

• hargreaves_samani: None.

Default values for the parameters are: soil_heat_flux = 0, albedo = 0.23, alpha = 1.26, and arid_correction = False. An important parameter for priestley_taylor and penman_monteith methods is arid_correction which is used to correct the actual vapor pressure for arid regions. Since relative humidity is not provided by Daymet, the actual vapor pressure is computed assuming that the dew point temperature is equal to the minimum temperature. However, for arid regions, FAO 56 suggests subtracting minimum temperature by 2-3 °C to account for the fact that in arid regions, the air might not be saturated when its temperature is at its minimum. For such areas, you can pass {"arid_correction": True, ...} to subtract 2 °C from the minimum temperature for computing the actual vapor pressure.

You can find some example notebooks here.

Moreover, under the hood, PyDaymet uses PyGeoOGC and AsyncRetriever packages for making requests in parallel and storing responses in chunks. This improves the reliability and speed of data retrieval significantly.

You can control the request/response caching behavior and verbosity of the package by setting the following environment variables:

• HYRIVER_CACHE_NAME: Path to the caching SQLite database for asynchronous HTTP requests. It defaults to ./cache/aiohttp_cache.sqlite

• HYRIVER_CACHE_NAME_HTTP: Path to the caching SQLite database for HTTP requests. It defaults to ./cache/http_cache.sqlite

• HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to one week.

• HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.

• HYRIVER_SSL_CERT: Path to a SSL certificate file.

For example, in your code before making any requests you can do:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/aiohttp_cache.sqlite"
os.environ["HYRIVER_CACHE_NAME_HTTP"] = "path/to/http_cache.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"
os.environ["HYRIVER_SSL_CERT"] = "path/to/cert.pem"

You can also try using PyDaymet without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install PyDaymet using pip after installing libgdal on your system (for example, in Ubuntu run sudo apt install libgdal-dev):

$ pip install pydaymet

Alternatively, PyDaymet can be installed from the conda-forge repository using Conda:

$ conda install -c conda-forge pydaymet

Quick start

You can use PyDaymet using command-line or as a Python library. The commanda-line provides access to two functionality:

• Getting gridded climate data: You must create a geopandas.GeoDataFrame that contains the geometries of the target locations. This dataframe must have four columns: id, start, end, geometry. The id column is used as filenames for saving the obtained climate data to a NetCDF (.nc) file. The start and end columns are starting and ending dates of the target period. Then, you must save the dataframe as a shapefile (.shp) or geopackage (.gpkg) with CRS attribute.

• Getting single pixel climate data: You must create a CSV file that contains coordinates of the target locations. This file must have at four columns: id, start, end, lon, and lat. The id column is used as filenames for saving the obtained climate data to a CSV (.csv) file. The start and end columns are the same as the geometry command. The lon and lat columns are the longitude and latitude coordinates of the target locations.

$ pydaymet -h
Usage: pydaymet [OPTIONS] COMMAND [ARGS]...

Command-line interface for PyDaymet.

Options:
-h, --help  Show this message and exit.

Commands:
coords    Retrieve climate data for a list of coordinates.
geometry  Retrieve climate data for a dataframe of geometries.

The coords sub-command is as follows:

$ pydaymet coords -h
Usage: pydaymet coords [OPTIONS] FPATH

Retrieve climate data for a list of coordinates.

FPATH: Path to a csv file with four columns:
    - ``id``: Feature identifiers that daymet uses as the output netcdf filenames.
    - ``start``: Start time.
    - ``end``: End time.
    - ``lon``: Longitude of the points of interest.
    - ``lat``: Latitude of the points of interest.
    - ``time_scale``: (optional) Time scale, either ``daily`` (default), ``monthly`` or ``annual``.
    - ``pet``: (optional) Method to compute PET. Supported methods are:
                ``penman_monteith``, ``hargreaves_samani``, ``priestley_taylor``, and ``none`` (default).
    - ``snow``: (optional) Separate snowfall from precipitation, default is ``False``.

Examples:
    $ cat coords.csv
    id,lon,lat,start,end,pet
    california,-122.2493328,37.8122894,2012-01-01,2014-12-31,hargreaves_samani
    $ pydaymet coords coords.csv -v prcp -v tmin

Options:
-v, --variables TEXT  Target variables. You can pass this flag multiple
                        times for multiple variables.
-s, --save_dir PATH   Path to a directory to save the requested files.
                        Extension for the outputs is .nc for geometry and .csv
                        for coords.
--disable_ssl         Pass to disable SSL certification verification.
-h, --help            Show this message and exit.

And, the geometry sub-command is as follows:

$ pydaymet geometry -h
Usage: pydaymet geometry [OPTIONS] FPATH

Retrieve climate data for a dataframe of geometries.

FPATH: Path to a shapefile (.shp) or geopackage (.gpkg) file.
This file must have four columns and contain a ``crs`` attribute:
    - ``id``: Feature identifiers that daymet uses as the output netcdf filenames.
    - ``start``: Start time.
    - ``end``: End time.
    - ``geometry``: Target geometries.
    - ``time_scale``: (optional) Time scale, either ``daily`` (default), ``monthly`` or ``annual``.
    - ``pet``: (optional) Method to compute PET. Supported methods are:
                ``penman_monteith``, ``hargreaves_samani``, ``priestley_taylor``, and ``none`` (default).
    - ``snow``: (optional) Separate snowfall from precipitation, default is ``False``.

Examples:
    $ pydaymet geometry geo.gpkg -v prcp -v tmin

Options:
-v, --variables TEXT  Target variables. You can pass this flag multiple
                        times for multiple variables.
-s, --save_dir PATH   Path to a directory to save the requested files.
                        Extension for the outputs is .nc for geometry and .csv
                        for coords.
--disable_ssl         Pass to disable SSL certification verification.
-h, --help            Show this message and exit.

Now, let’s see how we can use PyDaymet as a library.

PyDaymet offers two functions for getting climate data; get_bycoords and get_bygeom. The arguments of these functions are identical except the first argument where the latter should be polygon and the former should be a coordinate (a tuple of length two as in (x, y)). The input geometry or coordinate can be in any valid CRS (defaults to EPSG:4326). The dates argument can be either a tuple of length two like (start_str, end_str) or a list of years like [2000, 2005]. It is noted that both functions have a pet flag for computing PET and a snow flag for separating snow from precipitation using Martinez and Gupta (2010) method. Additionally, we can pass time_scale to get daily, monthly or annual summaries. This flag by default is set to daily.

from pynhd import NLDI
import pydaymet as daymet

geometry = NLDI().get_basins("01031500").geometry[0]

var = ["prcp", "tmin"]
dates = ("2000-01-01", "2000-06-30")

daily = daymet.get_bygeom(geometry, dates, variables=var, pet="priestley_taylor", snow=True)
monthly = daymet.get_bygeom(geometry, dates, variables=var, time_scale="monthly")

If the input geometry (or coordinate) is in a CRS other than EPSG:4326, we should pass it to the functions.

coords = (-1431147.7928, 318483.4618)
crs = 3542
dates = ("2000-01-01", "2006-12-31")
annual = daymet.get_bycoords(coords, dates, variables=var, loc_crs=crs, time_scale="annual")

Additionally, the get_bycoords function accepts a list of coordinates and by setting the to_xarray flag to True it can return the results as a xarray.Dataset instead of a pandas.DataFrame:

coords = [(-94.986, 29.973), (-95.478, 30.134)]
idx = ["P1", "P2"]
clm_ds = daymet.get_bycoords(coords, range(2000, 2021), coords_id=idx, to_xarray=True)

Also, we can use the potential_et function to compute PET by passing the daily climate data. We can either pass a pandas.DataFrame or a xarray.Dataset. Note that, penman_monteith and priestley_taylor methods have parameters that can be passed via the params argument, if any value other than the default values are needed. For example, default value of alpha for priestley_taylor method is 1.26 (humid regions), we can set it to 1.74 (arid regions) as follows:

pet_hs = daymet.potential_et(daily, methods="priestley_taylor", params={"alpha": 1.74})

Next, let’s get annual total precipitation for Hawaii and Puerto Rico for 2010.

hi_ext = (-160.3055, 17.9539, -154.7715, 23.5186)
pr_ext = (-67.9927, 16.8443, -64.1195, 19.9381)
hi = daymet.get_bygeom(hi_ext, 2010, variables="prcp", region="hi", time_scale="annual")
pr = daymet.get_bygeom(pr_ext, 2010, variables="prcp", region="pr", time_scale="annual")

Some example plots are shown below:

PyGridMET: Daily climate data through GridMET

Features

PyGridMET is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services. This package provides access to daily climate data over contermonious US (CONUS) from GridMET database using NetCDF Subset Service (NCSS). Both single pixel (using get_bycoords function) and gridded data (using get_bygeom) are supported which are returned as pandas.DataFrame and xarray.Dataset, respectively.

You can find some example notebooks here.

Moreover, under the hood, PyGridMET uses PyGeoOGC and AsyncRetriever packages for making requests in parallel and storing responses in chunks. This improves the reliability and speed of data retrieval significantly.

You can control the request/response caching behavior and verbosity of the package by setting the following environment variables:

• HYRIVER_CACHE_NAME: Path to the caching SQLite database for asynchronous HTTP requests. It defaults to ./cache/aiohttp_cache.sqlite

• HYRIVER_CACHE_NAME_HTTP: Path to the caching SQLite database for HTTP requests. It defaults to ./cache/http_cache.sqlite

• HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to one week.

• HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.

• HYRIVER_SSL_CERT: Path to a SSL certificate file.

For example, in your code before making any requests you can do:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/aiohttp_cache.sqlite"
os.environ["HYRIVER_CACHE_NAME_HTTP"] = "path/to/http_cache.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"
os.environ["HYRIVER_SSL_CERT"] = "path/to/cert.pem"

You can also try using PyGridMET without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install PyGridMET using pip as follows:

$ pip install pygridmet

Alternatively, PyGridMET can be installed from the conda-forge repository using Conda:

$ conda install -c conda-forge pygridmet

Quick start

You can use PyGridMET using command-line or as a Python library. The commanda-line provides access to two functionality:

• Getting gridded climate data: You must create a geopandas.GeoDataFrame that contains the geometries of the target locations. This dataframe must have four columns: id, start, end, geometry. The id column is used as filenames for saving the obtained climate data to a NetCDF (.nc) file. The start and end columns are starting and ending dates of the target period. Then, you must save the dataframe as a shapefile (.shp) or geopackage (.gpkg) with CRS attribute.

• Getting single pixel climate data: You must create a CSV file that contains coordinates of the target locations. This file must have at four columns: id, start, end, lon, and lat. The id column is used as filenames for saving the obtained climate data to a CSV (.csv) file. The start and end columns are the same as the geometry command. The lon and lat columns are the longitude and latitude coordinates of the target locations.

$ pygridmet -h
Usage: pygridmet [OPTIONS] COMMAND [ARGS]...

Command-line interface for PyGridMET.

Options:
-h, --help  Show this message and exit.

Commands:
coords    Retrieve climate data for a list of coordinates.
geometry  Retrieve climate data for a dataframe of geometries.

The coords sub-command is as follows:

$ pygridmet coords -h
Usage: pygridmet coords [OPTIONS] FPATH

Retrieve climate data for a list of coordinates.

FPATH: Path to a csv file with four columns:
    - ``id``: Feature identifiers that gridmet uses as the output netcdf filenames.
    - ``start``: Start time.
    - ``end``: End time.
    - ``lon``: Longitude of the points of interest.
    - ``lat``: Latitude of the points of interest.
    - ``snow``: (optional) Separate snowfall from precipitation, default is ``False``.

Examples:
    $ cat coords.csv
    id,lon,lat,start,end
    california,-122.2493328,37.8122894,2012-01-01,2014-12-31
    $ pygridmet coords coords.csv -v pr -v tmmn

Options:
-v, --variables TEXT  Target variables. You can pass this flag multiple
                        times for multiple variables.
-s, --save_dir PATH   Path to a directory to save the requested files.
                        Extension for the outputs is .nc for geometry and .csv
                        for coords.
--disable_ssl         Pass to disable SSL certification verification.
-h, --help            Show this message and exit.

And, the geometry sub-command is as follows:

$ pygridmet geometry -h
Usage: pygridmet geometry [OPTIONS] FPATH

Retrieve climate data for a dataframe of geometries.

FPATH: Path to a shapefile (.shp) or geopackage (.gpkg) file.
This file must have four columns and contain a ``crs`` attribute:
    - ``id``: Feature identifiers that gridmet uses as the output netcdf filenames.
    - ``start``: Start time.
    - ``end``: End time.
    - ``geometry``: Target geometries.
    - ``snow``: (optional) Separate snowfall from precipitation, default is ``False``.

Examples:
    $ pygridmet geometry geo.gpkg -v pr -v tmmn

Options:
-v, --variables TEXT  Target variables. You can pass this flag multiple
                        times for multiple variables.
-s, --save_dir PATH   Path to a directory to save the requested files.
                        Extension for the outputs is .nc for geometry and .csv
                        for coords.
--disable_ssl         Pass to disable SSL certification verification.
-h, --help            Show this message and exit.

Now, let’s see how we can use PyGridMET as a library.

PyGridMET offers two functions for getting climate data; get_bycoords and get_bygeom. The arguments of these functions are identical except the first argument where the latter should be polygon and the former should be a coordinate (a tuple of length two as in (x, y)). The input geometry or coordinate can be in any valid CRS (defaults to EPSG:4326). The dates argument can be either a tuple of length two like (start_str, end_str) or a list of years like [2000, 2005]. It is noted that both functions have a snow flag for separating snow from precipitation using Martinez and Gupta (2010) method.

We can get a dataframe of available variables and their info by calling GridMET().gridmet_table:

Variable	Abbr	Unit
Precipitation	pr	mm
Maximum Relative Humidity	rmax	%
Minimum Relative Humidity	rmin	%
Specific Humidity	sph	kg/kg
Surface Radiation	srad	W/m2
Wind Direction	th	Degrees Clockwise from north
Minimum Air Temperature	tmmn	K
Maximum Air Temperature	tmmx	K
Wind Speed	vs	m/s
Burning Index	bi	Dimensionless
Fuel Moisture (100-hr)	fm100	%
Fuel Moisture (1000-hr)	fm1000	%
Energy Release Component	erc	Dimensionless
Reference Evapotranspiration (Alfalfa)	etr	mm
Reference Evapotranspiration (Grass)	pet	mm
Vapor Pressure Deficit	vpd	kPa

from pynhd import NLDI
import pygridmet as gridmet

geometry = NLDI().get_basins("01031500").geometry[0]

var = ["pr", "tmmn"]
dates = ("2000-01-01", "2000-06-30")

daily = gridmet.get_bygeom(geometry, dates, variables=var, snow=True)

If the input geometry (or coordinate) is in a CRS other than EPSG:4326, we should pass it to the functions.

coords = (-1431147.7928, 318483.4618)
crs = 3542
dates = ("2000-01-01", "2006-12-31")
data = gridmet.get_bycoords(coords, dates, variables=var, loc_crs=crs)

Additionally, the get_bycoords function accepts a list of coordinates and by setting the to_xarray flag to True it can return the results as a xarray.Dataset instead of a pandas.DataFrame:

coords = [(-94.986, 29.973), (-95.478, 30.134)]
idx = ["P1", "P2"]
clm_ds = gridmet.get_bycoords(coords, range(2000, 2021), coords_id=idx, to_xarray=True)

PyNLDAS2: Hourly NLDAS-2 Forcing Data

Features

PyNLDAS2 is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services. This package provides access NLDAS-2 Forcing dataset via Hydrology Data Rods. Currently, only hourly data is supported. There are three main functions:

• get_bycoords: Forcing data for a list of coordinates as a pandas.DataFrame or xarray.Dataset,

• get_bygeom: Forcing data within a geometry as a xarray.Dataset,

• get_grid_mask: NLDAS2 land/water grid mask as a xarray.Dataset.

PyNLDAS2 only provides access to the hourly NLDAS2 dataset, so if you need to access other NASA climate datasets you can check out tsgettoolbox developed by Tim Cera.

Moreover, under the hood, PyNLDAS2 uses PyGeoOGC and AsyncRetriever packages for making requests in parallel and storing responses in chunks. This improves the reliability and speed of data retrieval significantly.

You can control the request/response caching behavior and verbosity of the package by setting the following environment variables:

• HYRIVER_CACHE_NAME: Path to the caching SQLite database for asynchronous HTTP requests. It defaults to ./cache/aiohttp_cache.sqlite

• HYRIVER_CACHE_NAME_HTTP: Path to the caching SQLite database for HTTP requests. It defaults to ./cache/http_cache.sqlite

• HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to one week.

• HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.

• HYRIVER_SSL_CERT: Path to a SSL certificate file.

For example, in your code before making any requests you can do:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/aiohttp_cache.sqlite"
os.environ["HYRIVER_CACHE_NAME_HTTP"] = "path/to/http_cache.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"
os.environ["HYRIVER_SSL_CERT"] = "path/to/cert.pem"

You can find some example notebooks here.

You can also try using PyNLDAS2 without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install pynldas2 using pip:

$ pip install pynldas2

Alternatively, pynldas2 can be installed from the conda-forge repository using Conda:

$ conda install -c conda-forge pynldas2

Quick start

The NLDAS2 database provides forcing data at 1/8th-degree grid spacing and range from 01 Jan 1979 to present. Let’s take a look at NLDAS2 grid mask that includes land, water, soil, and vegetation masks:

import pynldas2 as nldas

grid = nldas.get_grid_mask()

Next, we use PyGeoHydro to get the geometry of a HUC8 with ID of 1306003, then we get the forcing data within the obtained geometry.

from pygeohydro import WBD

huc8 = WBD("huc8")
geometry = huc8.byids("huc8", "13060003").geometry[0]
clm = nldas.get_bygeom(geometry, "2010-01-01", "2010-01-31", 4326)

Road Map

• [ ] Add PET calculation functions similar to PyDaymet but at hourly timescale.

• [ ] Add a command line interfaces.

HydroSignatures: Tools for computing hydrological signatures

Features

HydroSignatures is a suite of tools for computing hydrological signatures and a part of HyRiver software stack. This package includes the following functions:

• exceedance: Exceedance probability that can be used plotting flow duration curves;

• flow_duration_curve_slope: Slope of flow duration curve;

• flashiness_index: Flashiness index;

• mean_monthly: Mean monthly summary of a time series that can be used for plotting regime curves;

• rolling_mean_monthly: Rolling mean monthly summary of a time series that can be used for plotting smoothed regime curves;

• baseflow: Extracting baseflow from a streamflow time series using the Lyne and Hollick digital filter (Ladson et al., 2013);

• baseflow_index: Baseflow index;

• aridity_index: Aridity index;

• seasonality_index_walsh: Seasonality index (Walsh and Lawler, 1981);

• seasonality_index_markham: Seasonality index (Markham, 1970);

• extract_extrema: Determining the location of local maxima and minima in a time series;

Moreover, the package has a class called HydroSignatures that can be used to compute all these signatures by passing a streamflow and a precipitation time series, both in millimeters per day (or any other unit of time). This class supports subtraction and inequality operators, which can be used to compare two HydroSignatures objects. You can serialize the class to a JSON object using the to_json method or convert it to a dictionary using the to_dict method.

Moreover, numba is an optional dependency for the baseflow function. Installing numba will speed up the computation of baseflow significantly. For more efficient handling of NaN values, you can also install numbagg.

You can also try using HydroSignatures without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install HydroSignatures using pip:

$ pip install hydrosignatures

or from the conda-forge repository using Conda or Mamba:

$ conda install -c conda-forge hydrosignatures

Quick start

Let’s explore the capabilities of HydroSignatures by getting streamflow using PyGeoHydro, basin geometry using PyNHD and precipitation using PyDaymet. In this example, we select West Branch Herring Run At Idlewylde, MD, as the watershed of interest and compute the hydrological signatures for the period from 2010 to 2020.

import pydaymet as daymet
import hydrosignatures as hs
import pygeohydro as gh
from hydrosignatures import HydroSignatures
from pygeohydro import NWIS
from pynhd import WaterData

site = "01585200"
start = "2010-01-01"
end = "2020-12-31"

First, we get the basin geometry of the watershed using gagesii_basins layer of the USGS’s WaterData web service.

wd = WaterData("gagesii_basins")
geometry = wd.byid("gage_id", site).geometry[0]

Then, we obtain the station’s info and streamflow data using NWIS. Note that we should convert the streamflow from cms to mm/day.

nwis = NWIS()
info = nwis.get_info({"site": site})
area_sqm = info.drain_sqkm.values[0] * 1e6
q_cms = nwis.get_streamflow(site, (start, end))
q_mmpd = q_cms * (24.0 * 60.0 * 60.0) / area_sqm * 1e3
q_mmpd.index = pd.to_datetime(q_mmpd.index.date)

Next, we retrieve the precipitation data using PyDaymet over the whole basin using the basin geometry and take its mean as the basin’s precipitation.

prcp = daymet.get_bygeom(geometry, (start, end), variables="prcp")
p_mmpd = prcp.prcp.mean(dim=["x", "y"]).to_pandas()
p_mmpd.index = pd.to_datetime(p_mmpd.index.date)
q_mmpd = q_mmpd.loc[p_mmpd.index]

Now, we can pass these two to the HydroSignatures class:

sig = HydroSignatures(q_mmpd, p_mmpd)

The values property of this class contains the computed signatures. For example, let’s plot the regime curves:

sig.values.mean_monthly.plot()

Note that, you can also use the functions directly. For example, let’s get streamflow observations for another station and separate the baseflow using various filter parameters and compare them:

import numpy as np
import pandas as pd

q = nwis.get_streamflow("12304500", ("2019-01-01", "2019-12-31"))
alpha = np.arange(0.9, 1, 0.01)
qb = pd.DataFrame({a: hs.baseflow(q.squeeze(), alpha=a) for a in alpha})

Lastly, let’s compute Markham’s seasonality index for all streamflow time series of the stations in the CAMELS dataset. We retrieve the CAMELS dataset using PyGeoHydro:

import xarray as xr

_, camels_qobs = gh.get_camels()
discharge = camels_qobs.discharge.dropna("station_id")
discharge = xr.where(discharge < 0, 0, discharge)
si = hs.seasonality_index_markham(discharge.to_pandas())

More examples can be found here.

AsyncRetriever: Asynchronous requests with persistent caching

Features

AsyncRetriever is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services. This package serves as HyRiver’s engine for asynchronously sending requests and retrieving responses as text, binary, or json objects. It uses persistent caching using aiohttp-client-cache to speed up the retrieval even further. Moreover, thanks to nest_asyncio you can use this package in Jupyter notebooks. Although this package is part of the HyRiver software stack, it can be used for any web calls. There are three functions that you can use to make web calls:

• retrieve_text: Get responses as text objects.

• retrieve_binary: Get responses as binary objects.

• retrieve_json: Get responses as json objects.

• stream_write: Stream responses and write them to disk in chunks.

You can also use the general-purpose retrieve function to get responses as any of the three types. All responses are returned as a list that has the same order as the input list of requests. Moreover, there is another function called delete_url_cache for removing all requests from a cache file that contains a given URL.

You can control the request/response caching behavior and verbosity of the package by setting the following environment variables:

• HYRIVER_CACHE_NAME: Path to the caching SQLite database. It defaults to ./cache/aiohttp_cache.sqlite

• HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to one week.

• HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.

• HYRIVER_SSL_CERT: Path to a SSL certificate file.

For example, in your code before making any requests you can do:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/file.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"
os.environ["HYRIVER_SSL_CERT"] = "path/to/cert.pem"

You can find some example notebooks here.

You can also try using AsyncRetriever without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install async-retriever using pip:

$ pip install async-retriever

Alternatively, async-retriever can be installed from the conda-forge repository using Conda:

$ conda install -c conda-forge async-retriever

Quick start

AsyncRetriever by default creates and/or uses ./cache/aiohttp_cache.sqlite as the cache that you can customize by the cache_name argument. Also, by default, the cache doesn’t have any expiration date and the delete_url_cache function should be used if you know that a database on a server was updated, and you want to retrieve the latest data. Alternatively, you can use the expire_after to set the expiration date for the cache.

As an example for retrieving a binary response, let’s use the DAAC server to get NDVI. The responses can be directly passed to xarray.open_mfdataset to get the data as a xarray Dataset. We can also disable SSL certificate verification by setting ssl=False.

import io
import xarray as xr
import async_retriever as ar
from datetime import datetime

west, south, east, north = (-69.77, 45.07, -69.31, 45.45)
base_url = "https://thredds.daac.ornl.gov/thredds/ncss/ornldaac/1299"
dates_itr = ((datetime(y, 1, 1), datetime(y, 1, 31)) for y in range(2000, 2005))
urls, kwds = zip(
    *[
        (
            f"{base_url}/MCD13.A{s.year}.unaccum.nc4",
            {
                "params": {
                    "var": "NDVI",
                    "north": f"{north}",
                    "west": f"{west}",
                    "east": f"{east}",
                    "south": f"{south}",
                    "disableProjSubset": "on",
                    "horizStride": "1",
                    "time_start": s.strftime("%Y-%m-%dT%H:%M:%SZ"),
                    "time_end": e.strftime("%Y-%m-%dT%H:%M:%SZ"),
                    "timeStride": "1",
                    "addLatLon": "true",
                    "accept": "netcdf",
                }
            },
        )
        for s, e in dates_itr
    ]
)
resp = ar.retrieve_binary(urls, kwds, max_workers=8, ssl=False)
data = xr.open_mfdataset(io.BytesIO(r) for r in resp)

We can remove these requests and their responses from the cache like so:

ar.delete_url_cache(base_url)

For a json response example, let’s get water level recordings of an NOAA’s water level station, 8534720 (Atlantic City, NJ), during 2012, using CO-OPS API. Note that this CO-OPS product has a 31-day limit for a single request, so we have to break the request down accordingly.

import pandas as pd

station_id = "8534720"
start = pd.to_datetime("2012-01-01")
end = pd.to_datetime("2012-12-31")

s = start
dates = []
for e in pd.date_range(start, end, freq="m"):
    dates.append((s.date(), e.date()))
    s = e + pd.offsets.MonthBegin()

url = "https://api.tidesandcurrents.noaa.gov/api/prod/datagetter"

urls, kwds = zip(
    *[
        (
            url,
            {
                "params": {
                    "product": "water_level",
                    "application": "web_services",
                    "begin_date": f'{s.strftime("%Y%m%d")}',
                    "end_date": f'{e.strftime("%Y%m%d")}',
                    "datum": "MSL",
                    "station": f"{station_id}",
                    "time_zone": "GMT",
                    "units": "metric",
                    "format": "json",
                }
            },
        )
        for s, e in dates
    ]
)

resp = ar.retrieve_json(urls, kwds)
wl_list = []
for rjson in resp:
    wl = pd.DataFrame.from_dict(rjson["data"])
    wl["t"] = pd.to_datetime(wl.t)
    wl = wl.set_index(wl.t).drop(columns="t")
    wl["v"] = pd.to_numeric(wl.v, errors="coerce")
    wl_list.append(wl)
water_level = pd.concat(wl_list).sort_index()
water_level.attrs = rjson["metadata"]

Now, let’s see an example without any payload or headers. Here’s how we can retrieve harmonic constituents of several NOAA stations from CO-OPS:

stations = [
    "8410140",
    "8411060",
    "8413320",
    "8418150",
    "8419317",
    "8419870",
    "8443970",
    "8447386",
]

base_url = "https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations"
urls = [f"{base_url}/{i}/harcon.json?units=metric" for i in stations]
resp = ar.retrieve_json(urls)

amp_list = []
phs_list = []
for rjson in resp:
    sid = rjson["self"].rsplit("/", 2)[1]
    const = pd.DataFrame.from_dict(rjson["HarmonicConstituents"]).set_index("name")
    amp = const.rename(columns={"amplitude": sid})[sid]
    phase = const.rename(columns={"phase_GMT": sid})[sid]
    amp_list.append(amp)
    phs_list.append(phase)

amp = pd.concat(amp_list, axis=1)
phs = pd.concat(phs_list, axis=1)

PyGeoOGC: Retrieve Data from RESTful, WMS, and WFS Services

Features

PyGeoOGC is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services. This package provides general interfaces to web services that are based on ArcGIS RESTful, WMS, and WFS. Although all these web services have limits on the number of features per request (e.g., 1000 object IDs for a RESTful request or 8 million pixels for a WMS request), PyGeoOGC, first, divides the large requests into smaller chunks, and then returns the merged results.

Moreover, under the hood, PyGeoOGC uses AsyncRetriever for making requests asynchronously with persistent caching. This improves the reliability and speed of data retrieval significantly. AsyncRetriever caches all request/response pairs and upon making an already cached request, it will retrieve the responses from the cache if the server’s response is unchanged.

You can control the request/response caching behavior and verbosity of the package by setting the following environment variables:

• HYRIVER_CACHE_NAME: Path to the caching SQLite database for asynchronous HTTP requests. It defaults to ./cache/aiohttp_cache.sqlite

• HYRIVER_CACHE_NAME_HTTP: Path to the caching SQLite database for HTTP requests. It defaults to ./cache/http_cache.sqlite

• HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds. It defaults to one week.

• HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache. The default is false.

• HYRIVER_SSL_CERT: Path to a SSL certificate file.

For example, in your code before making any requests you can do:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/aiohttp_cache.sqlite"
os.environ["HYRIVER_CACHE_NAME_HTTP"] = "path/to/http_cache.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"
os.environ["HYRIVER_SSL_CERT"] = "path/to/cert.pem"

There is also an inventory of URLs for some of these web services in form of a class called ServiceURL. These URLs are in four categories: ServiceURL().restful, ServiceURL().wms, ServiceURL().wfs, and ServiceURL().http. These URLs provide you with some examples of the services that PyGeoOGC supports. If you have success using PyGeoOGC with a web service please consider submitting a request to be added to this URL inventory. You can get all the URLs in the ServiceURL class by just printing it print(ServiceURL()).

PyGeoOGC has three main classes:

• ArcGISRESTful: This class can be instantiated by providing the target layer URL. For example, for getting Watershed Boundary Data we can use ServiceURL().restful.wbd. By looking at the web service’s website we see that there are nine layers. For example, 1 for 2-digit HU (Region), 6 for 12-digit HU (Subregion), and so on. We can pass the URL to the target layer directly, like this f"{ServiceURL().restful.wbd}/6" or as a separate argument via layer.

  Afterward, we request for the data in two steps. First, we need to get the target object IDs using oids_bygeom (within a geometry), oids_byfield (specific field IDs), or oids_bysql (any valid SQL 92 WHERE clause) class methods. Then, we can get the target features using get_features class method. The returned response can be converted into a geopandas.GeoDataFrame using json2geodf function from PyGeoUtils.

• WMS: Instantiation of this class requires at least 3 arguments: service URL, layer name(s), and output format. Additionally, target CRS and the web service version can be provided. Upon instantiation, we can use getmap_bybox method class to get the target raster data within a bounding box. The box can be in any valid CRS and if it is different from the default CRS, EPSG:4326, it should be passed using box_crs argument. The service response can be converted into a xarray.Dataset using gtiff2xarray function from PyGeoUtils.

• WFS: Instantiation of this class is similar to WMS. The only difference is that only one layer name can be passed. Upon instantiation there are three ways to get the data:

  • getfeature_bybox: Get all the target features within a bounding box in any valid CRS.

  • getfeature_byid: Get all the target features based on the IDs. Note that two arguments should be provided: featurename, and featureids. You can get a list of valid feature names using get_validnames class method.

  • getfeature_byfilter: Get the data based on any valid CQL filter.

  You can convert the returned response of this function to a GeoDataFrame using json2geodf function from PyGeoUtils package.

PyGeoOGC also includes several utilities:

• streaming_download for downloading large files in parallel and in chunks, efficiently.

• traverse_json for traversing a nested JSON object.

• match_crs for reprojecting a geometry or bounding box to any valid CRS.

You can find some example notebooks here.

Furthermore, you can also try using PyGeoOGC without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install PyGeoOGC using pip:

$ pip install pygeoogc

Alternatively, PyGeoOGC can be installed from the conda-forge repository using Conda or Mamba:

$ conda install -c conda-forge pygeoogc

Quick start

We can access NHDPlus HR via RESTful service, National Wetlands Inventory from WMS, and FEMA National Flood Hazard via WFS. The output for these functions are of type requests.Response that can be converted to GeoDataFrame or xarray.Dataset using PyGeoUtils.

Let’s start the National Map’s NHDPlus HR web service. We can query the flowlines that are within a geometry as follows:

from pygeoogc import ArcGISRESTful, WFS, WMS, ServiceURL
import pygeoutils as geoutils
from pynhd import NLDI

basin_geom = NLDI().get_basins("01031500").geometry[0]

hr = ArcGISRESTful(ServiceURL().restful.nhdplushr, 2, outformat="json")

resp = hr.get_features(hr.oids_bygeom(basin_geom, 4326))
flowlines = geoutils.json2geodf(resp)

Note oids_bygeom has three additional arguments: sql_clause, spatial_relation, and distance. We can use sql_clause for passing any valid SQL WHERE clauses and spatial_relation for specifying the target predicate such as intersect, contain, cross, etc. The default predicate is intersect (esriSpatialRelIntersects). Additionally, we can use distance for specifying the buffer distance from the input geometry for getting features.

We can also submit a query based on IDs of any valid field in the database. If the measure property is desired you can pass return_m as True to the get_features class method:

oids = hr.oids_byfield("PERMANENT_IDENTIFIER", ["103455178", "103454362", "103453218"])
resp = hr.get_features(oids, return_m=True)
flowlines = geoutils.json2geodf(resp)

Additionally, any valid SQL 92 WHERE clause can be used. For more details look here. For example, let’s limit our first request to only include catchments with areas larger than 0.5 sqkm.

oids = hr.oids_bygeom(basin_geom, geo_crs=4326, sql_clause="AREASQKM > 0.5")
resp = hr.get_features(oids)
catchments = geoutils.json2geodf(resp)

A WMS-based example is shown below:

wms = WMS(
    ServiceURL().wms.fws,
    layers="0",
    outformat="image/tiff",
    crs=3857,
)
r_dict = wms.getmap_bybox(
    basin_geom.bounds,
    1e3,
    box_crs=4326,
)
wetlands = geoutils.gtiff2xarray(r_dict, basin_geom, 4326)

Query from a WFS-based web service can be done either within a bounding box or using any valid CQL filter.

wfs = WFS(
    ServiceURL().wfs.fema,
    layer="public_NFHL:Base_Flood_Elevations",
    outformat="esrigeojson",
    crs=4269,
)
r = wfs.getfeature_bybox(basin_geom.bounds, box_crs=4326)
flood = geoutils.json2geodf(r.json(), 4269, 4326)

layer = "wmadata:huc08"
wfs = WFS(
    ServiceURL().wfs.waterdata,
    layer=layer,
    outformat="application/json",
    version="2.0.0",
    crs=4269,
)
r = wfs.getfeature_byfilter(f"huc8 LIKE '13030%'")
huc8 = geoutils.json2geodf(r.json(), 4269, 4326)

PyGeoUtils: Utilities for (Geo)JSON and (Geo)TIFF Conversion

Features

PyGeoUtils is a part of HyRiver software stack that is designed to aid in hydroclimate analysis through web services. This package provides utilities for manipulating (Geo)JSON and (Geo)TIFF responses from web services. These utilities are:

• Coordinates: Generate validated and normalized coordinates in WGS84.

• GeoBSpline: Create B-spline from a geopandas.GeoDataFrame of points.

• smooth_linestring: Smooth a shapely.geometry.LineString using B-spline.

• bspline_curvature: Compute tangent angles, curvature, and radius of curvature of a B-Spline at any points along the curve.

• arcgis2geojson: Convert ESRIGeoJSON format to GeoJSON.

• break_lines: Break lines at specified points in a given direction.

• gtiff2xarray: Convert (Geo)Tiff byte responses to xarray.Dataset.

• json2geodf: Create geopandas.GeoDataFrame from (Geo)JSON responses

• snap2nearest: Find the nearest points on a line to a set of points.

• xarray2geodf: Vectorize a xarray.DataArray to a geopandas.GeoDataFrame.

• geodf2xarray: Rasterize a geopandas.GeoDataFrame to a xarray.DataArray.

• xarray_geomask: Mask a xarray.Dataset based on a geometry.

• query_indices: A wrapper around geopandas.sindex.query_bulk. However, instead of returning an array of positional indices, it returns a dictionary of indices where keys are the indices of the input geometry and values are a list of indices of the tree geometries that intersect with the input geometry.

• nested_polygons: Determining nested (multi)polygons in a geopandas.GeoDataFrame.

• multi2poly: For converting a MultiPolygon to a Polygon in a geopandas.GeoDataFrame.

• geometry_reproject: For reprojecting a geometry (bounding box, list of coordinates, or any shapely.geometry) to a new CRS.

• gtiff2vrt: For converting a list of GeoTIFF files to a VRT file.

You can find some example notebooks here.

You can also try using PyGeoUtils without installing it on your system by clicking on the binder badge. A Jupyter Lab instance with the HyRiver stack pre-installed will be launched in your web browser, and you can start coding!

Moreover, requests for additional functionalities can be submitted via issue tracker.

Citation

If you use any of HyRiver packages in your research, we appreciate citations:

@article{Chegini_2021,
    author = {Chegini, Taher and Li, Hong-Yi and Leung, L. Ruby},
    doi = {10.21105/joss.03175},
    journal = {Journal of Open Source Software},
    month = {10},
    number = {66},
    pages = {1--3},
    title = {{HyRiver: Hydroclimate Data Retriever}},
    volume = {6},
    year = {2021}
}

Installation

You can install PyGeoUtils using pip after installing libgdal on your system (for example, in Ubuntu run sudo apt install libgdal-dev).

$ pip install pygeoutils

Alternatively, PyGeoUtils can be installed from the conda-forge repository using Conda:

$ conda install -c conda-forge pygeoutils

Quick start

We start by smoothing a shapely.geometry.LineString using B-spline:

import pygeoutils as pgu
from shapely import LineString

line = LineString(
    [
        (-97.06138, 32.837),
        (-97.06133, 32.836),
        (-97.06124, 32.834),
        (-97.06127, 32.832),
    ]
)
line = pgu.geometry_reproject(line, 4326, 5070)
sp = pgu.smooth_linestring(line, 5070, 5)
line_sp = pgu.geometry_reproject(sp.line, 5070, 4326)

Next, we use PyGeoOGC to access National Wetlands Inventory from WMS, and FEMA National Flood Hazard via WFS, then convert the output to xarray.Dataset and GeoDataFrame, respectively.

from pygeoogc import WFS, WMS, ServiceURL
from shapely.geometry import Polygon


geometry = Polygon(
    [
        [-118.72, 34.118],
        [-118.31, 34.118],
        [-118.31, 34.518],
        [-118.72, 34.518],
        [-118.72, 34.118],
    ]
)
crs = 4326

wms = WMS(
    ServiceURL().wms.mrlc,
    layers="NLCD_2011_Tree_Canopy_L48",
    outformat="image/geotiff",
    crs=crs,
)
r_dict = wms.getmap_bybox(
    geometry.bounds,
    1e3,
    box_crs=crs,
)
canopy = pgu.gtiff2xarray(r_dict, geometry, crs)

mask = canopy > 60
canopy_gdf = pgu.xarray2geodf(canopy, "float32", mask)

url_wfs = "https://hazards.fema.gov/gis/nfhl/services/public/NFHL/MapServer/WFSServer"
wfs = WFS(
    url_wfs,
    layer="public_NFHL:Base_Flood_Elevations",
    outformat="esrigeojson",
    crs=4269,
)
r = wfs.getfeature_bybox(geometry.bounds, box_crs=crs)
flood = pgu.json2geodf(r.json(), 4269, crs)

Example Gallery

The following notebooks demonstrate capabilities of the HyRiver software stack.

Relative Elevation Model

Navigate NHD Network

Flood Moments for CAMELS

Geospatial Hydrologic Data with Web Services

High Water Marks using STN

USGS 3DHP

Soil Storage Capacity

Spatiotemporal Distribution of Dams

Daymet Climate Data

GridMET Climate Data

NLDAS2 Climate Data

Split Catchment with PyGeoAPI

Real-time data with SensorThings API

River Elevation and Cross-Section

CONUS Coasts

National Inventory of Dams

Dams in Columbia River Network

Land Use/Land Cover

Computing PET

Hydrological Modeling

Actual Evapotranspiration

Topographic Data

Aridity

Water Quality Portal

River Discharge

River Network

NDVI

Water Levels and Tides

GeoConnex

Web Services Queries

This page was generated from 3dep.ipynb. Interactive online version:

Topographic Data

from pathlib import Path

import matplotlib.pyplot as plt
import networkx as nx
import numpy as np
import osmnx as ox
import osmnx.plot as ox_plot
import xarray as xr
import xrspatial

import py3dep
import pydaymet as daymet
from pynhd import NLDI

Py3DEP provides access to the 3DEP database which is a part of the National Map services. The 3DEP service has multi-resolution sources and depending on the user provided resolution, the data is resampled on the server-side based on all the available data sources.

The following functionalities are currently available:

• get_map: Get topographic data the dynamic 3DEP service that supports the following layers:

  • DEM

  • Hillshade Gray

  • Aspect Degrees

  • Aspect Map

  • GreyHillshade Elevation Fill

  • Hillshade Multidirectional

  • Slope Degrees

  • Slope Map

  • Hillshade Elevation Tinted

  • Height Ellipsoidal

  • Contour 25

  • Contour Smoothed 25

• get_dem: Get DEM data from either the dynamic or static 3DEP service. Considering that the static service is much faster, if the target DEM resolution is 10 m, 30 m, or 60 m, then the static service is used. Otherwise, the dynamic service is used.

• static_3dep_dem: Get DEM data at 10 m, 30 m, or 60 m resolution from the staged 3DEP data. Since this function only returns DEM, for computing other terrain attributes you can use xarray-spatial. Just note that you should reproject the output DataArray to a projected CRS like 5070 before passing it to xarray-spatial like so: dem = dem.rio.reproject(5070).

• elevation_bygrid: For retrieving elevations of all the grid points in a 2D grid.

• elevation_bycoords: For retrieving elevation of a list of x and y coordinates.

• elevation_profile: For retrieving elevation profile along a line at a given spacing. This function converts the line to a B-spline and then calculates the elevation along the spline at a given uniform spacing.

• deg2mpm: For converting slope dataset from degree to meter per meter.

• query_3dep_sources: For querying bounds of 3DEP’s data sources within a bounding box.

• check_3dep_availability: For querying 3DEP’s resolution availability within a bounding box.

Let’s get a watershed geometry using NLDI and then get DEM and slope at 90 m resolution.

geometry = NLDI().get_basins("11092450").geometry[0]

We can either directly use get_map to get DEM and slope or use get_dem to get DEM and then use xarray-spatial to compute slope.

topo = py3dep.get_map(["DEM", "Slope Degrees"], geometry, 90, geo_crs=4326, crs=5070)

dem = py3dep.get_dem(geometry, 30)
dem = dem.rio.reproject(5070)
slope = py3dep.deg2mpm(xrspatial.slope(dem))
topo = xr.merge([dem, slope])

We can save the DEM DataArray as a raster file using rioxarray:

dem.rio.to_raster(Path("input_data", "dem_11092450.tif"))

fig, axs = plt.subplots(ncols=2, figsize=(13, 4))
dem.plot(ax=axs[0])
slope.plot(ax=axs[1])
for ax in axs:
    ax.set_title("")
    ax.set_axis_off()
fig.savefig("_static/dem_slope.png", bbox_inches="tight", facecolor="w")

We can also get elevations of a list of coordinates using py3dep.elevation_bycoords function. This function is particularly useful for getting elevations of nodes of a network, for example, is a river or a street network. Let’s use osmnx package to get a street network:

G = ox.graph_from_place("Piedmont, California, USA", network_type="drive")

Now, we can get the elevations for each node based on their coordinates and then plot the results.

x, y = nx.get_node_attributes(G, "x").values(), nx.get_node_attributes(G, "y").values()
elevation = py3dep.elevation_bycoords(list(zip(x, y)), crs="epsg:4326", source="airmap")
nx.set_node_attributes(G, dict(zip(G.nodes(), elevation)), "elevation")

nc = ox_plot.get_node_colors_by_attr(G, "elevation", cmap="terrain")
fig, ax = ox.plot_graph(
    G,
    node_color=nc,
    node_size=10,
    save=True,
    bgcolor="w",
    filepath="_static/street_elev.png",
    dpi=100,
)

Note that, this function gets the elevation data from the elevation map of the bounding box of all the coordinates. So, if the larger the extent of this bounding box, the longer is going to take for the function to get the data.

Additionally, we can get the elevations of set of x- and y- coordinates of a grid. For example, let’s get the minimum temperature data within the watershed from Daymet using PyDaymet then add the elevation as a new variable to the dataset:

clm = daymet.get_bygeom(geometry, ("2005-01-01", "2005-01-31"), variables="tmin")
elev = py3dep.elevation_bygrid(clm.x.values, clm.y.values, clm.rio.crs, 1000)
attrs = clm.attrs
clm = xr.merge([clm, elev])
clm["elevation"] = clm.elevation.where(~np.isnan(clm.isel(time=0).tmin))
clm.attrs.update(attrs)

fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(15, 5))
clm.tmin.isel(time=10).plot(ax=ax1)
clm.elevation.plot(ax=ax2)

<matplotlib.collections.QuadMesh at 0x1b2e299d0>

This page was generated from 3dhp.ipynb. Interactive online version:

USGS 3DHP

import shapely

from pynhd import HP3D, NLDI

USGS 3DHP web service has five layers: hydrolocation, flowline, waterbody, drainage_area, and catchment. Let’s start by getting the closest flowline within a 10 m radius of a point.

nhd3d = HP3D("flowline")
point = shapely.Point(-89.441, 43.487)
flw = nhd3d.bygeom(point, distance=10)
flw.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

Next, we use NLDI to get the basin boundary of a NHD flowline and use it to query different layers of the 3DHP web service.

comid = "937070225"
basin = NLDI().get_basins(comid, "comid")
basin.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

nhd3d = HP3D("flowline")
network = nhd3d.bygeom(basin.geometry[0])
network.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

nhd3d = HP3D("waterbody")
water = nhd3d.bygeom(basin.geometry[0])
water.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

nhd3d = HP3D("hydrolocation")
hydrolocation = nhd3d.bygeom(basin.geometry[0])
hydrolocation.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

We can also query the service using mainstems.

dm = ("https://geoconnex.us/ref/mainstems/323742", "https://geoconnex.us/ref/mainstems/312091")
nhd3d = HP3D("flowline")
down_mains = nhd3d.byids("mainstemid", dm)
down_mains.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

Or using NHD ReachCode.

reachcode = "07070004002889"
nhd3d = HP3D("hydrolocation")
hydrolocation = nhd3d.byids("universalreferenceid", reachcode)
hydrolocation

	geometry	OBJECTID	id3dhp	featuredate	mainstemid	universalreferenceid	gnisid	gnisidlabel	featuretype	featuretypelabel	edhuniqueid	workunitid
0	POINT (-89.44179 43.48829)	5800707	01HAJW7	1694649600000	https://geoconnex.us/ref/mainstems/359842	07070004002889	None	None	10	Reachcode Start	None	None
1	POINT (-89.43904 43.48568)	25271902	01SVVYQ	1694649600000	https://geoconnex.us/ref/mainstems/359842	07070004002889	None	None	11	Reachcode End	None	None

Let’s query flowlines and hydrolocations for a list of mainstems.

nhd3d = HP3D("hydrolocation")
mainstem_points = nhd3d.byids("mainstemid", hydrolocation["mainstemid"].to_list())
nhd3d = HP3D("flowline")
mainstem_lines = nhd3d.byids("mainstemid", hydrolocation["mainstemid"].to_list())
ax = mainstem_lines.plot(zorder=0, figsize=(8, 8))
mainstem_points.plot(ax=ax, color="k", markersize=8)
hydrolocation.plot(ax=ax, color="r", markersize=60, marker="*")
ax.legend(["Flowlines", "Mainstem Points", "Hydrolocation"])
ax.set_axis_off()
ax.figure.savefig("_static/hydrolocation.png", bbox_inches="tight", dpi=300)

This page was generated from aridity.ipynb. Interactive online version:

Aridity

Let’s start by getting the basin geometry of USGS-01031500 station (Piscataquis River near Dover-Foxcroft, Maine)

import os
from pathlib import Path

import geopandas as gpd
import hvplot.xarray  # noqa

import pynhd as nhd

root = Path("input_data")
os.makedirs(root, exist_ok=True)
BASE_PLOT = {"facecolor": "k", "edgecolor": "b", "alpha": 0.2, "figsize": (18, 9)}
CRS = "esri:102008"
station_id = "01031500"

nldi = nhd.NLDI()
cfile = Path(root, f"basin_{station_id}.feather")
if cfile.exists():
    basin = gpd.read_feather(cfile)
else:
    basin = nldi.get_basins(station_id)
    basin.to_feather(cfile)

ax = basin.to_crs(CRS).plot(**BASE_PLOT)
ax.axis("off")
ax.margins(0)

Precipitation and Potential Evapotranspiration

from tqdm.notebook import tqdm

import pydaymet as daymet

years = list(range(2006, 2016))
geometry = basin.iloc[0].geometry

for yr in tqdm(years, desc="Getting Climate"):
    cfile = Path(root, f"clm_{yr}.nc")
    if cfile.exists():
        continue
    daymet.get_bygeom(geometry, yr, variables="prcp", pet="hargreaves_samani").to_netcdf(cfile)

import xarray as xr

clm = xr.open_mfdataset(Path(root).glob("clm_*.nc"), coords="minimal")
clm["elevation"] = clm["elevation"].isel(time=0, drop=True)
clm

<xarray.Dataset>
Dimensions:    (time: 3650, y: 37, x: 40)
Coordinates:
  * time       (time) datetime64[ns] 2006-01-01T12:00:00 ... 2015-12-31T12:00:00
  * y          (y) float32 719.0 718.0 717.0 716.0 ... 686.0 685.0 684.0 683.0
  * x          (x) float32 2.21e+03 2.211e+03 2.212e+03 ... 2.248e+03 2.249e+03
Data variables:
    dayl       (time, y, x) float32 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
    lat        (time, y, x) float64 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
    lon        (time, y, x) float64 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
    prcp       (time, y, x) float32 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
    srad       (time, y, x) float32 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
    tmax       (time, y, x) float32 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
    tmin       (time, y, x) float32 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
    vp         (time, y, x) float32 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
    elevation  (y, x) float64 dask.array<chunksize=(37, 40), meta=np.ndarray>
    pet        (time, y, x) float64 dask.array<chunksize=(365, 37, 40), meta=np.ndarray>
Attributes: (12/17)
    start_year:          2006
    source:              Daymet Software Version 4.0
    Version_software:    Daymet Software Version 4.0
    Version_data:        Daymet Data Version 4.0
    Conventions:         CF-1.6
    citation:            Please see http://daymet.ornl.gov/ for current Dayme...
    ...                  ...
    geospatial_lon_max:  -69.13309404831536
    crs:                 +proj=lcc +lat_1=25 +lat_2=60 +lat_0=42.5 +lon_0=-10...
    nodatavals:          0.0
    transform:           [ 1.00000e+00  0.00000e+00  2.20625e+03  0.00000e+00...
    res:                 [1. 1.]
    bounds:              [2209.24651856  682.07851575 2250.2123966   719.9719...

xarray.Dataset

• Dimensions:
  • time: 3650
  • y: 37
  • x: 40
• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2006-01-01T12:00:00 ... 2015-12-...

    standard_name :

    time

    bounds :

    time_bnds

    long_name :

    24-hour day based on local time

    _ChunkSizes :

    1

    _CoordinateAxisType :

    Time

    array(['2006-01-01T12:00:00.000000000', '2006-01-02T12:00:00.000000000',
           '2006-01-03T12:00:00.000000000', ..., '2015-12-29T12:00:00.000000000',
           '2015-12-30T12:00:00.000000000', '2015-12-31T12:00:00.000000000'],
          dtype='datetime64[ns]')

  • y
    (y)
    float32
    719.0 718.0 717.0 ... 684.0 683.0

    units :

    km

    long_name :

    y coordinate of projection

    standard_name :

    projection_y_coordinate

    array([719., 718., 717., 716., 715., 714., 713., 712., 711., 710., 709., 708.,
           707., 706., 705., 704., 703., 702., 701., 700., 699., 698., 697., 696.,
           695., 694., 693., 692., 691., 690., 689., 688., 687., 686., 685., 684.,
           683.], dtype=float32)

  • x
    (x)
    float32
    2.21e+03 2.211e+03 ... 2.249e+03

    units :

    km

    long_name :

    x coordinate of projection

    standard_name :

    projection_x_coordinate

    array([2209.75, 2210.75, 2211.75, 2212.75, 2213.75, 2214.75, 2215.75, 2216.75,
           2217.75, 2218.75, 2219.75, 2220.75, 2221.75, 2222.75, 2223.75, 2224.75,
           2225.75, 2226.75, 2227.75, 2228.75, 2229.75, 2230.75, 2231.75, 2232.75,
           2233.75, 2234.75, 2235.75, 2236.75, 2237.75, 2238.75, 2239.75, 2240.75,
           2241.75, 2242.75, 2243.75, 2244.75, 2245.75, 2246.75, 2247.75, 2248.75],
          dtype=float32)

• Data variables: (10)
  • dayl
    (time, y, x)
    float32
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    long_name :

    daylength

    units :

    s/day

    grid_mapping :

    lambert_conformal_conic

    cell_methods :

    area: mean

    _ChunkSizes :

    [ 16777216 -402456576 -402456576]

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 20.61 MiB 2.06 MiB Shape (3650, 37, 40) (365, 37, 40) Count 30 Tasks 10 Chunks Type float32 numpy.ndarray

  • lat
    (time, y, x)
    float64
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    units :

    degrees_north

    long_name :

    latitude coordinate

    standard_name :

    latitude

    _CoordinateAxisType :

    Lat

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 41.21 MiB 4.12 MiB Shape (3650, 37, 40) (365, 37, 40) Count 40 Tasks 10 Chunks Type float64 numpy.ndarray

  • lon
    (time, y, x)
    float64
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    units :

    degrees_east

    long_name :

    longitude coordinate

    standard_name :

    longitude

    _CoordinateAxisType :

    Lon

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 41.21 MiB 4.12 MiB Shape (3650, 37, 40) (365, 37, 40) Count 40 Tasks 10 Chunks Type float64 numpy.ndarray

  • prcp
    (time, y, x)
    float32
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    long_name :

    daily total precipitation

    units :

    mm/day

    grid_mapping :

    lambert_conformal_conic

    cell_methods :

    area: mean time: sum

    _ChunkSizes :

    [ 16777216 -402456576 -402456576]

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 20.61 MiB 2.06 MiB Shape (3650, 37, 40) (365, 37, 40) Count 30 Tasks 10 Chunks Type float32 numpy.ndarray

  • srad
    (time, y, x)
    float32
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    long_name :

    daylight average incident shortwave radiation

    units :

    W/m2

    grid_mapping :

    lambert_conformal_conic

    cell_methods :

    area: mean time: mean

    _ChunkSizes :

    [ 16777216 -402456576 -402456576]

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 20.61 MiB 2.06 MiB Shape (3650, 37, 40) (365, 37, 40) Count 30 Tasks 10 Chunks Type float32 numpy.ndarray

  • tmax
    (time, y, x)
    float32
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    long_name :

    daily maximum temperature

    units :

    degrees C

    grid_mapping :

    lambert_conformal_conic

    cell_methods :

    area: mean time: maximum

    _ChunkSizes :

    [ 16777216 -402456576 -402456576]

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 20.61 MiB 2.06 MiB Shape (3650, 37, 40) (365, 37, 40) Count 30 Tasks 10 Chunks Type float32 numpy.ndarray

  • tmin
    (time, y, x)
    float32
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    long_name :

    daily minimum temperature

    units :

    degrees C

    grid_mapping :

    lambert_conformal_conic

    cell_methods :

    area: mean time: minimum

    _ChunkSizes :

    [ 16777216 -402456576 -402456576]

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 20.61 MiB 2.06 MiB Shape (3650, 37, 40) (365, 37, 40) Count 30 Tasks 10 Chunks Type float32 numpy.ndarray

  • vp
    (time, y, x)
    float32
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    long_name :

    daily average vapor pressure

    units :

    Pa

    grid_mapping :

    lambert_conformal_conic

    cell_methods :

    area: mean time: mean

    _ChunkSizes :

    [ 16777216 -402456576 -402456576]

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 20.61 MiB 2.06 MiB Shape (3650, 37, 40) (365, 37, 40) Count 30 Tasks 10 Chunks Type float32 numpy.ndarray

  • elevation
    (y, x)
    float64
    dask.array<chunksize=(37, 40), meta=np.ndarray>

    units :

    meters

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 11.56 kiB 11.56 kiB Shape (37, 40) (37, 40) Count 41 Tasks 1 Chunks Type float64 numpy.ndarray

  • pet
    (time, y, x)
    float64
    dask.array<chunksize=(365, 37, 40), meta=np.ndarray>

    units :

    mm/day

    crs :

    epsg:4326

    nodatavals :

    0.0

    Array Chunk Bytes 41.21 MiB 4.12 MiB Shape (3650, 37, 40) (365, 37, 40) Count 30 Tasks 10 Chunks Type float64 numpy.ndarray

• Attributes: (17)

  start_year :

  2006

  source :

  Daymet Software Version 4.0

  Version_software :

  Daymet Software Version 4.0

  Version_data :

  Daymet Data Version 4.0

  Conventions :

  CF-1.6

  citation :

  Please see http://daymet.ornl.gov/ for current Daymet data citation information

  references :

  Please see http://daymet.ornl.gov/ for current information on Daymet references

  History :

  Translated to CF-1.0 Conventions by Netcdf-Java CDM (CFGridWriter2) Original Dataset = /daacftp/daymet/Daymet_Daily_V4/data/daymet_v4_daily_na_dayl_2006.nc; Translation Date = 2021-05-26T07:00:14.233Z

  geospatial_lat_min :

  44.964817634118106

  geospatial_lat_max :

  45.5636129082089

  geospatial_lon_min :

  -69.95790349094842

  geospatial_lon_max :

  -69.13309404831536

  crs :

  +proj=lcc +lat_1=25 +lat_2=60 +lat_0=42.5 +lon_0=-100 +x_0=0 +y_0=0 +ellps=WGS84 +units=km +no_defs

  nodatavals :

  0.0

  transform :

  [ 1.00000e+00 0.00000e+00 2.20625e+03 0.00000e+00 -1.00000e+00 7.29500e+02 0.00000e+00 0.00000e+00 1.00000e+00]

  res :

  [1. 1.]

  bounds :

  [2209.24651856 682.07851575 2250.2123966 719.97190299]

clm.elevation.plot()

<matplotlib.collections.QuadMesh at 0x1b8c9a980>

clm.hvplot.violin(y="pet", by="time.month")

Aridity

variables = {"pet": "Mean Annual PET", "prcp": "Mean Annual P"}
data = {}
for v, n in variables.items():
    data[v] = clm[v].groupby("time.year").sum().mean(dim="year")
    data[v] = data[v].where(data[v] > 0)
    data[v] = data[v].rename(n)
    data[v] = data[v].assign_attrs(unit="mm/year")

aridity = data["pet"] / data["prcp"]
aridity = aridity.rename("Aridity Index")

ax = aridity.plot(size=6)
ax.figure.savefig(Path("_static", "aridity.png"), dpi=300, bbox_inches="tight", facecolor="w")

This page was generated from columbia.ipynb. Interactive online version:

Dams in Columbia River Network

import warnings
from pathlib import Path

warnings.filterwarnings("ignore", message=".*initial implementation of Parquet.*")
root = Path("input_data")
root.mkdir(parents=True, exist_ok=True)
BASE_PLOT = {"facecolor": "k", "edgecolor": "b", "alpha": 0.2, "figsize": (18, 9)}
CRS = "esri:102008"

Basin

import geopandas as gpd

import pynhd as nhd

nldi = nhd.NLDI()
station_id = "14246900"
cfile = Path(root, f"basin_{station_id}.feather")
if cfile.exists():
    basin = gpd.read_feather(cfile)
else:
    basin = nldi.get_basins(station_id)
    basin.to_feather(cfile)

Main

cfile = Path(root, f"flowline_main_{station_id}.feather")
if cfile.exists():
    flw_main = gpd.read_feather(cfile)
else:
    flw_main = nldi.navigate_byid(
        fsource="nwissite",
        fid=f"USGS-{station_id}",
        navigation="upstreamMain",
        source="flowlines",
        distance=2000,
    )
    flw_main.to_feather(cfile)

Tributaries

cfile = Path(root, f"flowline_trib_{station_id}.feather")
if cfile.exists():
    flw_trib = gpd.read_feather(cfile)
else:
    flw_trib = nldi.navigate_byid(
        fsource="nwissite",
        fid=f"USGS-{station_id}",
        navigation="upstreamTributaries",
        source="flowlines",
        distance=2000,
    )
    flw_trib.to_feather(cfile)
flw_trib["nhdplus_comid"] = flw_trib["nhdplus_comid"].astype("float").astype("Int64")

ax = basin.plot(**BASE_PLOT)
flw_trib.plot(ax=ax)
flw_main.plot(ax=ax, lw=3, color="r")
ax.legend(["Tributaries", "Main"])
ax.axis("off")
ax.margins(0)

Accumulated Dams

import pandas as pd

cfile = Path(root, "nid_flw.pkl")

if cfile.exists():
    nid_flw = pd.read_pickle(cfile)
else:
    meta = nhd.nhdplus_attrs()
    nid_years = (
        meta[meta.description.str.contains("dam", case=False)].sort_values("name").name.tolist()
    )
    nid_flw = {n.split("_")[-1]: nhd.nhdplus_attrs(n) for n in nid_years}
    pd.to_pickle(nid_flw, cfile)

Now, let’s see what catchment-level characteristics are available that are related to dams.

div_chars = nldi.valid_characteristics
div_chars[div_chars.characteristic_description.str.contains("dam")]

	characteristic_id	characteristic_description	units	dataset_label	dataset_url	theme_label	theme_url	characteristic_type
102	CAT_NORM_STORAGE2013	The normal dam storage (in acre-feet) defined ...	acre-feet	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	localCatch_name
118	CAT_MAJOR2013	Number of major dams built on or before YYYY i...	count	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	localCatch_name
120	CAT_NDAMS2013	Number of dams built on or before YYYY ,	count	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	localCatch_name
121	CAT_NID_STORAGE2013	The maximum dam storage (in acre-feet) defined...	acre-feet	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	localCatch_name
247	ACC_NDAMS2013	Number of dams built on or before YYYY ,	count	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	divRoute_name
248	ACC_NID_STORAGE2013	The maximum dam storage (in acre-feet) defined...	acre-feet	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	divRoute_name
249	ACC_NORM_STORAGE2013	The normal dam storage (in acre-feet) defined ...	acre-feet	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	divRoute_name
250	ACC_MAJOR2013	Number of major dams built on or before YYYY i...	count	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	divRoute_name
374	TOT_NDAMS2013	Number of dams built on or before YYYY ,	count	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	totRoute_name
375	TOT_NID_STORAGE2013	The maximum dam storage (in acre-feet) defined...	acre-feet	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	totRoute_name
376	TOT_NORM_STORAGE2013	The normal dam storage (in acre-feet) defined ...	acre-feet	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	totRoute_name
377	TOT_MAJOR2013	Number of major dams built on or before YYYY i...	count	Number of major dams built on or before YYYY i...	https://www.sciencebase.gov/catalog/item/58c30...	Hydrologic Modifications	unknown	totRoute_name

Let’s get ACC_NID_STORAGE2013:

We can achieve the same results using another function that uses ScienceBase instead of NLDI:

nid_vals = nldi.getcharacteristic_byid(flw_trib.nhdplus_comid.tolist(), "div", "ACC_NID_STORAGE2013")

comids = [int(c) for c in flw_trib.nhdplus_comid.tolist()]
nid_vals = {
    yr: df.loc[df.COMID.isin(comids), ["COMID", f"ACC_NID_STORAGE{yr}", f"ACC_NDAMS{yr}"]].rename(
        columns={
            "COMID": "comid",
            f"ACC_NID_STORAGE{yr}": "smax",
            f"ACC_NDAMS{yr}": "ndams",
        }
    )
    for yr, df in nid_flw.items()
}
nid_vals = pd.concat(nid_vals).reset_index().drop(columns="level_1")
nid_vals = nid_vals.rename(columns={"level_0": "year"}).astype({"year": int})

comids = [int(c) for c in flw_trib.nhdplus_comid.tolist()]
nid_vals = {
    yr: df.loc[df.COMID.isin(comids), ["COMID", f"ACC_NID_STORAGE{yr}", f"ACC_NDAMS{yr}"]].rename(
        columns={
            "COMID": "comid",
            f"ACC_NID_STORAGE{yr}": "smax",
            f"ACC_NDAMS{yr}": "ndams",
        }
    )
    for yr, df in nid_flw.items()
}
nid_vals = pd.concat(nid_vals).reset_index().drop(columns="level_1")
nid_vals = nid_vals.rename(columns={"level_0": "year"}).astype({"year": int})

Accumulated Max Storage

nid_vals = (
    nid_vals.set_index("comid")
    .merge(
        flw_trib.astype({"nhdplus_comid": int}).set_index("nhdplus_comid"),
        left_index=True,
        right_index=True,
        suffixes=(None, None),
    )
    .reset_index()
    .rename(columns={"index": "comid"})
)
smax = nid_vals.groupby(["year", "comid"]).sum(numeric_only=True)["smax"].unstack()
smax = gpd.GeoDataFrame(
    smax.T.merge(
        flw_trib.astype({"nhdplus_comid": int}).set_index("nhdplus_comid"),
        left_index=True,
        right_index=True,
        suffixes=(None, None),
    )
)

yr = 2013
ax = basin.plot(**BASE_PLOT)
smax.plot(ax=ax, scheme="Quantiles", k=2, column=yr, cmap="coolwarm", lw=0.5, legend=False)
ax.set_title(f"Accumulated Maximum Storage Capacity of Dams up to {yr}")
ax.axis("off")
ax.margins(0)
ax.figure.savefig(Path("_static", "columbia.png"), dpi=300, bbox_inches="tight", facecolor="w")

This page was generated from cross_section.ipynb. Interactive online version:

River Elevation and Cross-Section

from pathlib import Path

import geopandas as gpd
import numpy as np
import pandas as pd
from shapely.geometry import Point

import py3dep
import pygeoogc.utils as ogc_utils
import pynhd
from pynhd import NLDI, WaterData

We can retrieve elevation profile for tributaries of a given USGS station ID using PyNHD and Py3DEP. For this purpose, we get the elevation data for points along the tributaries’ flowlines every one kilometer. Note that since the distance is in meters we reproject the geospatial data into ESRI:102003.

crs_proj = 5070
station_id = "01031500"
distance = 1000  # in meters

First, let’s get the basin geometry and tributaries for the station.

nldi = NLDI()

basin = nldi.get_basins(station_id)
flw = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="flowlines",
    distance=100,
)
flw = flw.set_index("nhdplus_comid").to_crs(crs_proj)

First, we need to get the main river’s flowlines.

Now, we can compute the number of points along each river segment based on the target 1-km distance. Note that since we want to sample the data from a DEM in EPSG:4326 wen need to reproject the sample points from EPSG:102003 to EPSG:4326. We use pygeoogc.utils.match_crs to do so.

flw["n_points"] = np.ceil(flw.length / distance).astype("int")
flw_points = [
    (
        n,
        list(
            zip(
                *ogc_utils.match_crs(
                    [
                        (p[0][0], p[1][0])
                        for p in (
                            l.interpolate(x, normalized=True).xy
                            for x in np.linspace(0, 1, pts, endpoint=False)
                        )
                    ],
                    crs_proj,
                    4326,
                )
            )
        ),
    )
    for n, l, pts in flw.itertuples(name=None)
]

We use Py3DEP to get elevation data for these points. We can either use py3dep.elevation_bycoords to get elevations directly from 3DEP’s static DEM VRTs at 10-m resolution or use py3dep.get_dem to get the elevation data for the entire basin and then sample it for the target locations.

Using py3dep.elevation_bycoords is simpler but in this case we want to apply hydrologic conditioning to the DEM. For this purpose, we use py3dep.get_dem to get the elevation data for the entire basin and then sample it for the target locations.Here’s the code to get the elevation data using py3dep.elevation_bycoords:

flw_elevation = [(n, py3dep.elevation_bycoords(pts, CRS)) for n, pts in flw_points]

Let’s continue with using py3dep.get_dem.

dem = py3dep.get_dem(basin.geometry[0].bounds, resolution=10)

Py3DEP uses pyflwdir for filling depressions. It’s an optional dependency and you need to install it separately.

dem = py3dep.fill_depressions(dem)
dem.plot.imshow()

<matplotlib.image.AxesImage at 0x2d5024550>

flw_elevation = [
    (n, dem.interp(x=("z", list(pts[0])), y=("z", list(pts[1]))).values) for n, pts in flw_points
]

Now that we have both the coordinates and their elevation data for the points, we can create a new dataframe and plot the results.

points = pd.DataFrame(((n, list(zip(*pts))) for n, pts in flw_points)).set_index(0)[1].explode()
points = pd.DataFrame([(n, Point(p)) for n, p in points.items()])
points = points.rename(columns={0: "comid", 1: "geometry"}).set_index("comid")
points = gpd.GeoDataFrame(points, crs=4326).to_crs(crs_proj)
elevations = pd.DataFrame(flw_elevation).rename(columns={0: "comid", 1: "elevation"})
elevations = elevations.set_index("comid")["elevation"].explode().astype("float")
points["elevation"] = elevations

ax = basin.to_crs(crs_proj).plot(figsize=(6, 6), facecolor="none", edgecolor="black")
points.plot(ax=ax, column="elevation", markersize=6, legend=True, scheme="quantiles")
flw.plot(ax=ax, color="b", linewidth=0.8, alpha=0.8)
ax.set_axis_off()

Let’s extract corss-section profiles along the main river of the basin. We get cross-section profiles at every 3 km along the main river and within a buffer distance of 2 km.

distance = 3000  # in meters
width = 2000  # in meters
half_width = width * 0.5

First, we need to get the main river’s flowlines.

main = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamMain",
    source="flowlines",
    distance=100,
)
main = main.set_index("nhdplus_comid").to_crs(crs_proj)

nhd = WaterData("nhdflowline_network")
main_nhd = nhd.byid("comid", main.index.to_list())
main_nhd = pynhd.prepare_nhdplus(main_nhd, 0, 0, 0, purge_non_dendritic=True).to_crs(crs_proj)
cs = pynhd.network_xsection(main_nhd, distance, width)

Let’s plot the obtained cross-section lines.

ax = cs.plot(color="r", figsize=(6, 6))
main.plot(ax=ax, color="b", linewidth=1, alpha=0.8)
flw.plot(ax=ax, color="b", linewidth=0.5, alpha=0.8)
ax.set_axis_off()
ax.figure.savefig(Path("_static", "cross_section.png"), dpi=300, bbox_inches="tight", facecolor="w")

Using the obtained cross-section lines, we can compute the cross-section profiles every 500 meters along the cross-section lines following the same procedure as before.

distance = 100  # in meters
cs["n_points"] = np.ceil(cs.length / distance).astype("int")
cs_points = [
    (
        n,
        list(
            zip(
                *ogc_utils.match_crs(
                    [
                        (p[0][0], p[1][0])
                        for p in (
                            l.interpolate(x, normalized=True).xy
                            for x in np.linspace(0, 1, pts, endpoint=False)
                        )
                    ],
                    crs_proj,
                    4326,
                )
            )
        ),
    )
    for n, l, pts in cs.itertuples(name=None)
]
cs_elevation = [
    (n, dem.interp(x=("z", list(pts[0])), y=("z", list(pts[1]))).values) for n, pts in cs_points
]

points = pd.DataFrame(((n, list(zip(*pts))) for n, pts in cs_points)).set_index(0)[1].explode()
points = pd.DataFrame([(n, Point(p)) for n, p in points.items()])
points = points.rename(columns={0: "comid", 1: "geometry"}).set_index("comid")
points = gpd.GeoDataFrame(points, crs=4326).to_crs(crs_proj)
elevations = pd.DataFrame(cs_elevation).rename(columns={0: "comid", 1: "elevation"})
elevations = elevations.set_index("comid")["elevation"].explode().astype("float")
points["elevation"] = elevations

ax = basin.to_crs(crs_proj).plot(figsize=(6, 6), facecolor="none", edgecolor="black")
main.plot(ax=ax, color="b", linewidth=0.8, alpha=0.8)
points.plot(ax=ax, column="elevation", markersize=6, legend=True, scheme="quantiles")
cs.plot(ax=ax, color="b", linewidth=0.8, alpha=0.8)
ax.set_axis_off()

This page was generated from dam_impact.ipynb. Interactive online version:

Working with Geospatial Hydrologic Data Using Web Services

Dam Impact on streamflow: Python

Author	Affiliation
Taher Chegini	University of Houston
Mike Johnson	Lynker
Marc Weber	US Environmental Protection Agency

Introduction

This is the second part of the “Working with Geospatial Hydrologic Data Using Web Services” workshop. In the first part, we covered some basic concepts related to geospatial analysis. Although R is used in the first part, the introduced concepts are applicable to Python as well. So, in this tutorial, we won’t repeat the same concepts, instead we focus on equivalent tools in Python and provide an in-depth hands-on example geospatial analysis workflow.

Geospatial Tools in Python

This is non-exhaustive list of geospatial tools that we directly/indirectly use in this tutorial:

• PyNHD: Navigate and subset NHDPlus (MR and HR) dataset using web services.

• Py3DEP: Access topographic data through National Map’s 3DEP web service.

• PyGeoHydro: Access NWIS, NID, WQP, HCDN 2009, NLCD, CAMELS, and SSEBop databases.

• PyDaymet: Access Daymet for daily climate data both single pixel and gridded.

• xarray: An open-source project and Python package that makes working with labeled multi-dimensional arrays simple, efficient, and fun!

• rioxarray: Rasterio xarray extension.

• GeoPandas: An open-source project to make working with geospatial data in python easier.

• Folium: Makes it easy to visualize data that’s been manipulated in Python on an interactive leaflet map. It enables both the binding of data to a map for choropleth visualizations and passing rich vector/raster/HTML visualizations as markers on the map.

You can find more geospatial tools at Awesome Geospatial repository.

The primary goal of this hands-on tutorial is to introduce a handful of geospatial web services for conducting scientific studies. For this purpose, we’re going to take a look at the impact of building dams on their downstream streamflow. Here are some example peer-reviewed papers on this topic.

• Non-stationary flood frequency analysis in continental Spanish rivers, using climate and reservoir indices as external covariates

• Nonlinear Filtering Effects of Reservoirs on Flood Frequency Curves at the Regional Scale

We set the area of interest (AOI) to Texas and study dams that have been built in the 1995-2005 period.

Workflow:

• Get gauge stations within Texas that have enough streamflow data

• Get dams within Texas that have been built have been built between 1995-2005

• Filter out stations that don’t have any dams in their upstream up to 10 km

• Get streamflow data for the remaining stations

• Visually inspect the streamflow data for dam impacts

• Check land use/land cover type around the dams

Streamflow Gauges

First, we use the National Water Information System (NWIS) service to check streamflow data availability in our AOI and 10 years before and after the period of our study, i.e., the 1985-2015 period.

Let’s start by importing the required libraries.

from pathlib import Path

import geopandas as gpd
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

import pygeohydro as gh
import pygeoutils as geoutils
from pygeohydro import NID, NWIS
from pynhd import NLDI, WaterData, ZeroMatchedError

start = "1985-01-01"
end = "2015-01-01"
texas = gh.helpers.get_us_states("TX")

_, ax = plt.subplots(dpi=100, figsize=(6, 6))
texas.plot(ax=ax, facecolor="b", edgecolor="r", linewidth=2, alpha=0.8, figsize=(10, 10))
ax.set_axis_off()

nwis = NWIS()
query = {
    "stateCd": "TX",
    "startDt": start,
    "endDt": end,
    "outputDataTypeCd": "dv",  # daily values
    "hasDataTypeCd": "dv",  # daily values
    "parameterCd": "00060",  # discharge
}
sites = nwis.get_info(query, nhd_info=True)
sites.shape[0]

/Users/tchegini/.local/apps/mambaforge/envs/hyriver-dev/lib/python3.11/site-packages/pynhd/pynhd.py:985: UserWarning: 95 of 3717 requests failed.. Indices of the failed requests are [735, 736, 737, 738, 739, 740, 741, 742, 743, 969, 970, 971, 972, 973, 974, 987, 988, 989, 990, 991, 992, 1570, 1571, 1572, 1573, 1574, 1575, 2395, 2396, 2397, 2398, 2399, 2400, 2401, 2402, 2403, 2404, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412, 2413, 2414, 2415, 2416, 2417, 2418, 2419, 2420, 2421, 2422, 2423, 2796, 2797, 2798, 2799, 2806, 2856, 2857, 2858, 2859, 2860, 2861, 2862, 2863, 2864, 2970, 2971, 2972, 2973, 2974, 2975, 2986, 2987, 2988, 2989, 2990, 2991, 3639, 3640, 3641, 3642, 3643, 3644, 3686, 3694, 3695, 3696, 3697, 3698, 3699]
  return self._get_urls(urls, False)

31039

We can see that there are 30604 streamflow gauge stations in Texas that fit our criteria.

_, ax = plt.subplots(dpi=100, figsize=(6, 6))
texas.plot(ax=ax, facecolor="b", edgecolor="r", linewidth=2, alpha=0.8, figsize=(10, 10))
sites.plot(ax=ax, marker="*", markersize=90, color="w", edgecolor="k", label="NWIS sites")
ax.legend(loc="upper left")
ax.set_axis_off()

Now, let’s filter these stations a step further to include only those stations that have around 30 years of daily streamflow data with drainage area of larger than 10 km\(^2\) that have been impacted by human activities.

sites = sites[
    (sites.parm_cd == "00060")  # discharge
    & (sites.stat_cd == "00003")  # mean
    & (sites.count_nu >= 30 * 365)  # at least 30 years of data
    & ~sites.hcdn_2009  # not pristine
    & (sites.nhd_areasqkm > 10)
].copy()
sites.shape[0]

2003

Upon applying this filter, we’re left with 280 stations.

_, ax = plt.subplots(dpi=100, figsize=(6, 6))
texas.plot(ax=ax, facecolor="b", edgecolor="r", linewidth=2, alpha=0.8, figsize=(10, 10))
sites.plot(ax=ax, marker="*", markersize=90, color="w", edgecolor="k", label="NWIS sites")
ax.legend(loc="upper left")
ax.set_axis_off()

Dams

Next, we need to retrieve the dams in Texas that have been built between 1995 and 2005. We use National Inventory of Dams (NID) web service for this purpose. First, let’s check out the fields that NID exposes through their web service.

nid = NID()
nid.valid_fields[nid.valid_fields.str.contains("year", case=False)]

55                yearCompleted
56              yearCompletedId
57                yearsModified
178        hydroApprovedIdfYear
183    hydroOrigIdfApprovedYear
188        hydroPmfApprovedYear
194           hydroOtAepDevYear
200            hydroTopInfoYear
Name: name, dtype: object

nid.valid_fields[nid.valid_fields.str.contains("state", case=False)]

16                stateFedId
19                     state
21               countyState
27          stateRegulatedId
30     stateRegulatoryAgency
91                 cityState
250                 stateKey
Name: name, dtype: object

Based on these, we should use the stateKey and yearCompleted fields.

dam_list = nid.get_byfilter(
    [
        {
            "stateKey": ["TX"],
            "yearCompleted": ["[1995 2005]"],
        },
    ],
)
dams = dam_list[0]

_, ax = plt.subplots(dpi=100, figsize=(6, 6))
texas.plot(ax=ax, facecolor="b", edgecolor="r", linewidth=2, alpha=0.8, figsize=(10, 10))
sites.plot(ax=ax, marker="*", markersize=110, color="w", edgecolor="k", label="NWIS sites")
dams.plot(ax=ax, marker="o", markersize=20, color="g", edgecolor="k", label="NID dams")
ax.legend(loc="upper left")
ax.set_axis_off()

As is evident from the plot above, there are many stations that don’t have any dams in their vicinity. One way to eliminate these stations is using a spatial query based on a search radius. We can determine an estimation for our search radius based on the upstream drainage area distribution of the streamflow gauges.

ax = sites.hist("nhd_areasqkm", figsize=(4, 2.5), bins="auto")
_ = ax[0][0].set_title("Drainage Area (km$^2$) Histogram")

We can see that most stations have a drainage area of less than 15000 km\(^2\). Since they’re not very large a search radius of 10 km should be sufficient. Now, we define a function that carries out an efficient spatial query to find the stations that have at least one dam within a 10-km radius.

def distance_filter(gdf1, gdf2, dist_km):
    """Filter gdf1 to only include points within distance of gdf2."""
    buff = gdf1.to_crs(5070).buffer(dist_km * 1e3)
    idx2, idx1 = buff.sindex.query(gdf2.to_crs(5070).geometry)
    return gdf1.iloc[pd.unique(idx1)].reset_index(drop=True), gdf2.iloc[
        pd.unique(idx2)
    ].reset_index(drop=True)

sites, dams = distance_filter(sites, dams, 10)

ax = sites.hist("nhd_areasqkm", figsize=(4, 2.5), bins="auto")
_ = ax[0][0].set_title("Drainage Area (km$^2$) Histogram")

_, ax = plt.subplots(dpi=100, figsize=(6, 6))
texas.plot(ax=ax, facecolor="b", edgecolor="r", linewidth=2, alpha=0.8, figsize=(10, 10))
sites.plot(ax=ax, marker="*", markersize=110, color="w", edgecolor="k", label="NWIS sites")
dams.plot(ax=ax, marker="o", markersize=20, color="g", edgecolor="k", label="NID dams")
ax.legend(loc="upper left")
ax.set_axis_off()

Flowlines and Hydrolinking

So far, we obtained only the stations that have at least one dam in their 10-km radius, but we didn’t find if those dams are in their upstream or downstream. We use the Hydro Network-Linked Data Index (NLDI) web service to obtain the upstream flowlines of the streamflow gauges up to 10 km. Note that this 10 km is the distance along the flowlines.

nldi = NLDI()

flw_up = {}
noflw = []
for agency, fid in sites[["agency_cd", "site_no"]].itertuples(index=False, name=None):
    try:
        flw_up[fid] = nldi.navigate_byid(
            fsource="nwissite",
            fid=f"{agency}-{fid}",
            navigation="upstreamTributaries",
            source="flowlines",
            distance=10,
        )
    except ZeroMatchedError:
        noflw.append(fid)
noflw

[]

sites = sites[sites.site_no.isin(flw_up)].copy()
sites, dams = distance_filter(sites, dams, 10)

We now have flowlines and dams that are closer than 10 km, but we still don’t have a way of linking these features. For this purpose, we use NHDPlus’s Common Identifier (comid). One way to obtain the first downstream flowline of coordinate is to use NLDI’s hydrolocation endpoint.

coords = list(dams[["longitude", "latitude"]].astype(float).itertuples(index=False, name=None))
nid_comids = nldi.feature_byloc(coords)
dams["nhdplus_comid"] = nid_comids.comid.values

coords = list(sites[["dec_long_va", "dec_lat_va"]].astype(float).itertuples(index=False, name=None))
site_comids = nldi.feature_byloc(coords)
sites["nhdplus_comid"] = site_comids.comid.values

We can then use these obtained comids to hydrolink the dams, stations, and flowlines.

flw_all = pd.concat(flw_up.values(), keys=flw_up.keys()).reset_index().drop(columns="level_1")
flw_all = gpd.GeoDataFrame(
    flw_all.rename(columns={"level_0": "usgs_id"}), crs=flw_up["07312200"].crs
)
flw_all.head()

	usgs_id	geometry	nhdplus_comid
0	08044000	LINESTRING (-97.69269 33.23604, -97.69307 33.2...	1305635
1	08044000	LINESTRING (-97.69154 33.23755, -97.69228 33.2...	1305593
2	08044000	LINESTRING (-97.71624 33.23529, -97.71599 33.2...	1305595
3	08044000	LINESTRING (-97.69438 33.24547, -97.69389 33.2...	1305591
4	08044000	LINESTRING (-97.69002 33.24999, -97.68994 33.2...	1305589

Finally, upon hydrolinking these features we can determine the stations that have at least one dam in their upstream up to 10 km.

flw_dam = pd.merge(
    flw_all[["usgs_id", "nhdplus_comid"]],
    dams[["federalId", "nhdplus_comid"]].rename(columns={"federalId": "dam_id"}),
    on="nhdplus_comid",
    how="left",
)
flw_dam = flw_dam[~flw_dam.dam_id.isna()].reset_index(drop=True)
flw_dam

	usgs_id	nhdplus_comid	dam_id
0	08158920	5781345	TX07494
1	08158920	5781345	TX07223
2	08095000	5531532	TX07074
3	08095000	5531532	TX07061
4	08178700	10839872	TX07211
5	08057445	1260363	TX09216
6	08177300	1638809	TX07200
7	07312200	13730353	TX07042

site_dam = flw_dam.usgs_id.unique()
site_dam

array(['08158920', '08095000', '08178700', '08057445', '08177300',
       '07312200'], dtype=object)

Streamflow Observations

Upon finalizing the stations that satisfy our criteria, we use NWIS to obtain the daily streamflow data. But first, we need to obtain the year that the construction of dams were completed, so we can exactly specify the required streamflow data period.

dams_info = nid.inventory_byid(flw_dam.dam_id.to_list())
dams_info.head()

	associatedStructuresCount	conditionAssessDate	conditionAssessDetail	conditionAssessId	damHeight	damLength	distance	drainageArea	eapId	dsacAssignedDate	...	purposeIds	dsacId	femaRegion	lastEapExcerDate	dataUpdated	usaceDistrict	noConsequencesId	secretaryAgricultureBuiltId	nrcsWatershedAuthorizationId	geometry
id
544499	0	2014-06-18T12:00:00.000Z	NOT RATED	5	22.0	0	NaN	1.13	3	None	...	None	None	6	None	2023-09-18T12:00:00.000Z	Fort Worth District	0	0	5	POINT (-97.37830 28.77720)
543218	0	2014-06-18T12:00:00.000Z	FAIR	2	12.0	300	NaN	0.90	1	None	...	9	None	6	None	2023-09-18T12:00:00.000Z	Fort Worth District	0	0	5	POINT (-96.66611 32.73923)
543202	0	2022-11-01T12:00:00.000Z	FAIR	5	28.1	825	NaN	NaN	2	None	...	None	None	6	None	2023-09-18T12:00:00.000Z	Galveston District	0	0	5	POINT (-97.86973 30.24981)
542209	0	2021-04-30T12:00:00.000Z	SATISFACTORY	2	49.0	6615	0.0	16.72	1	None	...	9	None	6	None	2023-09-18T12:00:00.000Z	Galveston District	0	1	3	POINT (-98.45066 29.55025)
542221	0	2016-10-13T12:00:00.000Z	POOR	3	26.0	730	0.0	1.34	1	None	...	9	None	6	None	2023-09-18T12:00:00.000Z	Galveston District	0	0	5	POINT (-97.89934 30.24614)

5 rows × 96 columns

We use the yearCompleted field to determine the required study period by looking at 15 years before and after the year completed for each dam. The reason that we opted for 15 years is the fact that during the construction the streamflow is temporarily diverted from its original path and also upon completion it usually takes some time for dams to be filled and reach their operational capacity. So we consider a 5-year buffer around the year completed.

flw_dam = pd.merge(
    flw_dam,
    dams_info[["federalId", "nidStorage", "nidHeight", "yearCompleted"]].rename(
        columns={"federalId": "dam_id"}
    ),
    on="dam_id",
    how="left",
)
flw_dam["start"] = pd.to_datetime(flw_dam.yearCompleted - 15, format="%Y")
flw_dam["end"] = pd.to_datetime(flw_dam.yearCompleted + 15, format="%Y") + pd.DateOffset(
    years=1, days=-1
)
flw_dam.head()

	usgs_id	nhdplus_comid	dam_id	nidStorage	nidHeight	yearCompleted	start	end
0	08158920	5781345	TX07494	189.00	28.1	1996	1981-01-01	2011-12-31
1	08158920	5781345	TX07223	156.31	26.0	1999	1984-01-01	2014-12-31
2	08095000	5531532	TX07074	731.00	73.0	2000	1985-01-01	2015-12-31
3	08095000	5531532	TX07061	410.00	22.6	1998	1983-01-01	2013-12-31
4	08178700	10839872	TX07211	8741.00	53.0	2004	1989-01-01	2019-12-31

We then use these dates to obtain the streamflow.

dates = (flw_dam.start.min(), flw_dam.end.max())
qobs = nwis.get_streamflow(site_dam, dates)

Now, let’s take a look at the data and assess the station by visual inspection. Note the there are several methods for quantifying the impact of dams on streamflow such as Reservoir Index, but in this tutorial we just rely on visual inspection for brevity.

ma = qobs.groupby(pd.Grouper(freq="Y")).mean()
yc = flw_dam.set_index("usgs_id").yearCompleted
fig, axs = plt.subplots(
    nrows=ma.shape[1], figsize=(8, 10), sharex=True, constrained_layout=True, dpi=100
)

for i, s in enumerate(ma):
    (line1,) = axs[i].plot(ma[s], label="Mean annual flow (cms)")
    axs[i].set_ylim(0, ma[s].quantile(0.95) * 1.7)
    axs[i].set_title(s)
    yrs = yc.loc[s.replace("USGS-", "")]
    yrs = [yrs] if isinstance(yrs, np.int64) else yrs
    for yr in yrs:
        line2 = axs[i].axvline(
            pd.to_datetime(yr, format="%Y"),
            color="r",
            linestyle="--",
            linewidth=2,
            label="Dam Year Completed",
        )
_ = axs[0].legend(handles=[line1, line2], loc="best")

We can see that based on the available data and visual inspection, the first station shows a noticeable difference before and after the dam construction. Next we take a closer look at this station, USGS-07312200.

Land Use/Land Cover (LULC)

To gain a better understanding of the land features around and near the USGS-07312200 station and its dams, we use the LULC data. For this purpose, first, we obtain the local catchments up to 30 km upstream and downstream of the station. The WaterData web service has several layers one of which is local catchments. Let’s use the COMIDs to obtain their corresponding catchments.

station_id = "07312200"
flw_ut = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="flowlines",
    distance=30,
)
flw_dd = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="downstreamDiversions",
    source="flowlines",
    distance=30,
)
flowlines = gpd.GeoDataFrame(pd.concat([flw_ut, flw_dd]), crs=4326)
wd = WaterData("catchmentsp")
catchment = wd.byid("featureid", flowlines.nhdplus_comid.to_list())
flw_dd["nhdplus_comid"] = flw_dd.nhdplus_comid.astype(catchment.featureid.dtype)
catchment["direction"] = "upstream"
catchment.loc[catchment.featureid.isin(flw_dd.nhdplus_comid), "direction"] = "downstream"
catchment.head()

	geometry	gridcode	featureid	sourcefc	areasqkm	shape_length	shape_area	direction
0	MULTIPOLYGON (((-98.95702 34.00904, -98.95757 ...	1347768	13728581	NHDFlowline	3.667051	0.082690	3.579443e-04	upstream
1	MULTIPOLYGON (((-98.95302 34.00727, -98.95292 ...	1347933	13728583	NHDFlowline	0.005346	0.003188	5.217419e-07	upstream
2	MULTIPOLYGON (((-98.95483 34.00319, -98.95537 ...	1348044	13728585	NHDFlowline	0.150286	0.018680	1.466643e-05	upstream
3	MULTIPOLYGON (((-98.94654 34.00018, -98.94817 ...	1347921	13728587	NHDFlowline	0.010543	0.004362	1.028886e-06	upstream
4	MULTIPOLYGON (((-98.94831 34.00390, -98.94784 ...	1347981	13728589	NHDFlowline	0.142392	0.027063	1.389548e-05	upstream

_, ax = plt.subplots(dpi=100, figsize=(6, 6))
catchment.plot(ax=ax, column="direction", edgecolor="k", lw=0.5)
flowlines.plot(ax=ax, color="b", linewidth=1.5, zorder=1)
sites[sites.site_no.isin([station_id])].plot(
    ax=ax, edgecolor="k", c="r", marker="*", markersize=150, label="Station"
)
dams[dams.nhdplus_comid.isin(flowlines.nhdplus_comid)].plot(
    ax=ax, edgecolor="k", c="m", marker="^", markersize=50, label="Dam"
)
ax.legend(loc="upper left")
ax.set_axis_off()

Now, we can retrieve the LULC data at 30-m resolution within the union of these catchments using the Multi-Resolution Land Characteristics (MRLC) Consortium web service.

geom = gpd.GeoSeries([catchment.unary_union], crs=catchment.crs, index=[station_id])
nlcd = gh.nlcd_bygeom(geom, 30, years={"cover": 2019})[station_id]
nlcd

<xarray.Dataset>
Dimensions:      (x: 1258, y: 804)
Coordinates:
  * x            (x) float64 -99.12 -99.12 -99.12 ... -98.71 -98.71 -98.71
  * y            (y) float64 34.04 34.04 34.04 34.04 ... 33.82 33.82 33.82 33.82
    spatial_ref  int64 0
Data variables:
    cover_2019   (y, x) uint8 127 127 127 127 127 127 ... 127 127 127 127 127
Attributes:
    AREA_OR_POINT:           Area
    TIFFTAG_RESOLUTIONUNIT:  1 (unitless)
    TIFFTAG_XRESOLUTION:     1
    TIFFTAG_YRESOLUTION:     1
    scale_factor:            1.0
    add_offset:              0.0
    _FillValue:              255
    nodatavals:              (255,)

xarray.Dataset

• Dimensions:
  • x: 1258
  • y: 804
• Coordinates: (3)
  • x
    (x)
    float64
    -99.12 -99.12 ... -98.71 -98.71

    axis :

    X

    long_name :

    longitude

    standard_name :

    longitude

    units :

    degrees_east

    array([-99.119952, -99.119628, -99.119304, ..., -98.71327 , -98.712946,
           -98.712622])

  • y
    (y)
    float64
    34.04 34.04 34.04 ... 33.82 33.82

    axis :

    Y

    long_name :

    latitude

    standard_name :

    latitude

    units :

    degrees_north

    array([34.039062, 34.038791, 34.038521, ..., 33.822522, 33.822252, 33.821981])

  • spatial_ref
    ()
    int64
    0

    crs_wkt :

    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]

    semi_major_axis :

    6378137.0

    semi_minor_axis :

    6356752.314245179

    inverse_flattening :

    298.257223563

    reference_ellipsoid_name :

    WGS 84

    longitude_of_prime_meridian :

    0.0

    prime_meridian_name :

    Greenwich

    geographic_crs_name :

    WGS 84

    horizontal_datum_name :

    World Geodetic System 1984

    grid_mapping_name :

    latitude_longitude

    spatial_ref :

    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]

    GeoTransform :

    -99.120114 0.000324049284578708 0.0 34.039197 0.0 -0.0002703370646766173

    array(0)

• Data variables: (1)
  • cover_2019
    (y, x)
    uint8
    127 127 127 127 ... 127 127 127 127

    AREA_OR_POINT :

    Area

    TIFFTAG_RESOLUTIONUNIT :

    1 (unitless)

    TIFFTAG_XRESOLUTION :

    1

    TIFFTAG_YRESOLUTION :

    1

    scale_factor :

    1.0

    add_offset :

    0.0

    _FillValue :

    127

    nodatavals :

    (127,)

    units :

    classes

    array([[127, 127, 127, ..., 127, 127, 127],
           [127, 127, 127, ..., 127, 127, 127],
           [127, 127, 127, ..., 127, 127, 127],
           ...,
           [127, 127, 127, ..., 127, 127, 127],
           [127, 127, 127, ..., 127, 127, 127],
           [127, 127, 127, ..., 127, 127, 127]], dtype=uint8)

• Indexes: (2)
  • x
    PandasIndex

    PandasIndex(Index([-99.11995197535771, -99.11962792607314, -99.11930387678855,
           -99.11897982750398, -99.11865577821939, -99.11833172893482,
           -99.11800767965023, -99.11768363036566, -99.11735958108108,
            -99.1170355317965,
           ...
           -98.71553846820349,  -98.7152144189189, -98.71489036963433,
           -98.71456632034975, -98.71424227106517, -98.71391822178059,
           -98.71359417249602, -98.71327012321143, -98.71294607392686,
           -98.71262202464227],
          dtype='float64', name='x', length=1258))

  • y
    PandasIndex

    PandasIndex(Index([ 34.03906183146766, 34.038791494402986,  34.03852115733831,
            34.03825082027363, 34.037980483208955,  34.03771014614428,
             34.0374398090796, 34.037169472014924,  34.03689913495025,
            34.03662879788558,
           ...
           33.824414202114426,  33.82414386504975,  33.82387352798508,
             33.8236031909204, 33.823332853855725,  33.82306251679105,
            33.82279217972637,  33.82252184266169, 33.822251505597016,
            33.82198116853234],
          dtype='float64', name='y', length=804))

• Attributes: (8)

  AREA_OR_POINT :

  Area

  TIFFTAG_RESOLUTIONUNIT :

  1 (unitless)

  TIFFTAG_XRESOLUTION :

  1

  TIFFTAG_YRESOLUTION :

  1

  scale_factor :

  1.0

  add_offset :

  0.0

  _FillValue :

  255

  nodatavals :

  (255,)

meta = gh.helpers.nlcd_helper()
pd.Series(meta["classes"])

11     Open Water - All areas of open water, generall...
12     Perennial Ice/Snow - All areas characterized b...
21     Developed, Open Space - Includes areas with a ...
22     Developed, Low Intensity -Includes areas with ...
23     Developed, Medium Intensity - Includes areas w...
24     Developed, High Intensity - Includes highly de...
31     Barren Land (Rock/Sand/Clay) - Barren areas of...
41     Deciduous Forest - Areas dominated by trees ge...
42     Evergreen Forest - Areas dominated by trees ge...
43     Mixed Forest - Areas dominated by trees genera...
45     Shrub-Forest - Areas identified as currently s...
46     Herbaceous-Forest - Areas identified as curren...
51     Dwarf Scrub - Alaska only areas dominated by s...
52     Shrub/Scrub - Areas dominated by shrubs; less ...
71     Grassland/Herbaceous - Areas dominated by gram...
72     Sedge/Herbaceous - Alaska only areas dominated...
73     Lichens - Alaska only areas dominated by fruti...
74     Moss - Alaska only areas dominated by mosses, ...
81     Pasture/Hay - Areas of grasses, legumes, or gr...
82     Cultivated Crops - Areas used for the producti...
90     Woody Wetlands - Areas where forest or shrub l...
95     Emergent Herbaceous Wetlands - Areas where per...
127                                     Background value
dtype: object

cmap, norm, levels = gh.plot.cover_legends()
fig, ax = plt.subplots(figsize=(8, 4), dpi=100)
cover = nlcd.cover_2019.where(nlcd.cover_2019 != nlcd.cover_2019.rio.nodata)
cover.plot(ax=ax, cmap=cmap, levels=levels, cbar_kwargs={"ticks": levels[:-1]})
ax.set_xlabel("")
ax.set_ylabel("")
ax.set_title("Land Use/Land Cover 2019")
dams[dams.nhdplus_comid.isin(flw_up[station_id].nhdplus_comid)].plot(
    ax=ax, edgecolor="k", c="m", marker="^", markersize=50, label="Dam"
)
ax.legend(loc="upper left")
ax.set_axis_off()

fig.savefig(Path("_static", "dam_impact.png"), bbox_inches="tight", facecolor="w")

Now, let’s take a look at the differences in land cover types, upstream and downstream of the station. We use cover_statistics function of PyGeoHydro to get the statistics for each land cover types. For this purpose, first we need to mask the NLCD dataset based on the direction attribute of the catchment geo-dataframe using xarray_geomask function from PyGeoUtils.

ut_geom = catchment[catchment.direction == "upstream"].unary_union
ut_cover = gh.cover_statistics(geoutils.xarray_geomask(nlcd.cover_2019, ut_geom, catchment.crs))

dd_geom = catchment[catchment.direction == "downstream"].unary_union
dd_cover = gh.cover_statistics(geoutils.xarray_geomask(nlcd.cover_2019, dd_geom, catchment.crs))

stats = pd.DataFrame([ut_cover.categories, dd_cover.categories]).T
stats.columns = ["Upstream", "Downstream"]
stats["Upstream - Downstream"] = stats.Upstream - stats.Downstream
stats.round(1)

	Upstream	Downstream	Upstream - Downstream
Water	1.1	0.4	0.7
Developed	1.4	3.6	-2.2
Barren	1.6	0.4	1.2
Forest	0.8	7.6	-6.8
Shrubland	28.9	12.9	16.0
Herbaceous	62.5	50.4	12.1
Planted/Cultivated	3.6	24.4	-20.8
Wetlands	0.1	0.3	-0.2

Notably we see that the upstream area has significantly more grassland than downstream, while the downstream catchments have significantly more cropland. The implication is that these dams were constructed to support the transition of grassland to agricultural land.

Final Remarks

This tutorial demonstrated the application of various web services for obtaining hydrolinked geospatial data using only web services. These web services are powerful tools that make big geospatial data more equitable and the rich ecosystem of tools that are available in R and Python facilitate accessing them.

This page was generated from daymet.ipynb. Interactive online version:

Climate Data from Daymet

from pathlib import Path

import matplotlib.pyplot as plt

import pydaymet as daymet
from pynhd import NLDI

The Daymet database provides climatology data at 1-km resolution. First, we use PyNHD to get the contributing watershed geometry of a NWIS station with the ID of USGS-01031500:

geometry = NLDI().get_basins("01031500").geometry[0]

PyDaymet allows us to get the data for a single pixel or for a region as gridded data. The function to get single pixel is called pydaymet.get_bycoords and for gridded data is called pydaymet.get_bygeom. The arguments of these functions are identical except the first argument where the latter should be polygon and the former should be a coordinate (a tuple of length two as in (x, y)).

The input geometry or coordinate can be in any valid CRS (defaults to EPSG:4326). The date argument can be either a tuple of length two like (start_str, end_str) or a list of years like [2000, 2005].

Additionally, we can pass time_scale to get daily, monthly or annual summaries. This flag by default is set to daily. We can pass time_scale as daily, monthly, or annual to get_bygeom or get_bycoords functions to download the respective summaries.

PyDaymet can also compute Potential EvapoTranspiration (PET) at daily timescale using three methods: penman_monteith, priestley_taylor, and hargreaves_samani. Let’s get the data for six days and plot PET.

dates = ("2000-01-01", "2000-01-06")
daily = daymet.get_bygeom(geometry, dates, variables="prcp", pet="hargreaves_samani")

ax = daily.pet.plot(x="x", y="y", row="time", col_wrap=3)
ax.fig.savefig(Path("_static", "daymet_grid.png"), facecolor="w", bbox_inches="tight")

Now, let’s get the monthly summaries for six months.

var = ["prcp", "tmin"]
dates = ("2000-01-01", "2000-06-30")
monthly = daymet.get_bygeom(geometry, dates, variables=var, time_scale="monthly")

ax = monthly.prcp.plot(x="x", y="y", row="time", col_wrap=3)

Note that the default CRS is EPSG:4326. If the input geometry (or coordinate) is in a different CRS we can pass it to the function. The gridded data are automatically masked to the input geometry. Now, Let’s get the data for a coordinate in EPSG:3542 CRS.

coords = (-1431147.7928, 318483.4618)
crs = "epsg:3542"
dates = ("2000-01-01", "2006-12-31")
annual = daymet.get_bycoords(
    coords, dates, variables=["prcp", "tmin"], crs=crs, time_scale="annual"
)

fig = plt.figure(figsize=(6, 4), facecolor="w")

gs = fig.add_gridspec(1, 2)
axes = gs[:].subgridspec(2, 1, hspace=0).subplots(sharex=True)
annual["tmin (degrees C)"].plot(ax=axes[0], color="r")
axes[0].set_ylabel(r"$T_{min}$ ($^\circ$C)")
axes[0].xaxis.set_ticks_position("none")
annual["prcp (mm/day)"].plot(ax=axes[1])
axes[1].set_ylabel("$P$ (mm/day)")

plt.tight_layout()
fig.savefig("_static/daymet_loc.png", facecolor="w", bbox_inches="tight")

Next, let’s get annual total precipitation for Hawaii and Puerto Rico for 2010.

hi_ext = (-160.3055, 17.9539, -154.7715, 23.5186)
pr_ext = (-67.9927, 16.8443, -64.1195, 19.9381)
hi = daymet.get_bygeom(hi_ext, 2010, variables="prcp", region="hi", time_scale="annual")
pr = daymet.get_bygeom(pr_ext, 2010, variables="prcp", region="pr", time_scale="annual")

ax = hi.prcp.plot(size=4)
plt.title("Hawaii, 2010")
ax.figure.savefig("_static/hi.png", facecolor="w", bbox_inches="tight")

ax = pr.prcp.plot(size=4)
plt.title("Puerto Rico, 2010")
ax.figure.savefig("_static/pr.png", facecolor="w", bbox_inches="tight")

This page was generated from flood_peak.ipynb. Interactive online version:

Flood Moments for CAMELS

In this tutorial, we learn about computing three statistical flood moments, namely, mean annual flood (MAF), coefficient of variation (CV), and skewness (CS) for all 671 USGS gage stations that are included in the CAMELS dataset.

from pathlib import Path

import cytoolz as tlz
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

import pygeohydro as gh

First, we use pygeohydro.get_camels to get the CAMELS dataset. This function returns one geopandas.GeoDataFrame containing basin geometries of the stations and 59 other basin properties, and a xarray.Dataset containing streamflow values and the same 59 basin properties. Essentially, the difference between the two is that one has basin geometries and the other has streamflows.

camels_basins, camels_qobs = gh.get_camels()

Now, let’s extract the station IDs, their drainage area, and start and end dates of the streamflows.

area_sqkm = camels_qobs.area_geospa_fabric.to_series()
start = camels_qobs.time[0].dt.strftime("%Y-%m-%d").item()
end = camels_qobs.time[-1].dt.strftime("%Y-%m-%d").item()
station_ids = camels_basins.index

Next, instead of using mean daily streamflow data from the CAMELS dataset, we use NWIS’s peak streamflow service. The pygeohydro.NWIS class has a method called retrieve_rdb for pulling data from NWIS services in the RDB format. This function is very handy for sending requests to NWIS and processing the responses to a pandas.DataFrame.

The peak streamflow service accepts multiple station IDs per request by using multiple_site_no keyword. However, since the service uses the GET request method, there’s a limit on the number of characters per request. Thus, we need to partition our station IDs into smaller chunks per request. We use cytoolz library for this purpose.

nwis = gh.NWIS()

base_url = "https://nwis.waterdata.usgs.gov/nwis/peak"
payloads = [
    {
        "multiple_site_no": ",".join(sid),
        "search_site_no_match_type": "exact",
        "group_key": "NONE",
        "sitefile_output_format": "html_table",
        "column_name": "station_nm",
        "begin_date": start,
        "end_date": end,
        "date_format": "YYYY-MM-DD",
        "rdb_compression": "file",
        "hn2_compression": "file",
        "list_of_search_criteria": "multiple_site_no",
    }
    for sid in tlz.partition_all(100, station_ids)
]
peak = nwis.retrieve_rdb(base_url, payloads)[["site_no", "peak_dt", "peak_va"]]

Next, we extract year from the peak_dt column and covert peak_va to float. Also, peak_va is in cubic feet per second and for computing the flood moments we need to convert it into cubic meter per second per square km using the stations’ upstream drainage area.

peak["year"] = peak.peak_dt.str.split("-").str[0].astype(int)
peak["peak_va"] = pd.to_numeric(peak.peak_va, errors="coerce")
peak = peak.groupby(["year", "site_no"]).peak_va.max()
peak = peak.unstack(level=1) * 0.3048**3 / area_sqkm

Since there are missing values in our data let’s use CAMEL’s dataset to fill those.

print(
    "No. of missing values in data from Peak Streamflow service: ",
    peak.isna().sum().sum(),
)

No. of missing values in data from Peak Streamflow service:  3440

camels_peak = camels_qobs.discharge.to_series()
camels_peak = (
    camels_peak.reset_index().set_index("time").groupby(["station_id", pd.Grouper(freq="Y")]).max()
)
camels_peak = camels_peak.unstack(level=0)
camels_peak.index = camels_peak.index.year
camels_peak.columns = camels_peak.columns.droplevel(0)
camels_peak = camels_peak * 0.3048**3 / area_sqkm

print(
    "No. of missing values in data from CAMELS dataset: ",
    camels_peak.isna().sum().sum(),
)

No. of missing values in data from CAMELS dataset:  463

peak = peak.fillna(camels_peak)

print(
    "No. of missing values after the filling operation: ",
    peak.isna().sum().sum(),
)

No. of missing values after the filling operation:  392

The peak streamflow data is now ready to be used for computing the flood moments.

def compute_flood_moments(annual_peak: pd.DataFrame) -> pd.DataFrame:
    """Compute flood moments (MAF, CV, CS) from annual peak discharge values."""
    maf = annual_peak.mean()
    n = annual_peak.shape[0]
    s2 = np.power(annual_peak - maf, 2).sum() / (n - 1)
    cv = np.sqrt(s2) / maf
    cs = n * np.power(annual_peak - maf, 3).sum() / ((n - 1) * (n - 2) * np.power(s2, 3.0 / 2.0))

    fm = pd.concat([maf, cv, cs], axis=1)
    fm.columns = ["MAF", "CV", "CS"]
    return fm


fm = compute_flood_moments(peak)

Lastly, let’s merge the obtained flood moments with the basin geometries for visualization.

basins = camels_basins.to_crs(5070).geometry.centroid.to_frame("geometry")
camels_fm = basins.merge(fm, left_index=True, right_index=True)

We can use the pygeohydro.helpers.get_us_states to get the CONUS states geometries.

conus = gh.helpers.get_us_states("contiguous").to_crs(camels_fm.crs)

fig, ax = plt.subplots(figsize=(8, 8), dpi=100)
ax.set_title("Mean Annual Flood for CAMELS' stations")
conus.plot(ax=ax, facecolor="none", edgecolor="black", linewidth=0.2)
camels_fm.plot(
    ax=ax,
    column="MAF",
    scheme="User_Defined",
    legend=True,
    cmap="viridis",
    alpha=0.7,
    classification_kwds={"bins": [0.2, 0.4, 0.8]},
    legend_kwds={"title": "MAF", "ncols": 1},
)
ax.set_axis_off()
fig.savefig(Path("_static", "flood_peak.png"), dpi=300)

We can also use explore function of geopandas to plot an interactive map of the results.

def plot_flood_moments(column):
    """Plot flood moments (MAF, CV, CS) from annual peak discharge values."""
    vmin, vmax = camels_fm[column].quantile([0.1, 0.9])
    return camels_fm.explore(
        column=column,
        vmin=vmin,
        vmax=vmax,
        cmap="viridis",
        marker_kwds={"radius": 4},
        style_kwds={"stroke": False},
    )

plot_flood_moments("CV")

Make this Notebook Trusted to load map: File -> Trust Notebook

plot_flood_moments("CS")

Make this Notebook Trusted to load map: File -> Trust Notebook

This page was generated from geoconnex.ipynb. Interactive online version:

GeoConnex

from pathlib import Path

from pynhd import GeoConnex

GeoConnex is a web service developed by Internet Of Water that provides access to many hydro-linked datasets. PyNHD provides access to all GeoConnex endpoints.

To explore available datasets in GeoConnex (endpoints), we start by instantiating a pynhd.GeoConnex class.

gcx = GeoConnex(dev=False)
print("\n".join(f"{n}: {e.description}" for n, e in gcx.endpoints.items()))

hu02: Two-digit Hydrologic Regions
hu04: Four-digit Hydrologic Subregion
hu06: Six-digit Hydrologic Basins
hu08: Eight-digit Hydrologic Subbasins
hu10: Ten-digit Watersheds
nat_aq: National Aquifers of the United States from USGS National Water Information System National Aquifer code list.
principal_aq: Principal Aquifers of the United States from 2003 USGS data release
sec_hydrg_reg: Secondary Hydrogeologic Regions of the Conterminous United States from 2018 USGS data release
gages: US Reference Stream Gage Monitoring Locations
mainstems: US Reference Mainstem Rivers
states: U.S. States
counties: U.S. Counties
aiannh: Native American Lands
cbsa: U.S. Metropolitan and Micropolitan Statistical Areas
ua10: Urbanized Areas and Urban Clusters (2010 Census)
places: U.S. legally incororated and Census designated places
pws: U.S. Public Water Systems
dams: US Reference Dams

pynhd.GeoConnex provides three ways of querying GeoConnex: Spatial query, by ID, and using CQL filters. CQL filter is powerful and can be used for all sort of queries, however, using them require some knowledge of CQL filter which you can learn from here. For example, let’s say that we want to find all urban clusters in the US that have water area of larger than 100 km\(^2\). We can use the following CQL filter to do so:

gcx.item = "ua10"
awa = gcx.bycql({"gt": [{"property": "awater10"}, 100e6]})
awa.name10

0                Boston, MA--NH--RI
1                    Cape Coral, FL
2                   Chicago, IL--IN
3                  Jacksonville, FL
4                         Miami, FL
5     Minneapolis--St. Paul, MN--WI
6      New York--Newark, NY--NJ--CT
7                       Orlando, FL
8           Palm Bay--Melbourne, FL
9      Philadelphia, PA--NJ--DE--MD
10          Sarasota--Bradenton, FL
11                      Seattle, WA
12        Tampa--St. Petersburg, FL
13               Virginia Beach, VA
Name: name10, dtype: object

For demonstrating spatial query and query by ID, let’s say that we are interested in getting all mainstems within HUC 02 (Mid Atlantic Region). We can start by getting HUC 02’s geometry and then carry out a spatial query to get all mainstems within HUC 02. For this purpose, first we select huc02 endpoint (or item) from GeoConnex and then use query to get the geometry of HUC 02. We can the queryable fields of this item by just printing the instance of GeoConnext class that we just created.

gcx.item = "hu02"
gcx

Item: 'hu02'
Description: Two-digit Hydrologic Regions
Queryable Fields: fid, uri, name, gnis_url, gnis_id, huc2, loaddate
Extent: (-170, 15, -51, 72)

Therefore, we should query huc2 field and pass 02 as the value.

h2 = gcx.byid("huc2", "02")
ax = h2.plot(figsize=(6, 6))
ax.set_axis_off()

With this geometry, we can query the mainstems that are within HUC 02:

gcx.item = "mainstems"
ms = gcx.bygeometry(h2.geometry.iloc[0], predicate="within", crs=h2.crs)
ax = h2.plot(figsize=(6, 6), facecolor="none", edgecolor="k")
ms.plot(ax=ax, color="b", lw=0.3)
ax.set_axis_off()

Next, let’s say that we need all dams that are located on the headwater flowline of the mainstems that we just obtained. We set the endpoint to dams and find its queryable field of that’s related to NHDPlusc2 ComIDs.

gcx.item = "dams"
gcx

Item: 'dams'
Description: US Reference Dams
Queryable Fields: fid, id, uri, name, description, subjectof, provider, provider_id, nhdpv2_comid, feature_data_source
Extent: (-170, 15, -51, 72)

dam = gcx.byid("nhdpv2_comid", ms.head_nhdpv2_comid)
ax = h2.plot(figsize=(6, 6), facecolor="none", edgecolor="k")
ms.plot(ax=ax, color="b", lw=0.3)
dam.plot(ax=ax, color="r", markersize=10, label="Dams")
ax.legend()
ax.set_axis_off()

Let’s find gage stations that located outlets of the mainstems. We use gages endpoint and find its queryable field that’s related to NHDPlusc2 ComIDs.

gcx.item = "gages"
gcx

Item: 'gages'
Description: US Reference Stream Gage Monitoring Locations
Queryable Fields: fid, id, uri, name, description, subjectof, provider, provider_id, nhdpv2_reachcode, nhdpv2_reach_measure, nhdpv2_comid, mainstem_uri
Extent: (-170, 15, -51, 72)

We can check out the data type of the nhdpv2_comid in this dataset using the endpoints property of the GeoConnex class.

gcx.endpoints["gages"].dtypes

{'fid': 'int64',
 'id': 'str',
 'uri': 'str',
 'name': 'str',
 'description': 'str',
 'subjectof': 'str',
 'provider': 'str',
 'provider_id': 'str',
 'nhdpv2_reachcode': 'str',
 'nhdpv2_reach_measure': 'f8',
 'nhdpv2_comid': 'f8',
 'mainstem_uri': 'str'}

We notice that nhdpv2_comid is a float and since in the mainstem dataset outlet_nhdpv2_comid is a URL, we need to extract the ComIDs values. First, we should find the outlet ComIDs of the mainstems that these dams are located on.

outlet_ids = ms.loc[ms["head_nhdpv2_comid"].isin(dam["nhdpv2_comid"]), "outlet_nhdpv2_comid"]
comids = outlet_ids.str.rsplit("/", n=1).str[-1].astype(float)
gages = gcx.byid("nhdpv2_comid", comids)
ax = h2.plot(figsize=(6, 6), facecolor="none", edgecolor="k")
ms.plot(ax=ax, color="b", lw=0.3)
dam.plot(ax=ax, color="r", markersize=10, label="Dams")
gages.plot(ax=ax, color="g", markersize=10, marker="^", label="Gages")
ax.legend()
ax.set_axis_off()

We notice that some dams don’t have any downstream gages, so we need to filter those out:

gage_comid = "https://geoconnex.us/nhdplusv2/comid/" + gages.nhdpv2_comid.astype("Int64").astype(
    str
)
cond = ms["head_nhdpv2_comid"].isin(dam["nhdpv2_comid"]) & ms["outlet_nhdpv2_comid"].isin(
    gage_comid
)
pair_ids = ms.loc[cond, ["head_nhdpv2_comid", "outlet_nhdpv2_comid"]]
dam_ids = dam[dam["nhdpv2_comid"].isin(pair_ids["head_nhdpv2_comid"])]
pair_ids["outlet_nhdpv2_comid"] = (
    pair_ids["outlet_nhdpv2_comid"].str.rsplit("/", n=1).str[-1].astype(float)
)
gage_ids = gages[gages["nhdpv2_comid"].isin(pair_ids["outlet_nhdpv2_comid"])]

We need to add a common column for dams and gages, so we can pair them later on, if needed.

gage_ids = gage_ids.set_index("nhdpv2_comid")
gage_ids["head_nhdpv2_comid"] = pair_ids.set_index("outlet_nhdpv2_comid")["head_nhdpv2_comid"]
gage_ids = gage_ids.reset_index()

ax = h2.plot(figsize=(6, 6), facecolor="none", edgecolor="k")
ms.plot(ax=ax, color="b", lw=0.3)
dam_ids.plot(ax=ax, color="r", markersize=10, label="Dams")
gage_ids.plot(ax=ax, color="g", markersize=10, marker="^", label="Gages")
ax.legend()
ax.set_axis_off()
ax.figure.savefig(Path("_static/geoconnex.png"), dpi=100, bbox_inches="tight")

This page was generated from gridmet.ipynb. Interactive online version:

Climate Data from GridMET

from pathlib import Path

import matplotlib.pyplot as plt
import pygridmet as gridmet
from pygridmet import GridMET

from pynhd import NLDI

The Daymet database provides climatology data at 1-km resolution. First, we use PyNHD to get the contributing watershed geometry of a NWIS station with the ID of USGS-01318500:

geometry = NLDI().get_basins("01318500").geometry[0]

PyGridMET allows us to get the data for a single pixel or for a region as gridded data. The function to get single pixel is called pygridmet.get_bycoords and for gridded data is called pygridmet.get_bygeom. The arguments of these functions are identical except the first argument where the latter should be polygon and the former should be a coordinate (a tuple of length two as in (x, y)).

The input geometry or coordinate can be in any valid CRS (defaults to EPSG:4326). The date argument can be either a tuple of length two like (start_str, end_str) or a list of years like [2000, 2005].

We can get a dataframe of available variables and their info by calling GridMET().gridmet_table.

GridMET().gridmet_table

	variable	abbr	long_name	units
0	Precipitation	pr	precipitation_amount	mm
1	Maximum Relative Humidity	rmax	daily_maximum_relative_humidity	%
2	Minimum Relative Humidity	rmin	daily_minimum_relative_humidity	%
3	Specific Humidity	sph	daily_mean_specific_humidity	kg/kg
4	Surface Radiation	srad	daily_mean_shortwave_radiation_at_surface	W/m2
5	Wind Direction	th	daily_mean_wind_direction	Degrees clockwise from north
6	Minimum Air Temperature	tmmn	daily_minimum_temperature	K
7	Maximum Air Temperature	tmmx	daily_maximum_temperature	K
8	Wind Speed	vs	daily_mean_wind_speed	m/s
9	Burning Index	bi	daily_mean_burning_index_g	-
10	Fuel Moisture (100-hr)	fm100	dead_fuel_moisture_100hr	%
11	Fuel Moisture (1000-hr)	fm1000	dead_fuel_moisture_1000hr	%
12	Energy Release Component	erc	daily_mean_energy_release_component-g	-
13	Reference Evapotranspiration (Alfalfa)	etr	daily_mean_reference_evapotranspiration_alfalfa	mm
14	Reference Evapotranspiration (Grass)	pet	daily_mean_reference_evapotranspiration_grass	mm
15	Vapor Pressure Deficit	vpd	daily_mean_vapor_pressure_deficit	kPa

dates = ("2000-01-01", "2000-01-06")
daily = gridmet.get_bygeom(geometry, dates, variables=["pr", "pet"])

ax = daily.where(daily.pet > 0).pet.plot(x="lon", y="lat", row="time", col_wrap=3)
ax.fig.savefig(Path("_static", "gridmet_grid.png"), facecolor="w", bbox_inches="tight")

Note that the default CRS is EPSG:4326. If the input geometry (or coordinate) is in a different CRS we can pass it to the function. The gridded data are automatically masked to the input geometry. Now, Let’s get the data for a coordinate in EPSG:3542 CRS.

coords = (-1431147.7928, 318483.4618)
crs = 3542
dates = ("2000-01-01", "2006-12-31")
clm = gridmet.get_bycoords(coords, dates, variables=["pr", "tmmn"], crs=crs)

fig = plt.figure(figsize=(6, 4), facecolor="w")

gs = fig.add_gridspec(1, 2)
axes = gs[:].subgridspec(2, 1, hspace=0).subplots(sharex=True)
clm["tmmn (K)"].plot(ax=axes[0], color="r")
axes[0].set_ylabel(r"$T_{min}$ ($^\circ$C)")
axes[0].xaxis.set_ticks_position("none")
clm["pr (mm)"].plot(ax=axes[1])
axes[1].set_ylabel("$P$ (mm/day)")

plt.tight_layout()
fig.savefig("_static/gridmet_loc.png", facecolor="w", bbox_inches="tight")

This page was generated from hwms.ipynb. Interactive online version:

USGS STN Flood Event Data: High Water Marks

The United States Geological Survey (USGS) operates a comprehensive flood event database known as the Short-Term Network (STN). The STN offers a user-friendly web front-end for easy access. For developers and scientists, there’s a RESTFul API available for more advanced queries and integrations. The STN offers access to a variety of data types including instruments, sites, high water marks, and peak summaries.

Focus: High Water Marks (HWMs)

In this notebook, we’ll delve into the specifics of high water marks (HWMs) within the STN database. Here’s what you can expect:

1. Data Dictionaries: Understand the structure and meaning of the data.

2. Data Retrieval: Learn how to fetch all available data by type.

3. Filtered Queries: Dive deeper with specific, filtered data requests.

4. Data Limitations: We’ll also touch upon some of the known limitations including inconsistent field names and possibility for user introduced errors.

Additional Resources:

For those interested in the methodology behind HWM collection, the USGS provides detailed resources: - Technical Guide on HWMs - High Water Marks & Flooding Overview - Video Guide: Interpreting High Water Marks

Let’s begin by importing necessary dependencies.

from pathlib import Path

import matplotlib.lines as mlines
import matplotlib.markers as mmarkers
import matplotlib.pyplot as plt
import pandas as pd

import pygeohydro as gh
from pygeohydro import STNFloodEventData

After importing, we can start with how we can obtain all the HWM data available in the database as a GeoDataFrame. We have two options for get STN data: STNFloodEventData class or stn_flood_event function. The stn_flood_event function can only pull either the information dataset about the supported data_types by STN as a pandas.DataFrame or a subset of the actual data for the supported STN data_types as a geopandas.GeoDataFrame. Moreover, the STNFloodEventData class provides access to some other data about the STN service such as data dictionary.

For example, we can get information about HWMS data either using STNFloodEventData.get_all_data("hwms") or gh.stn_flood_event("hwms").

hwm_all = STNFloodEventData.get_all_data(
    "hwms", as_list=False, async_retriever_kwargs={"disable": True, "max_workers": 6}
)
hwm_all.head()

	hwm_id	waterbody	site_id	event_id	hwm_type_id	hwm_quality_id	hwm_locationdescription	latitude_dd	longitude_dd	survey_date	...	survey_member_id	hwm_label	files	stillwater	peak_summary_id	last_updated	last_updated_by	uncertainty	hwm_uncertainty	geometry
0	13922	Swatara Creek	16106	123	1	1	HWM is located on inside of pavillion two	40.192896	-76.723080	2012-04-26T04:00:00	...	202.0	no_label	NaN	NaN	NaN	NaN	NaN	NaN	NaN	POINT (-76.72308 40.19290)
1	17897	Atlantic Ocean	19148	24	1	3	Mud line on bench rocks and plants near IBA Cl...	41.894148	-70.536629	2017-06-05T04:00:00	...	2.0	HWMMAPLY-402	NaN	0.0	NaN	NaN	NaN	NaN	NaN	POINT (-70.53663 41.89415)
2	19685	East Nishnabotna River	20005	168	5	6	U.S. Highway 34, seed line on flood wall (1 of...	41.026290	-95.243199	1998-07-28T05:00:00	...	1757.0	HWM U/S	NaN	1.0	3337.0	NaN	NaN	NaN	NaN	POINT (-95.24320 41.02629)
3	18530	Maquoketa River	19436	151	6	1	County Road X29/220th Avenue, southwest of Del...	42.410092	-91.363481	2010-07-30T05:00:00	...	1755.0	USGS HWM D/S	NaN	1.0	2710.0	NaN	NaN	NaN	NaN	POINT (-91.36348 42.41009)
4	19687	East Nishnabotna River	20005	168	2	6	U.S. Highway 34, debris line on guardrail (2 o...	41.026290	-95.243199	1998-07-28T05:00:00	...	1757.0	HWM U/S	NaN	1.0	3337.0	NaN	NaN	NaN	NaN	POINT (-95.24320 41.02629)

5 rows × 33 columns

hwm_all = gh.stn_flood_event("hwms")

print(f"There are {len(hwm_all)} HWMs in the database.")

There are 34722 HWMs in the database.

For an interactive map, we can use the explore method. There are at least 34,000 HWMs in the STN database scattered throughout the country. It’s important to note the possibility of outliers as this data is collected by people and liable to errors. Here, we plot a sample of 1000 HWMs.

hwm_all.sample(1000).explore(
    marker_kwds={"radius": 2},
    style_kwds={"stroke": False},
)

Make this Notebook Trusted to load map: File -> Trust Notebook

Next, we illustrate how a filtered query can be completed with the same HWM data. First we want to present what parameters are available to query. We can use the STNFloodEventData.hwms_query_params attribute for that.

STNFloodEventData.hwms_query_params

{'County',
 'EndDate',
 'Event',
 'EventStatus',
 'EventType',
 'StartDate',
 'States'}

We can a subset of HWMS data either using STNFloodEventData.get_filtered_data("hwms", query_params=query_params) or gh.stn_flood_event("hwms", query_params=query_params).

hwm_filtered = STNFloodEventData.get_filtered_data(
    "hwms",
    crs="ESRI:102003",
    async_retriever_kwargs={"disable": True},
    query_params={"States": "SC,NC"},
)
hwm_filtered.head()

	latitude	longitude	eventName	hwmTypeName	hwmQualityName	verticalDatumName	verticalMethodName	approvalMember	markerName	horizontalMethodName	...	hdatum_id	flag_member_id	survey_member_id	uncertainty	hwm_label	files	height_above_gnd	hwm_uncertainty	peak_summary_id	geometry
0	33.681111	-78.891667	2017 Irma	Seed line	Good: +/- 0.10 ft	NAVD88	Tape measure	Matt Petkewich	Marker	RT-GNSS	...	2	1381.0	1381.0	0.02	HWM	NaN	NaN	NaN	NaN	POINT (1566648.913 -286040.618)
1	33.681111	-78.891667	2020 Isaias	Mud	Good: +/- 0.10 ft	NAVD88	RT-GNSS	Andy Caldwell	Marker	Phone/Car GPS	...	2	1381.0	1381.0	0.02	HWM1	NaN	3.0	0.1	NaN	POINT (1566648.913 -286040.618)
2	33.681111	-78.891667	2020 Isaias	Mud	Good: +/- 0.10 ft	NAVD88	RT-GNSS	Andy Caldwell	Marker	Phone/Car GPS	...	2	1381.0	1381.0	0.02	HWM2	NaN	3.0	0.1	NaN	POINT (1566648.913 -286040.618)
3	35.281051	-76.662585	2011 Irene	Seed line	Excellent: +/- 0.05 ft	NAVD88				Map (digital or paper)	...	2	36.0	NaN	NaN	no_label	NaN	NaN	NaN	NaN	POINT (1731958.461 -71714.640)
4	34.311221	-77.733077	2018 Florence	Debris	Poor: +/- 0.40 ft	NAVD88	RT-GNSS	Steve Harden	Stake	Map (digital or paper)	...	2	761.0	1919.0	0.06	HWM-01	NaN	0.0	NaN	NaN	POINT (1658023.273 -197179.233)

5 rows × 54 columns

The first step involves retrieving the data dictionary for HWMs. We can use the as_dict argument to return the data as a dictionary but will prefer the default GeoDataFrame for this example. We can also pass keyword arguments to the async retriever as shown here where the caching is disabled.

hwm_dd = STNFloodEventData.data_dictionary(
    "hwms", as_dict=False, async_retriever_kwargs={"disable": True}
)
hwm_dd.head()

	Field	Definition
0	latitude_dd	Horizontal location of HWM in decimal degrees ...
1	latitude	repeat of latitude_dd
2	longitude	repeat of longitude_dd
3	eventName	Event name
4	hwmTypeName	Type of HWM evience. See hwm_type_id for curr...

It’s important to note that the schemas for the three requests: all data, filtered data, and data dictionaries don’t necessarily agree.

# compares the columns
print(
    f"Do the columns have the same length?: {set(hwm_all.columns) == set(hwm_filtered.columns) == set(hwm_dd.columns)}"
)

# compare columns
pd.concat(
    [
        pd.Series(hwm_all.columns, name="All HWM Fields"),
        pd.Series(hwm_filtered.columns, name="Filtered HWM Fields"),
        pd.Series(hwm_dd["Field"], name="HWM Data Dictionary Fields"),
    ],
    axis=1,
).head()

Do the columns have the same length?: False

	All HWM Fields	Filtered HWM Fields	HWM Data Dictionary Fields
0	hwm_id	latitude	latitude_dd
1	waterbody	longitude	latitude
2	site_id	eventName	longitude
3	event_id	hwmTypeName	eventName
4	hwm_type_id	hwmQualityName	hwmTypeName

While many of the differences can be inferred, some of the discrepancies could lead to columns with ambiguous information. The USGS is working on an updated RESTFul API that should address this. These differences are available for the other data types, “instruments”, “peaks”, and “sites”, as well.

Now we will plot some of the HWMs. First we retrieve some state lines and project those as well as the filtered HWMs to the ‘EPSG:4329’ CRS.

carolina_lines = gh.get_us_states(["NC", "SC"]).to_crs("EPSG:4329")
hwm_filtered = hwm_filtered.to_crs("EPSG:4329")

fig, ax = plt.subplots(figsize=(6, 4), dpi=150)

event_names = hwm_filtered.loc[:, "eventName"].unique()
markers = dict(zip(event_names, mmarkers.MarkerStyle.filled_markers[: len(event_names)]))

ax.set_title("HWMs - Height Above Ground (ft)", fontsize=9)
ax.set_xlabel("Longitude (deg)", fontsize=8)
ax.set_ylabel("Latitude (deg)", fontsize=8)

ax.tick_params(axis="both", which="major", labelsize=8)

vmin, vmax = (
    hwm_filtered.loc[:, "height_above_gnd"].min(),
    hwm_filtered.loc[:, "height_above_gnd"].max(),
)

legend = True
for i, (event_name, data) in enumerate(hwm_filtered.groupby("eventName")):
    if i == 1:
        legend = False

    data.plot(
        ax=ax,
        column="height_above_gnd",
        alpha=0.7,
        legend=legend,
        markersize=3,
        marker=markers[event_name],
        vmin=0,
        vmax=10,
    )

# Create a list of Line2D objects to use for the legend
legend_elements = [
    mlines.Line2D([0], [0], color="black", marker=markers[event_name], linestyle="None")
    for event_name in event_names
]

# Add the legend to the plot
ax.legend(
    legend_elements,
    event_names,
    loc="lower right",
    title="Event Names",
    bbox_to_anchor=(1, 0),
    prop={"size": 5},
)

colorbar = plt.gcf().get_axes()[-1]
colorbar.tick_params(labelsize=8)

carolina_lines.plot(ax=ax, facecolor="none", edgecolor="black", linewidth=0.2)
fig.savefig(Path("_static", "hmws.png"), bbox_inches="tight", dpi=150)

Data Quality Issues

Inspecting the figure above reveals a HWM in the Atlantic Ocean. Trying to pick that one out, we get the following information about the outlier. We only display a few of the fields that may contain the problem.

outlier = hwm_filtered.loc[hwm_filtered.latitude < 31, :].squeeze()
print(
    outlier.loc[
        [
            "siteDescription",
            "waterbody",
            "stateName",
            "countyName",
            "latitude_dd",
            "longitude_dd",
            "site_latitude",
            "site_longitude",
            "height_above_gnd",
        ]
    ]
)

siteDescription     US Route 17 crossing of the Pee Dee and Waccam...
waterbody                                    Pee Dee / Waccamaw River
stateName                                                          SC
countyName                                          Georgetown County
latitude_dd                                                  30.36694
longitude_dd                                                -79.26778
site_latitude                                               33.365582
site_longitude                                              -79.25334
height_above_gnd                                                 2.68
Name: 1127, dtype: object

Inspecting the fields above reveals that this potential outlier should be in Georgetown County which is on the coast of South Carolina just south of Myrtle Beach. Additionally, the fields show two different entries for latitude and longitudes. We look at the definitions for latitudes below.

print(
    f"'site_latitude' : {hwm_dd.loc[hwm_dd.loc[:,'Field'] == 'site_latitude','Definition'].iloc[0]}"
)
print(f"'latitude_dd' : {hwm_dd.loc[hwm_dd.loc[:,'Field'] == 'latitude_dd','Definition'].iloc[0]}")

'site_latitude' : Horizontal location of Site of common water surface (not location of HWM)
'latitude_dd' : Horizontal location of HWM in decimal degrees (not location of associated site)

From this, we can say that the ‘site_latitude’ field reveals horizontal locations of the common water surface while the ‘latitude_dd’ field refers to that of the HWM. This distinction indicates why these two fields are expected to differ. Nevertheless, the location of the HWM all but impossibly collected so far from the Pee Dee and Waccamaw Rivers. It’s likely that this was a typo. It’s important to note that this service is fed by data by real people who are liable to make simple mistakes. It’s advised to take a look at your data and inspect for any inconsistencies prior to using for analysis.

This page was generated from hymod_calibration.ipynb. Interactive online version:

Making Waves in Water Science: Open Source Tools

Hosted by CUAHSI

HyRiver: Hydroclimate Data Retriever

Presenter: Taher Chegini, University of Houston

Email: cheginit@gmail.com

Nov 6th, 2022

Introduction

The traditional approach for retrieving hydroclimate data from different source was mostly as follows:

• Download entire datasets from various data sources to a local workstation

• Process the data and subset for the region of interest on the workstation

• Store the processed input data and the analyses results locally

• Upload the data to a remote server for publication and sharing

Over the last decade using web services have become more common in the geospatial community.

Image source

HyRiver is a suite of eight open-source (MIT License) Python packages that bridges the gap between (non-technical) hydrologists and complexities of working with web services. Contribution of any kind are most welcome.

Package	Description
	Navigate and subset NHDPlus (MR and HR) using web services
	Access topographic data through National Map’s 3DEP web service
	Access gNATSGO, NWIS, NID, WQP, HCDN 2009, NLCD, CAMELS, and SSEBop databases
	Access daily, monthly, and annual climate data via Daymet
	Access hourly NLDAS2 forcing data
	A collection of tools for computing hydrological signatures
	High-level API for asynchronous requests with persistent caching
	Send queries to any ArcGIS RESTful-, WMS-, and WFS-based services
	Utilities for manipulating geospatial, (Geo)JSON, and (Geo)TIFF data

Here’s a chart of HyRiver packages dependencies:

An Application in Hydrological Modeling

from pathlib import Path

import contextily as ctx
import numpy as np
import pandas as pd
from hymod import HYMOD
from scipy import optimize

import pydaymet as daymet
import pygeohydro as gh
import pygeoutils as geoutils
from pygeohydro import NWIS
from pynhd import NLDI

We use HyRiver packages to get the required input data for hydrological modeling of a watershed. We use PyDaymet to get precipitation and potential evapotranspiration and PyGeoHydro to get streamflow observations and soil storage capcity. Then, we use an implementation of the HYMOD model for our watershed analysis. The source for the model can be found in hymod.py file.

First, we use the CAMELS dataset to select a natural watershed with no significant snow since this implementation of HYMOD does not have a snow component.

camels_basins, camels_qobs = gh.get_camels()

stations = camels_basins[
    (camels_basins.area_gages2 > 300)
    & (camels_basins.area_gages2 < 500)
    & (camels_basins.frac_snow < 0.1)
]
station = stations.iloc[0].name

Next, we get more information about our selected watershed drainage area geometry using PyNHD and PyGeoHydro. To ensure that the station is located at the outlet of the watershed, we set split_catchment flag to True to split the catchment at the station’s location.

dates = ("2003-01-01", "2012-12-31")
nwis = NWIS()

basin = NLDI().get_basins(station, split_catchment=True)
info = nwis.get_info({"site": station})
info

	agency_cd	site_no	station_nm	site_tp_cd	dec_lat_va	dec_long_va	coord_acy_cd	dec_coord_datum_cd	alt_va	alt_acy_va	alt_datum_cd	huc_cd	nhd_id	nhd_areasqkm	hcdn_2009	geometry
0	USGS	01666500	Robinson River Near Locust Dale, VA	ST	38.32513	-78.095556	U	NAD83	282.64	0.08	NAVD88	02080103	8468559	463.61	True	POINT (-78.09556 38.32513)

Let’s check the area of the watershed and plot it.

area_sqm = basin.to_crs(basin.estimate_utm_crs()).area.values[0]
ax = basin.plot(figsize=(9, 4), alpha=0.5, edgecolor="black")
info.plot(ax=ax, markersize=50, color="red")
ax.set_title(f"USGS-{station} ({area_sqm * 1e-6:.1f} sqkm)")
ctx.add_basemap(ax, crs=basin.crs)
ax.margins(0)
ax.set_axis_off()
ax.figure.savefig(Path("_static", "hymod.png"), dpi=300, bbox_inches="tight", facecolor="w")

Now, we get some information about this station and then retrieve streamflow observations within the period of our study for calibrating the model from PyGeoHydro. Since HYMOD’s output is in mm/day we can either set the mmd flag of NWIS.get_streamflow to True to return the data in mm/day instead of the default cms, or use area_sqm that we obtained previously to do the conversion. Note that these two methods might not return identical results since NWIS.get_streamflow uses GagesII and NWIS to get the drainage area while here we used NLDI to obtain the area and these two are not always the same.

qobs = nwis.get_streamflow(station, dates)
qobs = qobs * (1000.0 * 24.0 * 3600.0) / area_sqm

We can check the difference in areas produced by these two methods as follows:

area_nwis = nwis._drainage_area_sqm(info, "dv")
f"NWIS: {area_nwis.iloc[0] * 1e-6:.1f} sqkm, NLDI: {area_sqm * 1e-6:.1f} sqkm"

'NWIS: 463.6 sqkm, NLDI: 465.5 sqkm'

PyDaymet can compute Potential EvapoTranspiration (PET) at daily timescale using three methods: penman_monteith, priestley_taylor, and hargreaves_samani. Let’s use hargreaves_samani and get the data for our study period.

clm = daymet.get_bygeom(basin.geometry[0], dates, variables="prcp", pet="hargreaves_samani")

_ = (
    clm.where(clm.pet > 0)
    .pet.isel(time=range(100, 700, 100))
    .plot(x="x", y="y", row="time", col_wrap=3)
)

Since our HYMOD implementation is a lumped model, we need to take the areal average of the input precipitation and potential evapotranspiration data. Moreover, since Daymet’s calendar doesn’t have leap years (365-day year), we need to make sure that the time index of the input streamflow observation matches that of the climate data.

clm_df = clm.mean(dim=["x", "y"]).to_dataframe()[["prcp", "pet"]]
clm_df.index = pd.to_datetime(clm_df.index.date)

qobs.index = pd.to_datetime(qobs.index.to_series().dt.tz_localize(None).dt.date)
idx = qobs.index.intersection(clm_df.index)
qobs, clm_df = qobs.loc[idx], clm_df.loc[idx]

We can use soil data to estimate \(C_{max}\) parameter of HYMOD model instead of calibrating it.

geometry = basin.geometry[0]
crs = basin.crs

porosity = gh.soil_properties("por").porosity
porosity = geoutils.xarray_geomask(porosity, geometry, crs)
porosity = porosity.where(porosity > porosity.rio.nodata)
porosity = porosity.rio.write_nodata(np.nan)

thickness = gh.soil_gnatsgo("tk0_999a", geometry, crs).tk0_999a
thickness = thickness.where(thickness < 2e6, drop=False) * 10
thickness = thickness.rio.write_nodata(np.nan)
thickness = thickness.rio.reproject_match(porosity, resampling=5)

storage = porosity * thickness * 1e-3
storage = storage.rio.write_nodata(np.nan)
cmax = storage.mean().compute().item()

Now, let’s use scipy to calibrate the model parameters using 70% of the observation data and Kling-Gupta Efficiency as the objective function.

cal_idx = int(0.7 * clm_df.shape[0])
model = HYMOD(clm_df.iloc[:cal_idx], qobs.iloc[:cal_idx].squeeze(), cmax, 1)
bounds = {
    # "c_max": (1, 1500),  # Maximum storage capacity
    "b_exp": (0.0, 1.99),  # Degree of spatial variability of the soil moisture capacity
    "alpha": (0.01, 0.99),  # Factor for dividing flow to slow and quick releases
    "k_s": (0.01, 0.14),  # Residence time of the slow release reservoir
    "k_q": (0.14, 0.99),  # Residence time of the quick release reservoir
}
results = optimize.differential_evolution(
    model.fit,
    list(bounds.values()),
    popsize=200,
    seed=1,
)
dict(zip(bounds, results.x.round(2).tolist()))

{'b_exp': 0.32, 'alpha': 0.99, 'k_s': 0.14, 'k_q': 0.65}

With the obtained calibrated parameters, we can validate the model over the validation period (30%) of the streamflow observation.

warm_up = clm_df.iloc[:cal_idx].index.year.unique().shape[0]
model = HYMOD(clm_df, qobs.squeeze(), cmax, warm_up)
qsim = model.simulate(results.x)
kge = -model.fit(results.x)
name = info.station_nm.values[0]
index = qobs.iloc[cal_idx:].index
discharge = pd.DataFrame(
    {"Simulated": qsim[model.cal_idx], "Observed": model.qobs[model.cal_idx]}, index=index
)
_ = gh.plot.signatures(
    discharge,
    clm_df.iloc[cal_idx:].prcp,
    title=f"{name}\nKGE = {kge:.2f}",
)

This page was generated from ndvi.ipynb. Interactive online version:

NDVI

import io
from pathlib import Path

import pandas as pd
import xarray as xr

import async_retriever as ar

Let’s use the DAAC server to get NDVI. We can use AsyncRetriever and pass it directly to xarray.open_mfdataset.

west, south, east, north = (-69.77, 45.07, -69.31, 45.45)
base_url = "https://thredds.daac.ornl.gov/thredds/ncss/ornldaac/1299"
dates_itr = ((pd.to_datetime(f"{y}0101"), pd.to_datetime(f"{y}0131")) for y in range(2000, 2005))
urls, kwds = zip(
    *(
        (
            f"{base_url}/MCD13.A{s.year}.unaccum.nc4",
            {
                "params": {
                    "var": "NDVI",
                    "north": f"{north}",
                    "west": f"{west}",
                    "east": f"{east}",
                    "south": f"{south}",
                    "disableProjSubset": "on",
                    "horizStride": "1",
                    "time_start": s.strftime("%Y-%m-%dT%H:%M:%SZ"),
                    "time_end": e.strftime("%Y-%m-%dT%H:%M:%SZ"),
                    "timeStride": "1",
                    "addLatLon": "true",
                    "accept": "netcdf",
                }
            },
        )
        for s, e in dates_itr
    )
)
resp = ar.retrieve(urls, "binary", request_kwds=kwds, max_workers=8)
data = xr.open_mfdataset(io.BytesIO(r) for r in resp)

ax = data.isel(time=slice(10, 16)).NDVI.plot(x="x", y="y", row="time", col_wrap=3)
ax.fig.savefig(Path("_static", "ndvi.png"), bbox_inches="tight", facecolor="w")

This page was generated from nhd_navigation.ipynb. Interactive online version:

NHD Navigation

import folium
import networkx as nx

import pynhd
from pynhd import GeoConnex, WaterData

In this example, we explore navigating the National Hydrography Dataset (NHD) using the networkx package. First, we use pynhd.enhd_flowlines_nx to obtain the CONUS drainage network based on the ENHD. This function returns a networkx.DiGraph object, a mapping from the node IDs of the generated network to ComIDs of ENHD, and a list of topologically sorted ComIDs. Note that for performance reasons, the node IDs of the generated network are different than the ComIDs of ENHD.

graph, node2comid, _ = pynhd.enhd_flowlines_nx()
comid2node = {v: k for k, v in node2comid.items()}

We validate our navigation approach with the Mainstems’ dataset that we can retrieve from the GeoConnex web service. Moreover, for visualization purposes, we obtain the flowline geometries from the WaterData web service.

gcx = GeoConnex("mainstems")
nhd_flw = WaterData("nhdflowline_network")

We pick Wisconsin River for this tutorial. Let’s start by getting the mainstem of Wisconsin River from GeoConnex.

ms = gcx.byid("id", "323742")
ms.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

GeoConnex returns a GeoDataFrame containing the requested mainstem(s) and their headwater and outlet NHDPlus v2 ComIDs. We use these two ComIDs to the navigate Wisconsin River in the ENHD network.

outlet_comid = int(ms["outlet_nhdpv2_comid"].str.rsplit("/", n=1).iloc[0][-1])
head_comid = int(ms["head_nhdpv2_comid"].str.rsplit("/", n=1).iloc[0][-1])

Navigating from a headwater to an outlet is straightforward. We use networkx.shortest_path that returns a list of nodes that we can use to get the ComIDs of the flowlines. Finding the shortest path between two nodes in a network is a well-known problem in graph theory. For a weighted graph with positive weight values, the Dijkstra’s algorithm is the most efficient approach. Note that in graph theory, the shortest path is a path that has the minimum sum of its edge weights, i.e., the path with the least resistance (minimum navigation cost).

So, for navigating a NHDPlus from upstream to downstream, we need to consider using an attribute as edge weights that favors the flowlines without divergence. Thus, using the divergence attribute is recommended for cases that divergence is a concern. However, ENHD provides a dendritic drainage network for CONUS, i.e., no divergence. So, we can simply navigate between two nodes in the ENHD network without using any weights.

main_comids = [
    node2comid[n] for n in nx.shortest_path(graph, comid2node[head_comid], comid2node[outlet_comid])
]
flw_main = nhd_flw.byid("comid", main_comids)

For visualization purposes and validating our approach, we overlay the flowlines that we obtained through navigation, on the mainstem of the Wisconsin River.

m = ms.explore(style_kwds={"color": "blue", "weight": 4})
folium.GeoJson(flw_main, style_function=lambda f: {"color": "red", "weight": 1}).add_to(m)
m

Make this Notebook Trusted to load map: File -> Trust Notebook

Now, let’s navigate based on distance. For this purpose, we need to use a Breadth-First Search algorithm. networkx provides networkx.bfs_edges that returns an iterator over edges in a breadth-first-search starting at source. However, it does not take into account the distance or direction. Thus, we write a custom BFS method that allows us to navigate in upstream (including both the mainstem and its tributaries), upstream only through mainstam, and downstream. Note that since ENHD is a dendritic network, we don’t have to worry about downstream navigation through mainstem only since it will be the same as downstream navigation through the whole network. For upstream navigation along mainstems, we need to use the stream order attribute. NHDPlus provides Strahler stream order as an attribute called streamorde. In Strahler’s stream order, the stream order increases as streams merge, i.e., mainstems have the highest stream order. So, we can use this attribute to navigate upstream along the mainstems by favoring the flowlines with higher stream order. Also, we use lengthkm attribute from NHDPlus for computing the distance of the navigation path.

In the function below, note the use of Python’s deque built-in function for performance reasons.

from collections import deque
from typing import Literal


def navigate_by_distance(
    graph: nx.DiGraph,
    source_node: int,
    nav_dir: Literal["downstream", "upstream", "upstream_main"],
    distance_km: int,
    lengthkm_attr: str = "lengthkm",
    streamorder_attr: str = "streamorde",
) -> list[int]:
    """Navigate a directed graph by distance, with modifications to handle edge cases.

    Parameters
    ----------
    graph: nx.DiGraph
        The directed graph to navigate.
    source_node: int
        The source node in `graph` from which to start navigation.
    nav_dir: {"downstream", "upstream", "upstream_main"}
        Direction of navigation.
    distance_km: int
        The maximum distance to navigate in kilometers.
    lengthkm_attr: str, optional
        The edge attribute name that indicates the length of edges.
        Defaults to ``length``.
    streamorder_attr: str, optional
        The edge attribute name that indicates the stream order of edges.
        Only used if ``nav_dir`` is ``upstream_main``.
        Defaults to ``streamorde``.

    Returns
    -------
    list
        Nodes within the specified distance from the source in the given
        navigation direction.
    """
    attrs = next(iter(graph.edges.data()))[-1]
    if lengthkm_attr not in attrs:
        raise ValueError(f"lengthkm_attr must be one of {list(attrs)}")

    if nav_dir == "upstream_main" and streamorder_attr not in attrs:
        raise ValueError(f"streamorder_attr must be one of {list(attrs)}")

    if nav_dir == "downstream":
        nb_func = graph.successors
        get_weight = lambda node, nb, w: graph.edges[node, nb][w]
    elif nav_dir == "upstream":
        nb_func = graph.predecessors
        get_weight = lambda node, nb, w: graph.edges[nb, node][w]
    elif nav_dir == "upstream_main":
        get_stream_order = lambda node, pred: graph.edges[pred, node][streamorder_attr]

        def max_predecessor(node: int)->list[int]:
            preds = graph.predecessors(node)
            try:
                return [max(preds, key=lambda pred: get_stream_order(node, pred))]
            except ValueError:
                return []

        nb_func = max_predecessor
        get_weight = lambda node, nb, w: graph.edges[nb, node][w]
    else:
        raise ValueError("nav_dir must be either 'downstream',  'upstream', or 'upstream_main'")

    if source_node not in graph:
        raise ValueError(f"Source node {source_node} not found in graph")

    visited = {source_node}
    queue = deque([(source_node, 0)])
    nodes_within_distance = []

    while queue:
        node, distance = queue.popleft()
        if distance <= distance_km:
            nodes_within_distance.append(node)
            for nb in nb_func(node):
                total_distance = distance + get_weight(node, nb, lengthkm_attr)

                if nb not in visited and total_distance <= distance_km:
                    visited.add(nb)
                    queue.append((nb, total_distance))

    return nodes_within_distance

First, let’s get all the flowlines that are upstream of the river’s outlet up to 80 km.

nodes_up = navigate_by_distance(graph, comid2node[outlet_comid], "upstream", 80)
comids = [node2comid[n] for n in nodes_up]
flw_up = nhd_flw.byid("comid", comids)

m = ms.explore(style_kwds={"color": "blue", "weight": 4})
folium.GeoJson(flw_up, style_function=lambda f: {"color": "red", "weight": 1}).add_to(m)
m

Make this Notebook Trusted to load map: File -> Trust Notebook

Next, let’s get all the flowlines that are upstream of the river’s outlet up to 80 km and along its mainstem.

nodes_up = navigate_by_distance(graph, comid2node[outlet_comid], "upstream_main", 80)
comids = [node2comid[n] for n in nodes_up]
flw_up = nhd_flw.byid("comid", comids)

m = ms.explore(style_kwds={"color": "blue", "weight": 4})
folium.GeoJson(flw_up, style_function=lambda f: {"color": "red", "weight": 1}).add_to(m)
m

Make this Notebook Trusted to load map: File -> Trust Notebook

Next, we get all the flowlines that are downstream of the river’s headwater up to 200 km.

nodes_dn = navigate_by_distance(graph, comid2node[head_comid], "downstream", 200)
comids = [node2comid[n] for n in nodes_dn]
flw_dn = nhd_flw.byid("comid", comids)

m = ms.explore(style_kwds={"color": "blue", "weight": 4})
folium.GeoJson(flw_dn, style_function=lambda f: {"color": "red", "weight": 1}).add_to(m)
m

Make this Notebook Trusted to load map: File -> Trust Notebook

This page was generated from nhdplus.ipynb. Interactive online version:

River Network

from pathlib import Path

import geopandas as gpd
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

import pynhd as nhd
from pynhd import NLDI, NHDPlusHR, WaterData

PyNHD provides access to the Hydro Network-Linked Data Index (NLDI) and the WaterData web services for navigating and subsetting NHDPlus V2 database. Additionally, you can download NHDPlus High Resolution data as well.

First, let’s get the watershed geometry of the contributing basin of a USGS station using NLDI:

nldi = NLDI()
station_id = "01031500"

basin = nldi.get_basins(station_id)

The navigate_byid class method can be used to navigate NHDPlus in both upstream and downstream of any point in the database. The available feature sources are comid, huc12pp, nwissite, wade, wqp. Let’s get ComIDs and flowlines of the tributaries and the main river channel in the upstream of the station.

flw_main = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamMain",
    source="flowlines",
    distance=1000,
)

flw_trib = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="flowlines",
    distance=1000,
)

We can get other USGS stations upstream (or downstream) of the station and even set a distance limit (in km):

st_all = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="nwissite",
    distance=1000,
)

st_d20 = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="nwissite",
    distance=20,
)

Now, let’s get the HUC12 pour points:

pp = nldi.navigate_byid(
    fsource="nwissite",
    fid=f"USGS-{station_id}",
    navigation="upstreamTributaries",
    source="huc12pp",
    distance=1000,
)

Let’s plot the vector data:

ax = basin.plot(facecolor="none", edgecolor="k", figsize=(8, 8))
st_all.plot(ax=ax, label="USGS stations", marker="*", markersize=300, zorder=4, color="b")
st_d20.plot(
    ax=ax,
    label="USGS stations up to 20 km",
    marker="v",
    markersize=100,
    zorder=5,
    color="darkorange",
)
pp.plot(ax=ax, label="HUC12 pour points", marker="o", markersize=50, color="k", zorder=3)
flw_main.plot(ax=ax, lw=3, color="r", zorder=2, label="Main")
flw_trib.plot(ax=ax, lw=1, zorder=1, label="Tributaries")
ax.legend(loc="best")
ax.set_aspect("auto")
ax.set_axis_off()
ax.figure.set_dpi(100)
ax.figure.savefig("_static/nhdplus_navigation.png", bbox_inches="tight", facecolor="w")

Next, we get the slope data for each river segment from NHDPlus VAA database:

vaa = nhd.nhdplus_vaa("input_data/nhdplus_vaa.parquet")

flw_trib["comid"] = pd.to_numeric(flw_trib.nhdplus_comid)
slope = gpd.GeoDataFrame(
    pd.merge(flw_trib, vaa[["comid", "slope"]], left_on="comid", right_on="comid"),
    crs=flw_trib.crs,
)
slope[slope.slope < 0] = np.nan

ax = slope.plot(
    figsize=(8, 8),
    column="slope",
    cmap="plasma",
    legend=True,
    legend_kwds={"label": "Slope (m/m)"},
)
ax.set_axis_off()

Now, let’s use WaterData service to get the headwater catchments for this basin:

wd_cat = WaterData("catchmentsp")
cat = wd_cat.bygeom(basin.geometry[0], predicate="overlaps")

ax = cat.plot(figsize=(8, 8))
basin.plot(ax=ax, facecolor="none", edgecolor="k")
ax.set_aspect("auto")
ax.figure.set_dpi(100)
ax.set_axis_off()

We might to get all the HUC12 pour points within a radius of an station. We can achieve this using bydistance method of WaterData class. For example, let’s take station 01031300 and find all the flowlines within a radius of 5 km.

eck4 = "+proj=eck4 +lon_0=0 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs"
st300 = st_all[st_all.identifier == "USGS-01031300"].to_crs(eck4)
coords = st300.geometry.values[0].coords.xy
x, y = coords[0][0], coords[1][0]
rad = 5e3

nhdp_mr = WaterData("nhdflowline_network")

flw_rad = nhdp_mr.bydistance((x, y), rad, loc_crs=eck4)
flw_rad = flw_rad.to_crs(eck4)

Instead of getting all features within a radius of the coordinate, we can snap to the closest flowline using NLDI:

comid_closest = nldi.comid_byloc((x, y), eck4)
flw_closest = nhdp_mr.byid("comid", comid_closest.comid.values[0])

ax = basin.to_crs(eck4).plot(figsize=(8, 8), facecolor="none", edgecolor="k")
flw_rad.plot(ax=ax)
flw_closest.to_crs(eck4).plot(ax=ax, color="orange", lw=3)
ax.legend(["Flowlines", "Closest Flowline"])

circle = mpatches.Circle((x, y), rad, ec="r", fill=False)
arrow = mpatches.Arrow(x, y, 0.6 * rad, -0.8 * rad, width=1500, color="g")
ax.text(x + 0.6 * rad, y + -rad, f"{int(rad)} m", ha="center")
ax.add_artist(circle)
ax.set_aspect("equal")
ax.add_artist(arrow)
ax.figure.set_dpi(100)
ax.axis("off")
ax.figure.savefig("_static/nhdplus_radius.png", bbox_inches="tight", facecolor="w")

WaterData gives us access to the medium-resolution NHDPlus database. We can use NHDPlusHR to retrieve high-resolution NHDPlus data. Let’s get both medium- and high-resolution flowlines within the bounding box of our watershed and compare them. Moreover, Since several web services offer access to NHDPlus database, NHDPlusHR has an argument for selecting a service and also an argument for automatically switching between services.

flw_mr = nhdp_mr.bybox(basin.geometry[0].bounds)

nhdp_hr = NHDPlusHR("flowline")
flw_hr = nhdp_hr.bygeom(basin.geometry[0].bounds)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 8), facecolor="w")

flw_mr.plot(ax=ax1)
ax1.set_title("NHDPlus Medium Resolution")
ax1.set_axis_off()
flw_hr.plot(ax=ax2)
ax2.set_title("NHDPlus High Resolution")
ax2.set_axis_off()
fig.savefig("_static/hr_mr.png", bbox_inches="tight", facecolor="w")

Since NHDPlus HR is still at the pre-release stage, let’s use the MR flowlines to demonstrate the vector-based accumulation.

Based on a topological sorted river network pynhd.vector_accumulation computes flow accumulation in the network. It returns a dataframe which is sorted from upstream to downstream that shows the accumulated flow in each node.

PyNHD has a utility called prepare_nhdplus that identifies such relationship among other things such as fixing some common issues with NHDPlus flowlines. But first we need to get all the NHDPlus attributes for each ComID since NLDI only provides the flowlines’ geometries and ComIDs which is useful for navigating the vector river network data. For getting the NHDPlus database we use WaterData. The WaterData web service layers are nhdflowline_network, nhdarea, nhdwaterbody, catchmentsp, gagesii, huc08, huc12, huc12agg, and huc12all. Let’s use the nhdflowline_network layer to get required info.

comids = [int(c) for c in flw_trib.nhdplus_comid.to_list()]
nhdp_trib = nhdp_mr.byid("comid", comids)
flw = nhd.prepare_nhdplus(nhdp_trib, 0, 0, purge_non_dendritic=False)

To demonstrate the use of routing, let’s use nhdplus_attrs function to get list of available NHDPlus attributes from Select Attributes for NHDPlus Version 2.1 Reach Catchments and Modified Network Routed Upstream Watersheds for the Conterminous United States item on ScienceBase service. These attributes are in catchment-scale and are available in three categories:

1. Local (local): For individual reach catchments,

2. Total (upstream_acc): For network-accumulated values using total cumulative drainage area,

3. Divergence (div_routing): For network-accumulated values using divergence-routed.

nldi.valid_characteristics.head(5)

	characteristic_id	characteristic_description	units	dataset_label	dataset_url	theme_label	theme_url	characteristic_type
0	CAT_WILDFIRE_2011	percent of wild fires per catchments for 2011	percent	Estimated percent of catchment that experience...	https://www.sciencebase.gov/catalog/item/57f66...	Land Cover	unknown	localCatch_name
1	CAT_CONTACT	Subsurface flow contact time index. The subsur...	days	Contact time, the length of time it takes for ...	https://www.sciencebase.gov/catalog/item/56f96...	Hydrologic	https://www.sciencebase.gov/catalog/item/5669a...	localCatch_name
2	CAT_EWT	Average depth to water table relatice to the l...	meters	Average depth to water table relative to land ...	https://www.sciencebase.gov/catalog/item/56f97...	Hydrologic	https://www.sciencebase.gov/catalog/item/5669a...	localCatch_name
3	CAT_FUNGICIDE	Fungicide use on agricultural land in kilogram...	kilogram per square kilometer	Fungicide use on agricultural land in kilogram...	https://www.sciencebase.gov/catalog/item/56fd6...	Chemical	https://www.sciencebase.gov/catalog/item/56fd6...	localCatch_name
4	CAT_IEOF	Percentage of Horton overland flow as a percent	percent	Mean infiltration-excess overland flow as a pe...	https://www.sciencebase.gov/catalog/item/56f97...	Hydrologic	https://www.sciencebase.gov/catalog/item/5669a...	localCatch_name

Let’s get Mean Annual Groundwater Recharge, RECHG, using getcharacteristic_byid class method and carry out the flow accumulation.

char = "CAT_RECHG"
area = "areasqkm"

local = nldi.getcharacteristic_byid(comids, "local", char_ids=char)
flw = flw.merge(local[char], left_on="comid", right_index=True)


def runoff_acc(qin, q, a):
    return qin + q * a


flw_r = flw[["comid", "tocomid", char, area]]
runoff = nhd.vector_accumulation(flw_r, runoff_acc, char, [char, area])


def area_acc(ain, a):
    return ain + a


flw_a = flw[["comid", "tocomid", area]]
areasqkm = nhd.vector_accumulation(flw_a, area_acc, area, [area])

runoff /= areasqkm

Since these are catchment-scale characteristic, let’s get the catchments then add the accumulated characteristic as a new column and plot the results.

catchments = wd_cat.byid("featureid", comids)

c_local = catchments.merge(local, left_on="featureid", right_index=True)
c_acc = catchments.merge(runoff, left_on="featureid", right_index=True)

/var/folders/jx/s06wj7hs325c_2yd2g5209sr0000gp/T/ipykernel_77507/1102694309.py:1: UserWarning: 8 of 432 requests failed.. IDs of the failed requests are ['1723523', '1723331', '1721829', '1721855', '1721857', '1723355', '1723559', '1721835']
  catchments = wd_cat.byid("featureid", comids)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 8), facecolor="w")

cmap = "viridis"
norm = plt.Normalize(vmin=c_local.CAT_RECHG.min(), vmax=c_acc.acc_CAT_RECHG.max())

c_local.plot(ax=ax1, column=char, cmap=cmap, norm=norm)
flw.plot(ax=ax1, column="streamorde", cmap="Blues", scheme="fisher_jenks")
ax1.set_title("Groundwater Recharge (mm/yr)")
ax1.set_axis_off()

c_acc.plot(ax=ax2, column=f"acc_{char}", cmap=cmap, norm=norm)
flw.plot(ax=ax2, column="streamorde", cmap="Blues", scheme="fisher_jenks")
ax2.set_title("Accumulated Groundwater Recharge (mm/yr)")
ax2.set_axis_off()

cax = fig.add_axes(
    [
        ax2.get_position().x1 + 0.01,
        ax2.get_position().y0,
        0.02,
        ax2.get_position().height,
    ]
)
sm = plt.cm.ScalarMappable(cmap=cmap, norm=norm)
fig.colorbar(sm, cax=cax)
fig.savefig(Path("_static", "flow_accumulation.png"), bbox_inches="tight", facecolor="w")

This page was generated from nid.ipynb. Interactive online version:

National Inventory of Dams

import matplotlib.pyplot as plt

import pygeohydro as gh

First, we need to instantiate the NID class.

nid = gh.NID()

Some dam coordinates are either missing or incorrect. Let’s get dams that are within Contiguous US with max storage larger than 200 acre-feet. Note that since we want to get all dams within CONUS it’s much more efficient set NID.gdf which is a GeoDataFrame of the NID dataset.

conus_geom = gh.get_us_states("conus")

min_storage = 2500
dam_list = nid.get_byfilter([{"maxStorage": [f"[{min_storage} +inf]"]}])
dams = nid.gdf[nid.gdf["federalId"].isin(dam_list[0].federalId.to_list())]

conus_dams = dams[dams.stateKey.isin(conus_geom.STUSPS)].reset_index(drop=True)

Next, we can get a count of the top 10 dams based on types.

purpose_count = conus_dams["primaryPurposeId"].value_counts()
_, ax = plt.subplots(figsize=(5, 3), dpi=100)
purpose_count.sort_values()[-10:].plot.barh(ax=ax)
ax.set_xlim(0, purpose_count.max() * 1.1)
ax.set_xlabel("Number of dams")
for p in ax.patches:
    ax.annotate(
        int(p.get_width()),
        (p.get_width() + 120, p.get_y() + p.get_height() / 2),
        ha="center",
        va="center",
    )

Let’s compare the spatial distribution of the top five categories, excluding Earth and Other categories.

conus_geom = conus_geom.to_crs(5070)
conus_dams = conus_dams.to_crs(5070)

fig, ax = plt.subplots(figsize=(9, 6), dpi=100)
ax.set_title(f"Dams within CONUS with max sotrage > {min_storage} acre-feet")
conus_geom.plot(ax=ax, facecolor="none", edgecolor="k")
top_5types = purpose_count.index[:5]
marker = dict(zip(top_5types, ["o", "^", "*", "X", "d"]))
color = dict(zip(top_5types, ["r", "b", "g", "k", "c"]))

for c in top_5types:
    conus_dams[conus_dams.primaryPurposeId == c].plot(
        ax=ax,
        alpha=0.4,
        markersize=12,
        marker=marker[c],
        color=color[c],
        label=c,
    )
ax.legend(loc="best", ncols=1)
ax.axis(False)
fig.savefig("_static/dams.png")

This page was generated from nid_dib.ipynb. Interactive online version:

Spatiotemporal Distribution of Dams

import shutil
from pathlib import Path

import geopandas as gpd
import matplotlib
import matplotlib.pyplot as plt
from IPython.display import YouTubeVideo
from matplotlib.font_manager import fontManager
from tqdm.auto import tqdm

import pygeohydro as gh

Get the data from the National Inventory of Dams using PyGeoHydro package within contiguous US. Note that since we want to get all dams within CONUS it’s much more efficient to “stage” the latest version of the entire NID database by calling NID().stage_nid_inventory(). This function downloads the entire NID dataset and saves it as a feather file.

nid = gh.NID()
nid.stage_nid_inventory()

dams = gpd.read_feather(nid.nid_inventory_path)
conus_geom = gh.helpers.get_us_states("contiguous").to_crs(dams.crs)
conus_dams = dams[dams.stateKey.isin(conus_geom.STUSPS)].reset_index(drop=True)

Since we’re going to use dam coordinates and completion year to filter the data, let’s check number of missing data to find the total available dams that include these two data.

missing = ~(conus_dams.is_valid).sum() + conus_dams.yearCompleted.isna().sum()
conus_dams.shape[0] - missing

165251

Let’s plot the number of missing data by state.

ax = (
    conus_dams[~conus_dams.is_valid | conus_dams.yearCompleted.isna()]
    .groupby("stateKey")
    .size()
    .plot.bar(figsize=(10, 4))
)
ax.set_title("Number of dams with missing coordinate or completion year data")

for p in ax.patches:
    ax.annotate(
        p.get_height(),
        (p.get_x() + p.get_width() / 2, p.get_height() + 80),
        ha="center",
        va="center",
    )

Plot the frames.

column = "damHeight"
cmap = "viridis"
min_q, max_q = 0.1, 0.9

conus_dams = conus_dams[conus_dams.is_valid & (conus_dams.yearCompleted > 0)]

label = nid.fields_meta[nid.fields_meta["name"] == column].label.iloc[0]
label = "\n".join([label, f"{min_q} - {max_q} Quantile"])
norm = plt.Normalize(
    vmin=conus_dams[column].quantile(min_q), vmax=conus_dams[column].quantile(max_q)
)

dpi = 300.0
figsize = (1920.0 / dpi, 1080.0 / dpi)
font = "Lato"
indent = "\n        "
if font in {f.name for f in fontManager.ttflist}:
    matplotlib.rcParams["font.sans-serif"] = font
    matplotlib.rcParams["font.family"] = "sans-serif"
plt.ioff()

root = Path("tmp")
shutil.rmtree(root, ignore_errors=True)
root.mkdir(exist_ok=True)

conus_geom = conus_geom.to_crs(5070)
conus_dams = conus_dams.to_crs(5070)


def get_ax():
    ax = conus_geom.plot(figsize=figsize, facecolor="none", edgecolor="k", lw=0.2, alpha=0.5)
    ax.axis(False)
    fig = ax.figure
    fig.set_dpi(dpi)
    cax = fig.add_axes(
        [
            ax.get_position().x1 + 0.01,
            ax.get_position().y0,
            0.02,
            ax.get_position().height,
        ]
    )
    sm = plt.cm.ScalarMappable(cmap=cmap, norm=norm)
    fig.colorbar(sm, cax=cax, label=label)
    return ax


yr_min = conus_dams.yearCompleted.astype("Int32").min()
yr_max = conus_dams.yearCompleted.astype("Int32").max()
years = range(yr_min + 1, yr_max + 1)
with tqdm(total=len(years), desc="Plotting") as pbar:
    for year in years:
        pbar.set_postfix_str(f"Year: {year}")
        dams = conus_dams[conus_dams.yearCompleted <= year]
        ax = get_ax()
        dams.plot(
            ax=ax,
            column=column,
            cmap=cmap,
            norm=norm,
            alpha=0.3,
            markersize=3,
        )
        ax.set_title(f"Dams Completed Up to {year}\nTotal = {len(dams):,}")
        h_max = dams[column].max()
        name_max = dams.iloc[dams[column].argmax()]["name"].title()
        ax.annotate(
            f"Largest Dam:{indent}Height: {h_max:.1f} ft{indent}Name: {name_max}",
            xy=(0, 0),
            xycoords=ax.transAxes,
        )
        ax.figure.savefig(Path(root, f"{year}.png"), bbox_inches="tight", dpi=dpi, facecolor="w")
        plt.close("all")
        pbar.update(1)

Repeat the last frame 100 times.

for i in range(1, 100):
    shutil.copy(Path("tmp", f"{years[-1]}.png"), Path("tmp", f"{years[-1] + i}.png"))
_ = shutil.copy(Path("tmp", "2022.png"), Path("_static", "nid_2022.png"))

PosixPath('_static/nid_2022.png')

Convert the frames to a video file.

!ffmpeg -hide_banner -loglevel panic -start_number 1641 -i tmp/%04d.png -pix_fmt yuv420p -vf scale=1920:-2 -y input_data/NID_2022.mp4

You can check out the video here:

YouTubeVideo("AM2f9MaBjiQ")

This page was generated from nlcd.ipynb. Interactive online version:

Land Use/Land Cover

from pathlib import Path

import matplotlib.pyplot as plt
import networkx as nx
import osmnx as ox

import pygeohydro as gh
from pynhd import NLDI

Land cover, imperviousness, and canopy data can be retrieved from the NLCD database. First, we use PyNHD to get the contributing watershed geometry of three NWIS stations with the ID of USGS-01031450, USGS-01031500, and USGS-01031510:

station_ids = ["01031450", "01318500", "01031510", "03460000"]
geometry = NLDI().get_basins(station_ids)
geometry.index = station_ids

We can now use the nlcd_bygeom and nlcd_bycoords functions to get the NLCD data.

Let’s start by nlcd_bygeom. This function has two positional arguments for passing the target geometries or points of interests and target resolution in meters. Note that, if a single geometry is passed and the geometry is not in EPSG:4326 CRS, geo_crs argument should be given as well. The second argument is the target resolution of the data in meters. The NLCD database is multi-resolution and based on the target resolution, the source data are resampled on the server side.

You should be mindful of the resolution since higher resolutions require more memory so if your requests requires more memory than the available memory on your system the code is likely to crash. You can either increase the resolution or divide your region of interest into smaller regions.

Moreover, the MRLC GeoServer has a limit of about 8 million pixels per request but PyGeoHydro takes care of the domain decomposition under-the-hood and divides the request to smaller requests then merges them. So the only bottleneck for requests is the amount of available memory on your system.

Both nlcd_bygeom and nlcd_bycoords functions can request for four layers from the MRLC web service; imperviousness, land use/land cover, impervious descriptors, and tree canopy. Since NLCD is released every couple of years, you can specify the target year via the years argument. Layers that are not included in this argument are ignored. By default, years is {'impervious': [2019], 'cover': [2019], 'canopy': [2019], 'descriptor': [2019]}.

Furthermore, we can specify the region of interest as well via the region argument. Valid values are L48 (for CONUS), HI (for Hawaii), AK (for Alaska), and PR (for Puerto Rico). By default, region is set to L48.

Let’s get the cover and impervious descriptor data at a 100 m resolution for all three stations

desc = gh.nlcd_bygeom(geometry, 100, years={"descriptor": 2019}, ssl=False)

This function returns a dict where its keys are the indices of the input GeoDataFrame.

cmap, norm, levels = gh.plot.descriptor_legends()

desc_2019 = desc["01318500"].descriptor_2019
ax = desc_2019.where(desc_2019 < 127).plot(
    size=5, cmap=cmap, levels=levels, cbar_kwargs={"ticks": levels[:-1]}
)
ax.axes.set_title("Urban Descriptor 2019")
ax.figure.savefig("_static/descriptor.png", bbox_inches="tight", facecolor="w")

Now let’s get the land cover data:

lulc = gh.nlcd_bygeom(geometry, 100, years={"cover": [2016, 2019]}, ssl=False)

Moreover, we can get the statistics of the cover data for each class or category:

stats = gh.cover_statistics(lulc["01318500"].cover_2019)
stats

Stats(classes={'Open Water': 3.218275304968691, 'Developed, Open Space': 1.6128251635756874, 'Developed, Low Intensity': 0.5524352553716665, 'Developed, Medium Intensity': 0.2678054930087095, 'Developed, High Intensity': 0.038027458129463905, 'Barren Land (Rock/Sand/Clay)': 0.0977190439205618, 'Deciduous Forest': 41.72741457073914, 'Evergreen Forest': 29.33253744552278, 'Mixed Forest': 14.194382537791228, 'Shrub-Forest': 0.692330207399452, 'Herbaceous-Forest': 0.33994242873308644, 'Shrub/Scrub': 0.05001186917632526, 'Grassland/Herbaceous': 0.18022710458933802, 'Pasture/Hay': 0.4438841476202877, 'Cultivated Crops': 0.008296899955519398, 'Woody Wetlands': 6.641437944950068, 'Emergent Herbaceous Wetlands': 0.6024471245479918}, categories={'Water': 3.218275304968691, 'Developed': 2.471093370085527, 'Barren': 0.0977190439205618, 'Forest': 86.28660719018569, 'Shrubland': 0.05001186917632526, 'Herbaceous': 0.18022710458933802, 'Planted/Cultivated': 0.4521810475758071, 'Wetlands': 7.24388506949806})

and an estimation of the overland roughness:

roughness = gh.overland_roughness(lulc["01318500"].cover_2019)

Additionally, PyGeoHydro provides a function for getting the official legends of the cover data. Let’s plot the data using this legends.

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 4))

cmap, norm, levels = gh.plot.cover_legends()
cover = lulc["01318500"].cover_2019
cover.where(cover < 127).plot(ax=ax1, cmap=cmap, levels=levels, cbar_kwargs={"ticks": levels[:-1]})
ax1.set_title("Land Use/Land Cover 2019")
ax1.set_axis_off()

roughness.plot(ax=ax2)
ax2.set_title("Overland Roughness")
ax2.set_axis_off()
fig.savefig(Path("_static", "lulc.png"), bbox_inches="tight", facecolor="w")

Now, let’s see nlcd_bycoords in action. The coordinates must be a list of (longitude, latitude) coordinates. Let’s use osmnx package to get a street network:

G = ox.graph_from_place("Piedmont, California, USA", network_type="drive")

Now, we can get land cover and tree canopy for each node based on their coordinates and then plot the results.

x, y = nx.get_node_attributes(G, "x").values(), nx.get_node_attributes(G, "y").values()
lulc = gh.nlcd_bycoords(list(zip(x, y)), years={"cover": [2019], "canopy": [2016]})
nx.set_node_attributes(G, dict(zip(G.nodes(), lulc.cover_2019)), "cover_2019")
nx.set_node_attributes(G, dict(zip(G.nodes(), lulc.canopy_2016)), "canopy_2016")

lulc.head()

	geometry	cover_2019	canopy_2016
0	POINT (-122.24760 37.82625)	23	27.0
1	POINT (-122.24719 37.82422)	22	21.0
2	POINT (-122.24611 37.82490)	23	20.0
3	POINT (-122.24534 37.82541)	23	24.0
4	POINT (-122.24447 37.82595)	23	25.0

nc = ox.plot.get_node_colors_by_attr(G, "canopy_2016", cmap="viridis_r")
fig, ax = ox.plot_graph(
    G,
    node_color=nc,
    node_size=20,
    save=True,
    bgcolor="w",
)

Next, let’s get the data for a larger area at a coarser resolution. For example, we get the Upper Hudson subregion boundary using pygeohydro.WBD, then get the cover data within the area.

wbd = gh.WBD("huc4")
hud = wbd.byids("huc4", ["0202", "0203", "0204", "0205", "0206", "0207", "0208"])
hud = hud.set_index("huc4")
lulc = gh.nlcd_bygeom(hud, 1000, years={"cover": [2019]})

cmap, norm, levels = gh.plot.cover_legends()
fig, ax = plt.subplots(1, 1, figsize=(7, 7))
xmin, ymin, xmax, ymax = hud.unary_union.bounds

cover = {h: da.cover_2019.where(da.cover_2019 < 127) for h, da in lulc.items()}

cover["0202"].plot(ax=ax, cmap=cmap, levels=levels, cbar_kwargs={"ticks": levels[:-1]})
_ = cover.pop("0202")
for da in cover.values():
    da.plot(ax=ax, cmap=cmap, levels=levels, add_colorbar=False)
hud.plot(ax=ax, facecolor="none", edgecolor="k", linewidth=0.8)
ax.set_xlim(xmin, xmax)
ax.set_ylim(ymin, ymax)
ax.set_axis_off()
ax.set_title("Land Use/Land Cover 2019")
ax.margins(0)

This page was generated from nldas.ipynb. Interactive online version:

NLDAS2 Forcing

from pathlib import Path

import pynldas2 as nldas
from pygeohydro import WBD

The NLDAS2 database provides forcing data at 1/8th-degree grid spacing and range from 01 Jan 1979 to present. Let’s take a look at NLDAS2 grid mask that includes land, water, soil, and vegetation masks:

grid = nldas.get_grid_mask()
grid

<xarray.Dataset>
Dimensions:      (lon: 464, lat: 224, time: 1, bnds: 2)
Coordinates:
  * lon          (lon) float32 -124.9 -124.8 -124.7 ... -67.31 -67.19 -67.06
  * lat          (lat) float32 25.06 25.19 25.31 25.44 ... 52.69 52.81 52.94
  * time         (time) datetime64[ns] 2000-01-01
    spatial_ref  int64 0
Dimensions without coordinates: bnds
Data variables:
    time_bnds    (time, bnds) datetime64[ns] ...
    NLDAS_mask   (time, lat, lon) float32 ...
    CONUS_mask   (time, lat, lon) float32 ...
    NLDAS_veg    (time, lat, lon) float32 ...
    NLDAS_soil   (time, lat, lon) float32 ...
Attributes: (12/13)
    missing_value:          -9999.0
    time_definition:        constant
    title:                  NLDAS masks and predominant vegetation/soil
    institution:            NASA GSFC
    history:                created on date: Fri Mar  8 15:58:50 2019
    references:             Mitchell_etal_JGR_2004; Xia_etal_JGR_2012
    ...                     ...
    website:                https://ldas.gsfc.nasa.gov/nldas/
    MAP_PROJECTION:         EQUIDISTANT CYLINDRICAL
    SOUTH_WEST_CORNER_LAT:  25.0625
    SOUTH_WEST_CORNER_LON:  -124.9375
    DX:                     0.125
    DY:                     0.125

xarray.Dataset

• Dimensions:
  • lon: 464
  • lat: 224
  • time: 1
  • bnds: 2
• Coordinates: (4)
  • lon
    (lon)
    float32
    -124.9 -124.8 ... -67.19 -67.06

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    vmin :

    -124.9375

    vmax :

    -67.0625

    axis :

    X

    array([-124.9375, -124.8125, -124.6875, ...,  -67.3125,  -67.1875,  -67.0625],
          dtype=float32)

  • lat
    (lat)
    float32
    25.06 25.19 25.31 ... 52.81 52.94

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    vmin :

    25.0625

    vmax :

    52.9375

    axis :

    Y

    array([25.0625, 25.1875, 25.3125, ..., 52.6875, 52.8125, 52.9375],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    2000-01-01

    long_name :

    time

    time_increment :

    1

    bounds :

    time_bnds

    array(['2000-01-01T00:00:00.000000000'], dtype='datetime64[ns]')

  • spatial_ref
    ()
    int64
    0

    crs_wkt :

    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]

    semi_major_axis :

    6378137.0

    semi_minor_axis :

    6356752.314245179

    inverse_flattening :

    298.257223563

    reference_ellipsoid_name :

    WGS 84

    longitude_of_prime_meridian :

    0.0

    prime_meridian_name :

    Greenwich

    geographic_crs_name :

    WGS 84

    horizontal_datum_name :

    World Geodetic System 1984

    grid_mapping_name :

    latitude_longitude

    spatial_ref :

    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]

    GeoTransform :

    -125.0 0.125 0.0 25.0 0.0 0.125

    array(0)

• Data variables: (5)
  • time_bnds
    (time, bnds)
    datetime64[ns]
    ...

    [2 values with dtype=datetime64[ns]]

  • NLDAS_mask
    (time, lat, lon)
    float32
    ...

    units :

    0 = water; 1 = land

    standard_name :

    land mask

    long_name :

    NLDAS UMD unified land mask

    cell_methods :

    time: mean

    vmin :

    0.0

    vmax :

    1.0

    [103936 values with dtype=float32]

  • CONUS_mask
    (time, lat, lon)
    float32
    ...

    units :

    0 = water; 1 = land

    standard_name :

    CONUS mask

    long_name :

    NLDAS CONUS mask

    cell_methods :

    time: mean

    vmin :

    0.0

    vmax :

    1.0

    [103936 values with dtype=float32]

  • NLDAS_veg
    (time, lat, lon)
    float32
    ...

    units :

    0 - 13

    standard_name :

    vegetation class

    long_name :

    NLDAS UMD predominant vegetation class

    cell_methods :

    time: mean

    vmin :

    0.0

    vmax :

    13.0

    [103936 values with dtype=float32]

  • NLDAS_soil
    (time, lat, lon)
    float32
    ...

    units :

    0 - 14

    standard_name :

    soil texture type

    long_name :

    NLDAS STATSGO predominant soil texture type

    cell_methods :

    time: mean

    vmin :

    0.0

    vmax :

    14.0

    [103936 values with dtype=float32]

• Indexes: (3)
  • lon
    PandasIndex

    PandasIndex(Index([-124.9375, -124.8125, -124.6875, -124.5625, -124.4375, -124.3125,
           -124.1875, -124.0625, -123.9375, -123.8125,
           ...
            -68.1875,  -68.0625,  -67.9375,  -67.8125,  -67.6875,  -67.5625,
            -67.4375,  -67.3125,  -67.1875,  -67.0625],
          dtype='float32', name='lon', length=464))

  • lat
    PandasIndex

    PandasIndex(Index([25.0625, 25.1875, 25.3125, 25.4375, 25.5625, 25.6875, 25.8125, 25.9375,
           26.0625, 26.1875,
           ...
           51.8125, 51.9375, 52.0625, 52.1875, 52.3125, 52.4375, 52.5625, 52.6875,
           52.8125, 52.9375],
          dtype='float32', name='lat', length=224))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2000-01-01'], dtype='datetime64[ns]', name='time', freq=None))

• Attributes: (13)

  missing_value :

  -9999.0

  time_definition :

  constant

  title :

  NLDAS masks and predominant vegetation/soil

  institution :

  NASA GSFC

  history :

  created on date: Fri Mar 8 15:58:50 2019

  references :

  Mitchell_etal_JGR_2004; Xia_etal_JGR_2012

  conventions :

  CF-1.6

  website :

  https://ldas.gsfc.nasa.gov/nldas/

  MAP_PROJECTION :

  EQUIDISTANT CYLINDRICAL

  SOUTH_WEST_CORNER_LAT :

  25.0625

  SOUTH_WEST_CORNER_LON :

  -124.9375

  DX :

  0.125

  DY :

  0.125

For example, let’s plot the vegetation mask.

ax = grid.NLDAS_veg.plot()
ax.figure.savefig(Path("_static", "nldas_grid.png"), facecolor="w", bbox_inches="tight")

Next, we use PyGeoHydro to get the geometry of a HUC8 with ID of 1306003:

huc8 = WBD("huc8")
geometry = huc8.byids("huc8", "13060003").geometry[0]

PyNLDAS2 allows us to get the data for a list of coordinates using pynldas2.get_bycoords or for a region as gridded data using pynldas2.get_bygeom. Here, we use the latter. Note that if we don’t pass any variables, all variables will be downloaded. Additionally, we can pass snow=True to separate snow portion of precipitation using temperature. PyNLDAS2 provides access to NLDAS2 data from two sources: grib and netcdf. They are mostly similar so if for some reason one of the sources is not temporarly available, you can use the other one. The default source is grib.

clm = nldas.get_bygeom(geometry, "2010-01-08", "2010-01-08", 4326, variables="prcp", snow=True)

ax = clm.snow.sel(time=slice("2010-01-08T05:00:00", "2010-01-08T010:00:00")).plot(
    col="time", col_wrap=3
)
ax.fig.savefig(Path("_static", "nldas_snow.png"), facecolor="w", bbox_inches="tight")

This page was generated from noaa.ipynb. Interactive online version:

Water Levels and Tides

from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd

import async_retriever as ar

We can get water level recordings of a NOAA’s water level station through the CO-OPS API, Let’s get the data for 8534720 station (Atlantic City, NJ), during 2012,. Note that this CO-OPS product has a 31-day for a single request, so we have to break the request accordingly.

station_id = "8534720"
start = pd.to_datetime("2012-01-01")
end = pd.to_datetime("2012-12-31")

s = start
dates = []
for e in pd.date_range(start, end, freq="m"):
    dates.append((s.date(), e.date()))
    s = e + pd.offsets.MonthBegin()

url = "https://api.tidesandcurrents.noaa.gov/api/prod/datagetter"

urls, kwds = zip(
    *(
        (
            url,
            {
                "params": {
                    "product": "water_level",
                    "application": "web_services",
                    "begin_date": f'{s.strftime("%Y%m%d")}',
                    "end_date": f'{e.strftime("%Y%m%d")}',
                    "datum": "MSL",
                    "station": f"{station_id}",
                    "time_zone": "GMT",
                    "units": "metric",
                    "format": "json",
                }
            },
        )
        for s, e in dates
    )
)

resp = ar.retrieve(urls, "json", request_kwds=kwds)
wl_list = []
for rjson in resp:
    wl = pd.DataFrame.from_dict(rjson["data"])
    wl["t"] = pd.to_datetime(wl.t)
    wl = wl.set_index(wl.t).drop(columns="t")
    wl["v"] = pd.to_numeric(wl.v, errors="coerce")
    wl_list.append(wl)
water_level = pd.concat(wl_list).sort_index()
water_level.attrs = rjson["metadata"]

ax = water_level.v.plot(figsize=(15, 5))
ax.set_xlabel("")
ax.figure.savefig("_static/water_level.png", bbox_inches="tight", facecolor="w")

Now, let’s see an example without any payload or headers. Here’s how we can retrieve harmonic constituents from CO-OPS:

stations = [
    "8410140",
    "8411060",
    "8413320",
    "8418150",
    "8419317",
    "8419870",
    "8443970",
    "8447386",
]

base_url = "https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations"
urls = [f"{base_url}/{i}/harcon.json?units=metric" for i in stations]
resp = ar.retrieve(urls, "json")

amp_list = []
phs_list = []
for rjson in resp:
    sid = rjson["self"].rsplit("/", 2)[1]
    const = pd.DataFrame.from_dict(rjson["HarmonicConstituents"]).set_index("name")
    amp = const.rename(columns={"amplitude": sid})[sid]
    phase = const.rename(columns={"phase_GMT": sid})[sid]
    amp_list.append(amp)
    phs_list.append(phase)

amp = pd.concat(amp_list, axis=1)
phs = pd.concat(phs_list, axis=1)

const = [
    "M2",
    "S2",
    "N2",
    "K1",
    "O1",
    "S1",
    "M1",
    "J1",
    "Q1",
    "T2",
    "R2",
    "P1",
    "L2",
    "K2",
]

ax = plt.subplot(111, projection="polar")
ax.figure.set_size_inches(5, 5)
for i in const:
    phs_ord = phs.loc[i].sort_values()
    amp_ord = amp.loc[i][phs_ord.index]
    ax.scatter(phs_ord, amp_ord, s=amp_ord * 200, cmap="hsv", alpha=0.75, label=i)

ax.legend(loc="upper left", bbox_to_anchor=(1.05, 1))
ax.figure.savefig(Path("_static", "tides.png"), bbox_inches="tight", facecolor="w")

This page was generated from nwis.ipynb. Interactive online version:

River Discharge

import pandas as pd
import xarray as xr

import pydaymet as daymet
import pygeohydro as gh
from pygeohydro import NWIS, plot

_ = xr.set_options(display_expand_attrs=False)

We can explore the available NWIS stations within a bounding box using interactive_map function. It returns an interactive map and by clicking on an station some of the most important properties of stations are shown so you can decide which station(s) to choose from.

bbox = (-69.5, 45, -69, 45.5)

nwis_kwds = {"hasDataTypeCd": "dv", "outputDataTypeCd": "dv", "parameterCd": "00060"}
gh.interactive_map(bbox, nwis_kwds=nwis_kwds)

Make this Notebook Trusted to load map: File -> Trust Notebook

We can get more information about these stations using the get_info function from NWIS class.

nwis = NWIS()
query = {
    "bBox": ",".join(f"{b:.06f}" for b in bbox),
    "hasDataTypeCd": "dv",
    "outputDataTypeCd": "dv",
}
info_box = nwis.get_info(query)

Now, let’s select all the stations that their daily mean streamflow data are available between 2000-01-01 and 2010-12-31.

dates = ("2000-01-01", "2010-12-31")
stations = info_box[
    (info_box.begin_date <= dates[0]) & (info_box.end_date >= dates[1])
].site_no.tolist()

One of the useful information in the database in Hydro-Climatic Data Network - 2009 (HCDN-2009) flag. This flag shows whether the station is natural (True) or affected by human activities (False). If an station is not available in the HCDN dataset None is returned.

query = {
    "site": ",".join(stations),
    "hasDataTypeCd": "dv",
    "outputDataTypeCd": "dv",
}
info = nwis.get_info(query, expanded=True)
info.set_index("site_no").hcdn_2009

site_no
01031450           False
01031500            True
01031500            True
01031500            True
01031500            True
01033000           False
451031069185301    False
451105069270801    False
Name: hcdn_2009, dtype: bool

We can get the daily mean streamflow for these stations using the get_streamflow function. This function has a flag to return the data mm/day rather than the default cms which is useful for hydrolgy models and plotting hydrologic signatures.

qobs = nwis.get_streamflow(stations, dates, mmd=True)

/var/folders/jx/s06wj7hs325c_2yd2g5209sr0000gp/T/ipykernel_90548/3705647618.py:1: UserWarning: Dropped 1 stations since they don't have discharge data from 2000-01-01 to 2010-12-31.
  qobs = nwis.get_streamflow(stations, dates, mmd=True)

By default, get_streamflow returns a pandas.DataFrame that has an attrs method containing metadata for all the stations.

Moreover, we can get the same data as xarray.Dataset as follows:

qobs_ds = nwis.get_streamflow(stations, dates, mmd=True, to_xarray=True)
qobs_ds

/var/folders/jx/s06wj7hs325c_2yd2g5209sr0000gp/T/ipykernel_90548/3292066144.py:1: UserWarning: Dropped 1 stations since they don't have discharge data from 2000-01-01 to 2010-12-31.
  qobs_ds = nwis.get_streamflow(stations, dates, mmd=True, to_xarray=True)

<xarray.Dataset>
Dimensions:       (time: 4018, station_id: 2)
Coordinates:
  * time          (time) datetime64[ns] 2000-01-01T05:00:00 ... 2010-12-31T05...
  * station_id    (station_id) object 'USGS-01031450' 'USGS-01031500'
Data variables:
    discharge     (time, station_id) float64 0.7128 0.667 0.6435 ... 1.534 1.407
    station_nm    (station_id) <U44 'Kingsbury Stream At Abbot Village, Maine...
    dec_lat_va    (station_id) float64 45.18 45.17
    dec_long_va   (station_id) float64 -69.45 -69.31
    alt_va        (station_id) float64 423.0 358.5
    alt_acy_va    (station_id) float64 0.01 0.01
    alt_datum_cd  (station_id) <U6 'NGVD29' 'NGVD29'
    huc_cd        (station_id) <U8 '01020004' '01020004'
    begin_date    (station_id) <U10 '1997-07-26' '1902-10-01'
    end_date      (station_id) <U10 '2023-11-26' '2023-11-26'
Attributes: (1)

xarray.Dataset

• Dimensions:
  • time: 4018
  • station_id: 2
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    2000-01-01T05:00:00 ... 2010-12-...

    array(['2000-01-01T05:00:00.000000000', '2000-01-02T05:00:00.000000000',
           '2000-01-03T05:00:00.000000000', ..., '2010-12-29T05:00:00.000000000',
           '2010-12-30T05:00:00.000000000', '2010-12-31T05:00:00.000000000'],
          dtype='datetime64[ns]')

  • station_id
    (station_id)
    object
    'USGS-01031450' 'USGS-01031500'

    array(['USGS-01031450', 'USGS-01031500'], dtype=object)

• Data variables: (10)
  • discharge
    (time, station_id)
    float64
    0.7128 0.667 0.6435 ... 1.534 1.407

    units :

    mm/day

    array([[0.7127834 , 0.66699154],
           [0.64348501, 0.62589727],
           [0.59398616, 0.5911252 ],
           ...,
           [1.61366241, 1.55842099],
           [1.53446425, 1.46358807],
           [1.53446425, 1.40668832]])

  • station_nm
    (station_id)
    <U44
    'Kingsbury Stream At Abbot Villa...

    long_name :

    station name

    array(['Kingsbury Stream At Abbot Village, Maine',
           'Piscataquis River Near Dover-Foxcroft, Maine'], dtype='<U44')

  • dec_lat_va
    (station_id)
    float64
    45.18 45.17

    long_name :

    latitude

    units :

    decimal_degrees

    array([45.1844444, 45.175    ])

  • dec_long_va
    (station_id)
    float64
    -69.45 -69.31

    long_name :

    longitude

    units :

    decimal_degrees

    array([-69.4522222, -69.3147222])

  • alt_va
    (station_id)
    float64
    423.0 358.5

    long_name :

    altitude

    units :

    ft

    array([423.  , 358.47])

  • alt_acy_va
    (station_id)
    float64
    0.01 0.01

    long_name :

    altitude accuracy

    array([0.01, 0.01])

  • alt_datum_cd
    (station_id)
    <U6
    'NGVD29' 'NGVD29'

    long_name :

    altitude datum

    array(['NGVD29', 'NGVD29'], dtype='<U6')

  • huc_cd
    (station_id)
    <U8
    '01020004' '01020004'

    long_name :

    hydrologic unit code

    array(['01020004', '01020004'], dtype='<U8')

  • begin_date
    (station_id)
    <U10
    '1997-07-26' '1902-10-01'

    long_name :

    availability begin date

    array(['1997-07-26', '1902-10-01'], dtype='<U10')

  • end_date
    (station_id)
    <U10
    '2023-11-26' '2023-11-26'

    long_name :

    availability end date

    array(['2023-11-26', '2023-11-26'], dtype='<U10')

• Indexes: (2)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2000-01-01 05:00:00', '2000-01-02 05:00:00',
                   '2000-01-03 05:00:00', '2000-01-04 05:00:00',
                   '2000-01-05 05:00:00', '2000-01-06 05:00:00',
                   '2000-01-07 05:00:00', '2000-01-08 05:00:00',
                   '2000-01-09 05:00:00', '2000-01-10 05:00:00',
                   ...
                   '2010-12-22 05:00:00', '2010-12-23 05:00:00',
                   '2010-12-24 05:00:00', '2010-12-25 05:00:00',
                   '2010-12-26 05:00:00', '2010-12-27 05:00:00',
                   '2010-12-28 05:00:00', '2010-12-29 05:00:00',
                   '2010-12-30 05:00:00', '2010-12-31 05:00:00'],
                  dtype='datetime64[ns]', name='time', length=4018, freq=None))

  • station_id
    PandasIndex

    PandasIndex(Index(['USGS-01031450', 'USGS-01031500'], dtype='object', name='station_id'))

• Attributes: (1)

  tz :

  UTC

Then we can use the signatures function to plot hydrologic signatures of the streamflow. Note that the input time series should be in mm/day. This function has argument, output, for saving the plot as a png file.

plot.signatures(qobs, output="_static/example_plots_signatures.png")

This function also can show precipitation data as a bar plot. Let’s use PyDaymet to get the precipitation at the NWIS stations location.

sid = "01031500"
coords = tuple(info[info.site_no == sid][["dec_long_va", "dec_lat_va"]].to_numpy()[0])
prcp = daymet.get_bycoords(coords, dates, variables="prcp", ssl=False)
qobs.index = pd.to_datetime(qobs.index.strftime("%Y-%m-%d"))

plot.signatures(
    qobs[f"USGS-{sid}"],
    prcp,
    title=f"Streamflow Observations for USGS-{sid}",
    output="_static/nwis.png",
)

We can also get instantaneous streamflow data using get_streamflow. This method assumes that the input dates are in UTC time zone and returns the data in UTC time zone as well.

qobs = nwis.get_streamflow("01646500", ("2005-01-01 12:00", "2005-01-12 15:00"), freq="iv")
_ = qobs.plot(figsize=(7, 3.5))

This page was generated from pet_validation.ipynb. Interactive online version:

Computing PET

import calendar
from pathlib import Path

import pandas as pd
import xarray as xr

import hydrosignatures as hsg
import pydaymet as daymet
from pygeohydro import NWIS
from pynhd import WaterData

PyDaymet offers three methods for computing PET: penman_monteith, priestley_taylor, and hargreaves_samani. Let’s validate its PET estimation methods with long-term mean monthly PET measurement from TexasET Network for Houston, TX.

We choose station ID of 08071000 and time period of 2000-2009.

station = "08071000"
dates = ("2000-01-01", "2009-01-12")
root = Path("input_data")

clm_par = Path(root, f"{station}_clm_single_pixel.parquet")
clm_nc = Path(root, f"{station}_clm_basin.nc")

site = NWIS().get_info({"site": station})
coords = site[["dec_long_va", "dec_lat_va"]].iloc[0].to_list()

if clm_par.exists() and clm_nc.exists():
    clm_c = pd.read_parquet(clm_par)
    clm_b = xr.open_dataset(clm_nc, decode_coords="all", engine="h5netcdf")
else:
    variables = ("prcp", "tmin", "tmax", "srad", "dayl")
    clm_c = daymet.get_bycoords(tuple(coords), dates, variables=variables)
    clm_c.to_parquet(clm_par)

    wd = WaterData("gagesii_basins")
    basin = wd.byid("gage_id", station)
    clm_b = daymet.get_bygeom(basin.geometry[0], dates, variables=variables)
    clm_b.to_netcdf(clm_nc)

Next, we get the long-term mean monthly PET records for Houston, TX, from TexasET Network.

eto_obs = pd.Series(
    [2.36, 2.83, 4.32, 5.01, 6.11, 6.57, 6.52, 6.08, 5.57, 4.28, 2.90, 2.35],
    index=range(1, 13),
)
eto_obs.index.name = "month"
month_abbr = dict(enumerate(calendar.month_abbr))
eto_obs.index = eto_obs.index.map(month_abbr)
# convert to mm/month from in./month
eto_obs = eto_obs * 25.4

Now, let’s compute PET using three methods by PyDaymet.

single, gridded, style = {}, {}, {}
methods = {"HS": "hargreaves_samani", "PM": "penman_monteith", "PT": "priestley_taylor"}
clm_b = daymet.potential_et(clm_b, method="penman_monteith")
for n, m in methods.items():
    pet_c = daymet.potential_et(clm_c, coords, method=m)
    single[f"{n} (Single Pixel)"] = pet_c["pet (mm/day)"]
    style[f"{n} (Single Pixel)"] = "-"
    pet_b = daymet.potential_et(clm_b, method=m)
    gridded[f"{n} (Areal Average)"] = pet_b.pet.mean(dim=["x", "y"]).values.flatten()
    style[f"{n} (Areal Average)"] = "--"
style["TexasET"] = "-"

Finally, let’s compute mean monthly PET for each method and compare with the long-term mean monthly PET records. For computing mean monthly PET from daily we use HyRiver’s HydroSignatures package.

mean_month = hsg.mean_monthly(pd.DataFrame({**single, **gridded}), index_abbr=True)
mean_month["TexasET"] = eto_obs
ax = mean_month.plot(figsize=(8, 4), ylabel="PET (mm/month)", style=style)
ax.figure.savefig(
    Path("_static", "pet_validation.png"), dpi=300, bbox_inches="tight", facecolor="w"
)

The plots show that the Hargreaves-Samani method produces better results for this location. However, we should note that for arid regions, the Priestley-Taylor and Penman-Monteith methods require a correction factor to account for the fact that in arid regions, the air might not be saturated when its temperature is at its minimum. PyDaymet, has an additional parameter called arid_correction for this purpose. For more information, please refer to PyDaymet documentation. Now that we know that the Hargreaves-Samani method produces better results for this location, we can use it to compute aridity index and ensure that Houston is humid and does not require the arid correction factor.

ai = hsg.aridity_index(single["HS (Single Pixel)"], clm_c["prcp (mm/day)"])
1 / ai

0.9231323220526481

Note that compute_ai calculates PET / Prcp, so in order to use the UNEP’s aridity classification table, we need to consider the inverse of this value. Considering that 0.65 is the threshold for aridity, we can conclude that Houston is humid and there is no need for the arid correction factor.

This page was generated from pygeoapi.ipynb. Interactive online version:

Split Catchment with PyGeoAPI

import geopandas as gpd
import shapely.geometry as sgeom

import pynhd

PyGeoAPI service provides four functionalities:

1. flow_trace: Trace flow from a starting point to up/downstream direction.

2. split_catchment: Split the local catchment of a point of interest at the point’s location.

3. elevation_profile: Extract elevation profile along a LineString.

4. endpoints_profile: Extract elevation profile along a path between two points.

5. cross_section: Extract cross-section at a point of interest along a flow line.

The pygeoapi function in PyNHD requires two inputs, a geopandas.GeoDataFrame that must contain all the required inputs corresponding to target services. Let’s take a look at them in an example:

gdf = gpd.GeoDataFrame(
    {
        "direction": [
            "none",
        ]
    },
    geometry=[sgeom.Point((1774209.63, 856381.68))],
    crs="ESRI:102003",
)
trace = pynhd.pygeoapi(gdf, "flow_trace")

In the split_catchment service we can set upstream to True to get the entire upstream catchments not just the split local catchments.

gdf = gpd.GeoDataFrame(
    {
        "upstream": [
            False,
        ]
    },
    geometry=[sgeom.Point((-73.82705, 43.29139))],
    crs=4326,
)
split = pynhd.pygeoapi(gdf, "split_catchment")

ax = split.plot(figsize=(8, 8), facecolor="lightgrey", edgecolor="black", alpha=0.8)
trace.plot(ax=ax, color="b", linewidth=3.0)
ax.axis("off")
ax.set_title("Split Catchment")
ax.margins(x=0)
ax.figure.set_dpi(100)
ax.figure.savefig("_static/split_catchment.png", bbox_inches="tight", facecolor="w", dpi=100)

The elevation_profile service returns the elevation profile along a LineString at a given number of points and a specified resolution for DEM.

coords = [
    [-108.45263, 38.97755],
    [-108.4535, 38.978],
    [-108.454393, 38.977915],
    [-108.45495, 38.97837],
]
gdf = gpd.GeoDataFrame(
    {
        "numpts": [
            101,
        ],
        "dem_res": [
            1,
        ],
    },
    geometry=[sgeom.LineString(coords)],
    crs=4326,
)
profile = pynhd.pygeoapi(gdf, "elevation_profile")

ax = profile.plot(
    column="elevation",
    legend=True,
    figsize=(6, 6),
    legend_kwds={"label": "Elevation (m)", "orientation": "horizontal", "shrink": 0.4},
)
ax.set_title("Elevation profile along a line")
ax.set_axis_off()

This page was generated from rem.ipynb. Interactive online version:

Relative Elevation Model

This tutorial is inspired by this blog post, this excellent story map, and this notebook.

from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import shapely.ops as ops
import xarray as xr
import xrspatial as xs
from datashader import transfer_functions as tf
from datashader import utils as ds_utils
from datashader.colors import Greys9, inferno
from scipy.spatial import KDTree

import py3dep
import pynhd

import warnings

warnings.filterwarnings("ignore", ".*invalid value encountered in.*")

Relative Elevation Model (REM) detrends a DEM based on the water surface of a stream. It’s especially useful for visualization of floodplains. We’re going to compute REM for a segment of Carson River and visualize the results using xarray-spatial and datashader.

First, let’s check out the available DEM resolutions in our area of interest (AOI).

bbox = (-119.59, 39.24, -119.47, 39.30)
dem_res = py3dep.check_3dep_availability(bbox)
dem_res

{'1m': True,
 '3m': False,
 '5m': False,
 '10m': True,
 '30m': True,
 '60m': False,
 'topobathy': False}

We can see that Lidar (1-m), 10-m, and 30-m are available. Obviously, Lidar data gives us the best results, but it can be computational expensive. So, we’re going to set the resolution 5 m since 1-m and 10-m data are available.

res = 5
dem = py3dep.get_map("DEM", bbox, res)

fig, ax = plt.subplots(figsize=(8, 4), dpi=100)
_ = dem.plot(ax=ax, robust=True)

Next, we need to get the river’s centerline. For this purpose, first we get the flowlines within our AOI. Then, we remove all the isolated flowlines using the remove_isolated flag of pynhd.prepare_nhdplus and find the main flowline based on the minimum value of the levelpathi attribute.

wd = pynhd.WaterData("nhdflowline_network")

flw = wd.bybox(bbox)
flw = pynhd.prepare_nhdplus(flw, 0, 0, 0, remove_isolated=True)
flw = flw[flw.levelpathi == flw.levelpathi.min()].copy()

fig, ax = plt.subplots(figsize=(8, 4), dpi=100)
flw.plot(ax=ax, color="r")
_ = dem.plot(ax=ax, robust=True)

Now, we can get the elevation profile along the obtained main flowline with spacing of 10 meters.

lines = ops.linemerge(flw.geometry.tolist())
riv_dem = py3dep.elevation_profile(lines, 10, crs=flw.crs)
_ = riv_dem.plot(x="distance", figsize=(7, 3.2))

There are several methods for detrending the DEM based on the river’s elevation profile. You can check these method in this poster. Here, we’re going to use Inverse Distance Weighting method using scipy’s KDTree function and setting the number of neighbors to 200.

def idw(riv_dem: xr.DataArray, dem: xr.DataArray, n_nb: int) -> xr.DataArray:
    """Interpolate grid DEM from river DEM using Inverse Distance Weighting."""
    riv_coords = np.column_stack((riv_dem.x, riv_dem.y))
    kdt = KDTree(riv_coords)

    dem_grid = np.dstack(np.meshgrid(dem.x, dem.y)).reshape(-1, 2)
    distances, idx = kdt.query(dem_grid, k=n_nb)

    weights = np.reciprocal(distances)
    weights = weights / weights.sum(axis=1, keepdims=True)

    interp = weights * riv_dem.to_numpy()[idx]
    interp = interp.sum(axis=1).reshape((dem.sizes["y"], dem.sizes["x"]))
    return xr.DataArray(interp, dims=("y", "x"), coords={"x": dem.x, "y": dem.y})


elevation = idw(riv_dem, dem, 200)
rem = dem - elevation

fig, ax = plt.subplots(figsize=(8, 4), dpi=100)
elevation.plot(ax=ax)
_ = flw.plot(ax=ax, color="red")

Let’s use datashader to stack DEM, Hillshade and REM for a nice visualization. There are a couple of parameters in this tutorial that can be change for extending it to other regions: Number of neighbors in IDW, DEM resolution, span argument of REM’s shading operation.

illuminated = xs.hillshade(dem, angle_altitude=10, azimuth=90)
tf.Image.border = 0
img = tf.stack(
    tf.shade(dem, cmap=Greys9, how="linear"),
    tf.shade(illuminated, cmap=["black", "white"], how="linear", alpha=180),
    tf.shade(rem, cmap=inferno[::-1], span=[0, 7], how="log", alpha=200),
)
ds_utils.export_image(img[::-1], Path("_static", "rem").as_posix())

This page was generated from signatures.ipynb. Interactive online version:

Hydrological Signatures

from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import xarray as xr

import hydrosignatures as hs
import pydaymet as daymet
import pygeohydro as gh
from hydrosignatures import HydroSignatures
from pygeohydro import NWIS
from pynhd import WaterData

Let’s explore the capabilities of HydroSignatures by getting streamflow using PyGeoHydro, basin geometry using PyNHD and precipitation using PyDaymet. In this example, we select West Branch Herring Run At Idlewylde, MD, as the watershed of interest and compute the hydrological signatures for the period from 2010 to 2020.

site = "01585200"
start = "2010-01-01"
end = "2020-12-31"

First, we get the basin geometry of the watershed using gagesii_basins layer of the USGS’s WaterData web service.

wd = WaterData("gagesii_basins")
geometry = wd.byid("gage_id", site).geometry[0]

Then, we obtain the station’s info and streamflow data using NWIS. Note that we should convert the streamflow from cms to mm/day.

nwis = NWIS()
info = nwis.get_info({"site": site}, nhd_info=True)
area_sqm = info.nhd_areasqkm.values[0] * 1e6
q_cms = nwis.get_streamflow(site, (start, end))
q_mmpd = q_cms.squeeze() * (24.0 * 60.0 * 60.0) / area_sqm * 1e3
q_mmpd.index = pd.to_datetime(q_mmpd.index.date)

Next, we retrieve the precipitation data using PyDaymet over the whole basin using the basin geometry and take its mean as the basin’s precipitation.

prcp = daymet.get_bygeom(geometry, (start, end), variables="prcp")
p_mmpd = prcp.prcp.mean(dim=["x", "y"]).to_pandas()
p_mmpd.index = pd.to_datetime(p_mmpd.index.date)
q_mmpd = q_mmpd.loc[p_mmpd.index]

Now, we can pass these two to the HydroSignatures class:

sig = HydroSignatures(q_mmpd, p_mmpd)

The values property of this class contains the computed signatures. For example, let’s plot the regime curves:

fig, ax = plt.subplots(figsize=(6, 4), dpi=100)
ax.set_xlabel("")
ax.set_ylabel("")
sig.values.mean_monthly.plot(ax=ax)
fig.savefig(Path("_static", "signatures_rc.png"))

Note that, you can also use the functions directly. For example, let’s get streamflow observations for another station and separate the baseflow using various filter parameters and compare them:

q = nwis.get_streamflow("12304500", ("2019-01-01", "2019-12-31"))
q = q.rename(columns={"USGS-12304500": "Streamflow"})
alpha = np.arange(0.9, 1, 0.01)
qb = pd.DataFrame({round(a, 2): hs.baseflow(q.squeeze(), alpha=a) for a in alpha})

fig, ax = plt.subplots(figsize=(8, 4), dpi=100)
ax.set_xlabel("")
ax.set_ylabel("Q (mm/day)")
q.plot(ax=ax, c="k", ls="--", lw=0.9)
qb.plot(ax=ax, lw=0.8)
fig.savefig(Path("_static", "signatures_bf.png"))

Lastly, let’s compute Markham’s seasonality index for all streamflow time series of the stations in the CAMELS dataset. We retrieve the CAMELS dataset using PyGeoHydro:

camels_basin, camels_qobs = gh.get_camels()

Since compute_si_markham function accepts pandas.DataFrame or pandas.Series, we need to convert the streamflows from xarray.Dataset to pandas.DataFrame. Some stations have negative discharge values that we need to replace with zero.

discharge = camels_qobs.discharge.dropna("station_id")
discharge = xr.where(discharge < 0, 0, discharge)
camels_basin = camels_basin.loc[discharge.station_id]

Finally, we compute the seasonality index and merge it with the camels_basin dataframe for visualization. We can also use get_us_states from PyGeoHydro to add CONUS to the plot.

camels_basin = camels_basin.merge(
    hs.seasonality_index_markham(discharge.to_pandas()), left_index=True, right_index=True
)

conus = gh.helpers.get_us_states("contiguous")

fig, ax = plt.subplots(figsize=(10, 7), dpi=100)
conus.to_crs(5070).plot(ax=ax, facecolor="none", edgecolor="k", lw=0.3, alpha=0.5)
camels_basin.to_crs(5070).plot(
    column="seasonality_index", cmap="viridis", ax=ax, legend=True, legend_kwds={"shrink": 0.5}
)
ax.set_axis_off()

This page was generated from soil.ipynb. Interactive online version:

Soil Storage Capacity

from pathlib import Path

import matplotlib.pyplot as plt
import numpy as np

import pygeohydro as gh
import pygeoutils as geoutils
from pynhd import NLDI

Some hydrological models have a parameter related the max soil storage capacity of watersheds. A simple way of estimating this parameter is to use porosity and thickness properties of a watershed soil. In this tutorial, we retrieve these data from two soil datasets that PyGeoHydro supports. PyGeoHydro has two functions for getting soil data:

• soil_gnatsgo: You can access all soil properties that are available on gNATSGO dataset

• soil_properties: Porosity, available water capacity, and field capacity properties are available through this function.

Let’s start by getting the basin geometry of the USGS station 11092450 using PyNHD:

basin = NLDI().get_basins("11092450")
geom = basin.geometry[0]

Next, we get the soil porosity. Note that soil_properties return the requested property for the whole US. But, thanks to dask and xarray, the dataset is loaded lazily, so we don’t have to worry about memory usage. This function downloads the requested dataset(s) each of which is around 300 MB, so the first time that you run this function it can take a few minutes, depending on your internet speed. However, after it gets downloaded once, next calls to this function uses the downloaded file, so it will be quick. By default, downloaded files are saved to ./cache folder, but you can specify a custom directory to save the zip files.

If you run this function without any arguments, it downloads all three soil properties and returns them as a single dataset. You can pass one or more of soil properties: por for porosity, awc for available water capacity, and fc for field capacity.

soil = gh.soil_properties("por")

Now, we can mask this large dataset to our basin geometry using PyGeoUtils.

soil = geoutils.xarray_geomask(soil, geom, basin.crs)
soil = soil.where(soil.porosity > soil.porosity.rio.nodata)
soil["porosity"] = soil.porosity.rio.write_nodata(np.nan)
_ = soil.porosity.plot()

Now that we have the porosity, we need to get soil thickness. We use Microsoft’s Planetary Computer service using soil_gnatsgo function. This function has two additional dependencies that need to be installed: pystac-client and planetary-computer. They can be installed using conda or mamba as follows:

mamba -c conda-forge install pystac-client planetary-computer

We use tk0_999a later to get the total soil thickness. You can check out the full description the layer and other available layers from the dataset’s webpage. Also, we convert the thickness unit from cm to mm.

thickness = gh.soil_gnatsgo("tk0_999a", geom, basin.crs).tk0_999a
thickness = thickness.where(thickness < 2e6, drop=False) * 10
thickness = thickness.rio.write_nodata(np.nan)
thickness.attrs["units"] = "mm"
thickness.attrs["long_name"] = "soil thickness"
thickness.name = "thickness"

Using thickness.rio.resolution() and soil.rio.resolution() we can see that the resolution of the thickness dataset is 10 m whereas that of the porosity is around 90 m. So, we should resample the dataset with finer resolution, thickness, to the coarser one. For this purpose we use rioxarray.reproject_match function. For resampling, we use the weighted averaging function of rasterio. Therefore, we need to 5 as the resampling method. The full list of supported resampling operations can be found on rasterio webpage.

soil["thickness"] = thickness.rio.reproject_match(soil, resampling=5)
_ = soil["thickness"].plot()

Considering that porosity is in mm/m and thickness is in mm, we can compute the storage capacity as follows:

soil["storage"] = soil.porosity * soil.thickness * 1e-3
soil["storage"] = soil["storage"].rio.write_nodata(np.nan)
soil["storage"].attrs["units"] = "mm"
soil["storage"].attrs["long_name"] = "soil storage capacity"
soil["storage"].plot()
plt.savefig(Path("_static", "soil_storage.png"), dpi=150)

This page was generated from ssebop.ipynb. Interactive online version:

Actual Evapotranspiration

from pathlib import Path

import networkx as nx
import osmnx as ox
import pandas as pd

import pygeohydro as gh
from pynhd import NLDI

The daily actual evapotranspiration can be retrieved from SEEBop database. Note that since this service does not offer a web service and data are available as raster files on the server, so this function is not as fast as other functions and download speed might be the bottleneck.

You can get the actual ET for position-based requests using ssebopeta_bycoords and for geometry-based requests using ssebopeta_bygeom.

Now, let’s see ssebopeta_bycoords in action. The coordinates must be a dataframe with three columns: id, x, and y. Let’s use osmnx package to get a street network:

G = ox.graph_from_place("Piedmont, California, USA", network_type="drive")

Now, we can get land cover and tree canopy for each node based on their coordinates and then plot the results.

dates = ("2005-10-01", "2005-10-05")
x, y = nx.get_node_attributes(G, "x"), nx.get_node_attributes(G, "y")
coords = pd.DataFrame({"id": x.keys(), "x": x.values(), "y": y.values()})
ds = gh.ssebopeta_bycoords(coords, dates=dates)

The function returns a xarray.Dataset:

ds

<xarray.Dataset>
Dimensions:      (time: 5, location_id: 348)
Coordinates:
  * time         (time) datetime64[ns] 2005-10-01 2005-10-02 ... 2005-10-05
  * location_id  (location_id) int64 53017091 53018397 ... 9656853491 9656853494
Data variables:
    eta          (time, location_id) float64 0.551 0.551 0.551 ... 0.551 0.551
    x            (location_id) float64 -122.2 -122.2 -122.2 ... -122.2 -122.2
    y            (location_id) float64 37.83 37.82 37.82 ... 37.82 37.82 37.82

xarray.Dataset

• Dimensions:
  • time: 5
  • location_id: 348
• Coordinates: (2)
  • time
    (time)
    datetime64[ns]
    2005-10-01 ... 2005-10-05

    array(['2005-10-01T00:00:00.000000000', '2005-10-02T00:00:00.000000000',
           '2005-10-03T00:00:00.000000000', '2005-10-04T00:00:00.000000000',
           '2005-10-05T00:00:00.000000000'], dtype='datetime64[ns]')

  • location_id
    (location_id)
    int64
    53017091 53018397 ... 9656853494

    array([  53017091,   53018397,   53018399, ..., 9231102999, 9656853491,
           9656853494])

• Data variables: (3)
  • eta
    (time, location_id)
    float64
    0.551 0.551 0.551 ... 0.551 0.551

    units :

    mm/day

    long_name :

    Actual ET

    nodatavals :

    (nan,)

    array([[0.551, 0.551, 0.551, ..., 0.551, 0.551, 0.551],
           [0.551, 0.551, 0.551, ..., 0.551, 0.551, 0.551],
           [0.551, 0.551, 0.551, ..., 0.551, 0.551, 0.551],
           [0.551, 0.551, 0.551, ..., 0.551, 0.551, 0.551],
           [0.551, 0.551, 0.551, ..., 0.551, 0.551, 0.551]])

  • x
    (location_id)
    float64
    -122.2 -122.2 ... -122.2 -122.2

    crs :

    EPSG:4326

    array([-122.2476037, -122.247194 , -122.2460801, -122.2453167,
           -122.2444669, -122.2479514, -122.2469627, -122.2303409,
           -122.2307834, -122.229516 , -122.2313266, -122.2323445,
           -122.2332423, -122.2348794, -122.2321195, -122.2322134,
           -122.2315938, -122.2295143, -122.2282824, -122.227674 ,
           -122.226868 , -122.2268764, -122.2270828, -122.2400125,
           -122.2392631, -122.2304996, -122.2151674, -122.2148121,
           -122.2384621, -122.2376286, -122.2252862, -122.2255209,
           -122.2294839, -122.2493328, -122.2139202, -122.2135729,
           -122.2403954, -122.2383402, -122.2377379, -122.2364161,
           -122.2402466, -122.2395479, -122.2373988, -122.2368123,
           -122.2263838, -122.2264641, -122.2184569, -122.2170302,
           -122.2256904, -122.2252276, -122.2252192, -122.2252259,
           -122.2252231, -122.2252253, -122.2252349, -122.226454 ,
           -122.2277511, -122.2470216, -122.2453148, -122.2292657,
           -122.2280234, -122.2244134, -122.2285133, -122.2285844,
           -122.2287105, -122.2191991, -122.2377687, -122.2367415,
           -122.2356666, -122.23496  , -122.234268 , -122.2332769,
           -122.2322968, -122.2313252, -122.2460947, -122.2441099,
           -122.2157668, -122.2163966, -122.2182735, -122.2455075,
    ...
           -122.225125 , -122.2254863, -122.2284781, -122.2283265,
           -122.2293315, -122.2303711, -122.237522 , -122.222832 ,
           -122.2234677, -122.2231941, -122.2236874, -122.2174245,
           -122.2155483, -122.2154558, -122.2152171, -122.2138585,
           -122.2143266, -122.2150145, -122.2239683, -122.2441145,
           -122.2437901, -122.2200048, -122.2170155, -122.2172113,
           -122.2172435, -122.2173427, -122.2179964, -122.2186181,
           -122.2252103, -122.2360598, -122.2405574, -122.2323471,
           -122.2320681, -122.2288842, -122.2218174, -122.2427311,
           -122.2429065, -122.2346781, -122.239482 , -122.231403 ,
           -122.215793 , -122.2296797, -122.228907 , -122.2447022,
           -122.2375576, -122.235016 , -122.2251515, -122.2294481,
           -122.2306964, -122.2303224, -122.2295148, -122.2156677,
           -122.2138784, -122.2331152, -122.2369394, -122.2411059,
           -122.2442009, -122.2158366, -122.2290392, -122.2293633,
           -122.2294711, -122.2409528, -122.2273591, -122.2274198,
           -122.2224286, -122.2364855, -122.2358275, -122.2414469,
           -122.2335275, -122.2391344, -122.2150595, -122.2251306,
           -122.219283 , -122.2429793, -122.2191302, -122.2403946,
           -122.2438277, -122.2413041, -122.2185562, -122.2183304])

  • y
    (location_id)
    float64
    37.83 37.82 37.82 ... 37.82 37.82

    crs :

    EPSG:4326

    array([37.8262501, 37.8242243, 37.8249179, 37.8254055, 37.8259478,
           37.823282 , 37.8239377, 37.8210859, 37.8221537, 37.8210912,
           37.8209662, 37.8207744, 37.8198958, 37.8238898, 37.8248262,
           37.8271461, 37.8272923, 37.8220724, 37.8220912, 37.8220836,
           37.8217266, 37.8212368, 37.8200158, 37.8184846, 37.8178393,
           37.8246486, 37.8122894, 37.8127499, 37.8194317, 37.8185718,
           37.8269588, 37.8287617, 37.8235785, 37.8236661, 37.8183966,
           37.8168525, 37.8260613, 37.8258729, 37.8245781, 37.8230529,
           37.8279374, 37.8287933, 37.8263178, 37.8248969, 37.8236878,
           37.82418  , 37.8156228, 37.8158031, 37.8211924, 37.8200253,
           37.8188728, 37.8185338, 37.8171025, 37.8165846, 37.8152545,
           37.8172015, 37.8172157, 37.821706 , 37.8210743, 37.8278325,
           37.8287374, 37.8269448, 37.8265169, 37.8268347, 37.8276842,
           37.8144208, 37.8288915, 37.8288333, 37.8287941, 37.8287615,
           37.8287657, 37.8291057, 37.8294433, 37.8297895, 37.8219782,
           37.8207612, 37.8216046, 37.8225273, 37.8186816, 37.8275098,
           37.8265536, 37.8240465, 37.8234806, 37.8307469, 37.831706 ,
           37.8181099, 37.8188165, 37.8188938, 37.8174977, 37.8166783,
           37.8250561, 37.8246094, 37.8256122, 37.8239692, 37.819794 ,
           37.8219943, 37.8220071, 37.8219975, 37.8171491, 37.8252024,
    ...
           37.8188336, 37.817708 , 37.8174718, 37.8154853, 37.8144439,
           37.8305302, 37.8177515, 37.8215772, 37.8191324, 37.823533 ,
           37.8217496, 37.8144682, 37.814856 , 37.8200198, 37.8198802,
           37.8201368, 37.8219929, 37.8218178, 37.8224344, 37.8224431,
           37.8196852, 37.8191121, 37.8189988, 37.8183045, 37.8212835,
           37.8273052, 37.8270584, 37.8271061, 37.8258721, 37.8186668,
           37.8188712, 37.8190513, 37.8190355, 37.8185311, 37.8169939,
           37.8125359, 37.8143421, 37.8286865, 37.8279836, 37.8140868,
           37.8139809, 37.8142733, 37.8147257, 37.8147924, 37.8177223,
           37.818045 , 37.8299617, 37.8202634, 37.8286228, 37.8172303,
           37.8172705, 37.8220821, 37.8177198, 37.8252141, 37.8251308,
           37.8198618, 37.8207764, 37.8212141, 37.818041 , 37.8251455,
           37.8312053, 37.8223909, 37.8183726, 37.8193909, 37.8268344,
           37.8295035, 37.824906 , 37.8210034, 37.8210021, 37.8156326,
           37.8140654, 37.8171444, 37.8223774, 37.8204642, 37.8284981,
           37.8125072, 37.8308162, 37.82956  , 37.8296606, 37.8202652,
           37.822033 , 37.8222571, 37.8177488, 37.8227966, 37.8216939,
           37.8284998, 37.8243487, 37.8269485, 37.8128516, 37.8309743,
           37.8236695, 37.8243212, 37.8236278, 37.828589 , 37.828173 ,
           37.8202085, 37.8154516, 37.8154062])

• Attributes: (0)

Let’s add the average value of obtained ETAs to the street nodes:

eta_m = ds.mean(dim="time").eta
eta_dict = {sid: eta_m.sel(location_id=sid).item() for sid in eta_m.location_id.values}
nx.set_node_attributes(G, eta_dict, "eta")

nc = ox.plot.get_node_colors_by_attr(G, "eta", cmap="viridis")
fig, ax = ox.plot_graph(
    G,
    node_color=nc,
    node_size=20,
    save=True,
    bgcolor="w",
)

Now, we get a watershed geometry using NLDI and then get the actual ET within its geometry.

geometry = NLDI().get_basins("01315500").geometry[0]

eta = gh.ssebopeta_bygeom(geometry, dates=dates)

ax = eta.isel(time=4).plot(size=5)
date = eta.isel(time=4).time.dt.strftime("%Y-%m-%d").values
ax.axes.set_title(f"Actual Evapotranspiration ({date})")
ax.figure.savefig("_static/eta.png", bbox_inches="tight", facecolor="w")
ax.figure.savefig(Path("_static", "ssebop.png"), dpi=300, bbox_inches="tight", facecolor="w")

This page was generated from things.ipynb. Interactive online version:

SensorThings

from pygeohydro import SensorThings

SensorThings is a recently released USGS web service that provides access to real-time USGS sensors. For more information about the service you can check out this to the service and also this which is overview or different USGS APIs. PyGeoHydro provides access to the Things endpoint which is “one of the most important components, and can be mapped to USGS Monitoring locations”.

sensor = SensorThings()

We can query information about stations using sensor_info. We can pass a single ID or list of them.

resp = sensor.sensor_info(["USGS-09380000", "USGS-09472050"])
resp

	description	@iot.id	name	properties.state	properties.active	properties.agency	properties.county	properties.country	properties.district	properties.stateFIPS	...	properties.districtCode	properties.altitudeDatum	properties.altitudeMethod	properties.hydrologicUnit	properties.altitudeAccuracy	properties.monitoringLocationUrl	properties.monitoringLocationName	properties.monitoringLocationType	properties.monitoringLocationNumber	properties.monitoringLocationAltitudeLandSurface
0	Stream	USGS-09380000	USGS-09380000	Arizona	True	U.S. Geological Survey	Coconino County	US	Arizona	US:04	...	04	North American Vertical Datum of 1988	Interpolated from Digital Elevation Model	140700061105	4.3	https://waterdata.usgs.gov/monitoring-location...	COLORADO RIVER AT LEES FERRY, AZ	Stream	09380000	3083
1	Stream	USGS-09472050	USGS-09472050	Arizona	True	U.S. Geological Survey	Pima County	US	Arizona	US:04	...	04	National Geodetic Vertical Datum of 1929	Interpolated from topographic map.	150502030503	10	https://waterdata.usgs.gov/monitoring-location...	SAN PEDRO R AT REDINGTON BRIDGE NR REDINGTON, AZ	Stream	09472050	2820.

2 rows × 23 columns

Additionally, we can get a sensor property using sensor_property.

resp = sensor.sensor_property("Datastreams", ["USGS-09380000", "USGS-09472050"])
resp

	description	@iot.id	name	observationType	phenomenonTime	resultTime	@iot.selfLink	ObservedProperty@iot.navigationLink	Sensor@iot.navigationLink	Thing@iot.navigationLink	Observations@iot.navigationLink	observedArea.type	observedArea.coordinates	properties.Thresholds	properties.ParameterCode	properties.StatisticCode	unitOfMeasurement.name	unitOfMeasurement.symbol	unitOfMeasurement.definition	properties.WebDescription
0	Specific cond at 25C / USGS-09380000-9d24cf502...	9d24cf50257a4f60b76b92e38f286cde	9d24cf50257a4f60b76b92e38f286cde	Instantaneous	2021-07-12T18:45:00.000Z/2023-02-05T03:00:00.000Z	None	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	Point	[-111.5878722, 36.8643333]	[{'Name': 'HIGH SC THRESHOLD', 'Type': 'Thresh...	00095	00011	uS/cm	uS/cm		NaN
1	None / USGS-09380000-0a10dcb7436f4af8a679a783a...	0a10dcb7436f4af8a679a783ae3d8d58	0a10dcb7436f4af8a679a783ae3d8d58	Instantaneous	2022-07-08T20:30:00.000Z/2022-07-20T19:00:00.000Z	None	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	Point	[-111.5878722, 36.8643333]	NaN	00095	00011	uS/cm	uS/cm		Starts 7/8/22
2	Discharge / USGS-09380000-a62122d8ff094125b63b...	a62122d8ff094125b63bb2f73410b2b4	a62122d8ff094125b63bb2f73410b2b4	Instantaneous	2021-09-16T11:00:00.000Z/2023-02-05T03:00:00.000Z	None	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	Point	[-111.5878722, 36.8643333]	NaN	00060	00011	Cubic Feet per Second	ft^3/s		NaN
3	Temperature, water / USGS-09380000-b3c374548a8...	b3c374548a8d4553868046dd9fda2582	b3c374548a8d4553868046dd9fda2582	Instantaneous	2021-07-12T20:30:00.000Z/2023-02-05T03:00:00.000Z	None	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	Point	[-111.5878722, 36.8643333]	NaN	00010	00011	Degrees Centigrade	degC		NaN
4	None / USGS-09380000-e1c4b44914ed43819cf8e2e21...	e1c4b44914ed43819cf8e2e2138e9064	e1c4b44914ed43819cf8e2e2138e9064	Instantaneous	2022-01-10T01:30:00.000Z/2023-02-05T03:00:00.000Z	None	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	Point	[-111.5878722, 36.8643333]	NaN	00045	NaN	in	in		NaN
5	None / USGS-09380000-c1d72c8390d144f78b51568a3...	c1d72c8390d144f78b51568a3755a5ad	c1d72c8390d144f78b51568a3755a5ad	Instantaneous	2022-07-08T18:30:00.000Z/2023-02-05T03:00:00.000Z	None	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	Point	[-111.5878722, 36.8643333]	NaN	00065	00011	Feet	ft		PRIMARY STAGE
6	Gage height / USGS-09472050-3c47927571274e388e...	3c47927571274e388e7f4a8fbf1049e9	3c47927571274e388e7f4a8fbf1049e9	Instantaneous	2021-07-16T22:45:00.000Z/2023-02-05T04:15:00.000Z	None	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	Point	[-110.4884183, 32.4461827]	[{'Name': 'Operational limit (maximum) Upper l...	00065	00011	Feet	ft		NaN
7	Discharge / USGS-09472050-9e9527c7d6c542cd92a5...	9e9527c7d6c542cd92a5178922712ed8	9e9527c7d6c542cd92a5178922712ed8	Instantaneous	2021-08-12T06:00:00.000Z/2023-02-05T04:15:00.000Z	None	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	https://labs.waterdata.usgs.gov/sta/v1.1/Datas...	Point	[-110.4884183, 32.4461827]	NaN	00060	00011	Cubic Feet per Second	ft^3/s		NaN

For more complex queries, we need to construct an Odata query.

odata = {
    "filter": "properties/monitoringLocationType eq 'Stream' and properties/stateFIPS eq 'US:04'",
}
df = sensor.query_byodata(odata)
df

	description	@iot.id	name	@iot.selfLink	Datastreams@iot.navigationLink	TaskingCapabilities@iot.navigationLink	HistoricalLocations@iot.navigationLink	Locations@iot.navigationLink	MultiDatastreams@iot.navigationLink	properties.state	...	properties.districtCode	properties.altitudeDatum	properties.altitudeMethod	properties.hydrologicUnit	properties.altitudeAccuracy	properties.monitoringLocationUrl	properties.monitoringLocationName	properties.monitoringLocationType	properties.monitoringLocationNumber	properties.monitoringLocationAltitudeLandSurface
0	Stream	USGS-09497700	USGS-09497700	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	North American Vertical Datum of 1988	Interpolated from topographic map.	150601030102	20	https://waterdata.usgs.gov/monitoring-location...	CIBECUE CREEK NEAR OVERGAARD, AZ	Stream	09497700	7200
1	Stream	USGS-09472050	USGS-09472050	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	National Geodetic Vertical Datum of 1929	Interpolated from topographic map.	150502030503	10	https://waterdata.usgs.gov/monitoring-location...	SAN PEDRO R AT REDINGTON BRIDGE NR REDINGTON, AZ	Stream	09472050	2820.
2	Stream	USGS-09424900	USGS-09424900	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	National Geodetic Vertical Datum of 1929	Interpolated from topographic map.	150302030506	20	https://waterdata.usgs.gov/monitoring-location...	SANTA MARIA RIVER NEAR BAGDAD, AZ	Stream	09424900	1360
3	Stream	USGS-09491980	USGS-09491980	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	North American Vertical Datum of 1988	Interpolated from Digital Elevation Model	150601020207	1.6	https://waterdata.usgs.gov/monitoring-location...	N-FORK WHITE RIVER BLW GOLD GULCH AT WHITERIVE...	Stream	09491980	5270
4	Stream	USGS-09537200	USGS-09537200	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	National Geodetic Vertical Datum of 1929	Interpolated from topographic map.	150803010307	20	https://waterdata.usgs.gov/monitoring-location...	LESLIE CREEK NEAR MCNEAL, AZ.	Stream	09537200	4620.
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
187	Stream	USGS-09403850	USGS-09403850	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	National Geodetic Vertical Datum of 1929	Interpolated from topographic map.	150100031004	20	https://waterdata.usgs.gov/monitoring-location...	KANAB CREEK ABOVE THE MOUTH NEAR SUPAI, AZ	Stream	09403850	1920.
188	Stream	USGS-09429070	USGS-09429070	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	North American Vertical Datum of 1988	Interpolated from topographic map.	150301040405	20	https://waterdata.usgs.gov/monitoring-location...	CRIR LWR MAIN DRAIN BLW TYSON WW, NR EHRENBERG...	Stream	09429070	272
189	Stream	USGS-09402000	USGS-09402000	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	National Geodetic Vertical Datum of 1929	Unknown.	150200160906	20	https://waterdata.usgs.gov/monitoring-location...	LITTLE COLORADO RIVER NEAR CAMERON, AZ	Stream	09402000	3979.20
190	Stream	USGS-09535300	USGS-09535300	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	04	National Geodetic Vertical Datum of 1929	Interpolated from topographic map.	150801010512	20	https://waterdata.usgs.gov/monitoring-location...	VAMORI WASH AT KOM VO, AZ	Stream	09535300	1770.
191	Stream	USGS-09415000	USGS-09415000	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	https://labs.waterdata.usgs.gov/sta/v1.1/Thing...	Arizona	...	32	National Geodetic Vertical Datum of 1929	Level or other surveyed method.	150100100602	1.0	https://waterdata.usgs.gov/monitoring-location...	VIRGIN RV AT LITTLEFIELD, AZ	Stream	09415000	1763.68

192 rows × 29 columns

We can get geospatial data by setting the format to geojson.

expand = {"Locations": {"select": "location"}}
max_count = 1000
conditionals = "properties/monitoringLocationType eq 'Stream' and properties/stateFIPS eq 'US:04'"
odata = sensor.odata_helper(expand=expand, max_count=max_count, conditionals=conditionals)

df = sensor.query_byodata(odata, outformat="geojson")
df.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

We can even make more complex queries. For such cases we can use the odata_helper function to contruct complex Odata filters.

expand = {
    "ObservedProperty": {"select": "name,description,@iot.id"},
    "Observations": {
        "select": "result,phenomenonTime,@iot.id",
        "orderby": "phenomenonTime desc",
        "top": 1,
    },
}
odata_inner = sensor.odata_helper(expand=expand)

columns = ["properties", "@iot.id"]
conditionals = " and ".join(
    [
        "Datastreams/ObservedProperty/@iot.id eq '00060' ",
        "properties/monitoringLocationType eq 'Stream' ",
        "startswith(properties/hydrologicUnit,'15')",
    ]
)
expand = {
    "Locations": {"select": "name,description,location,@iot.id"},
    "Datastreams": {
        "select": "name,unitOfMeasurement,@iot.id",
        "filter": "ObservedProperty/@iot.id eq '00060'",
        "expand": odata_inner["expand"],
    },
}
max_count = 1000
odata = sensor.odata_helper(
    columns=columns,
    conditionals=conditionals,
    expand=expand,
    max_count=max_count,
)

df = sensor.query_byodata(odata, outformat="geojson")
df.explore()

Make this Notebook Trusted to load map: File -> Trust Notebook

This page was generated from us_coasts.ipynb. Interactive online version:

CONUS Coasts

US States and Coastlines

Census TIGER (Topologically Integrated Geographic Encoding and Referencing database)

import os
import warnings
from pathlib import Path

import geopandas as gpd

warnings.filterwarnings("ignore", message=".*initial implementation of Parquet.*")
root = Path("input_data")
os.makedirs(root, exist_ok=True)
BASE_PLOT = {"facecolor": "k", "edgecolor": "b", "alpha": 0.2, "figsize": (12, 6)}
CRS = "epsg:5070"


def tiger_shp_2022(data):
    url = f"https://www2.census.gov/geo/tiger/TIGER2022/{data.upper()}/tl_2022_us_{data}.zip"
    return gpd.read_file(url)


cfile = Path(root, "us_state.feather")
if cfile.exists():
    state = gpd.read_feather(cfile)
else:
    state = tiger_shp_2022("state")
    state.to_feather(cfile)

ax = state.to_crs(CRS).plot(**BASE_PLOT)
ax.axis("off")
ax.margins(0)

cfile = Path(root, "us_coastline.feather")
if cfile.exists():
    coastline = gpd.read_feather(cfile)
else:
    coastline = tiger_shp_2022("coastline")
    coastline.to_feather(cfile)

ax = coastline.to_crs(CRS).plot(figsize=(12, 6))
ax.axis("off")
ax.margins(0)

Clip to CONUS

from shapely.geometry import box

conus_bounds = box(-125, 24, -65, 50)

cfile = Path(root, "conus_states.feather")
if cfile.exists():
    conus_states = gpd.read_feather(cfile)
else:
    conus_states = state[state.within(conus_bounds)]
    conus_states.to_feather(cfile)

ax = conus_states.to_crs(CRS).plot(**BASE_PLOT)
ax.axis("off")
ax.margins(0)

conus_coastline = coastline[coastline.within(conus_bounds)]

cfile = Path(root, "us_coast_states.feather")
if cfile.exists():
    coast_states = gpd.read_feather(cfile)
else:
    coast_states = state[state.intersects(conus_coastline.unary_union)]
    coast_states.to_feather(cfile)

ax = coast_states.to_crs(CRS).plot(**BASE_PLOT)
conus_coastline.to_crs(CRS).plot(ax=ax, edgecolor="b", lw=2, zorder=1)
ax.axis("off")
ax.margins(0)

Tidal and Estuary USGS stations

We need to look at the Water Services API.

import pandas as pd

from pygeohydro import NWIS, ZeroMatchedError

cfile = Path(root, "coast_stations.feather")
if cfile.exists():
    coast_stations = gpd.read_feather(cfile)
else:
    queries = [
        {
            "stateCd": s.lower(),
            "siteType": "ST-TS,ES",
            "hasDataTypeCd": "dv",
            "outputDataTypeCd": "dv",
        }
        for s in coast_states.STUSPS
    ]
    nwis = NWIS()

    def get_info(q):
        try:
            return nwis.get_info(q, True)
        except ZeroMatchedError:
            return None

    sites = pd.concat(get_info(q) for q in queries if q is not None)

    coast_stations = gpd.GeoDataFrame(
        sites,
        geometry=gpd.points_from_xy(sites.dec_long_va, sites.dec_lat_va),
        crs="epsg:4269",
    )
    coast_stations.to_feather(cfile)

st = coast_stations[["site_no", "site_tp_cd", "geometry"]].to_crs(CRS)
ts = st[st.site_tp_cd == "ST-TS"].drop_duplicates()
es = st[st.site_tp_cd == "ES"].drop_duplicates()

ax = conus_states.to_crs(CRS).plot(**BASE_PLOT)
coastline[coastline.within(conus_bounds)].to_crs(CRS).plot(ax=ax, edgecolor="g", lw=2, zorder=1)
ts.plot(ax=ax, lw=3, c="r")
es.plot(ax=ax, lw=3, c="b")
ax.legend(["Coastline", f"ST-TS ({ts.shape[0]})", f"ES ({es.shape[0]})"], loc="best")
ax.axis("off")
ax.margins(0)
ax.figure.savefig(Path("_static", "us_coasts.png"), dpi=300, bbox_inches="tight", facecolor="w")

Mean daily discharge for all stations

import numpy as np
import pandas as pd

cfile = Path(root, "discharge.parquet")
dates = ("2000-01-01", "2015-12-31")

if cfile.exists():
    discharge = pd.read_parquet(cfile)
else:
    nwis = NWIS()
    discharge = nwis.get_streamflow(
        coast_stations.site_no,
        dates,
    )
    discharge[discharge < 0] = np.nan
    discharge.to_parquet(cfile)

ax = discharge.plot(legend=False, lw=0.8, figsize=(7, 3.5))
ax.set_ylabel("Q (cms)")
ax.set_xlabel("")
ax.margins(x=0)

Let’s find the station that has the largest peak value and see it on a map.

station_id = discharge.max().idxmax().split("-")[1]

coast_stations[coast_stations.site_no == station_id].iloc[0]["station_nm"]

'COLUMBIA RIVER AT PORT WESTWARD, NEAR QUINCY, OR'

import pygeohydro as gh

lat, lon = coast_stations[coast_stations.site_no == station_id].iloc[0][
    ["dec_lat_va", "dec_long_va"]
]
bbox = (lon - 0.2, lat - 0.2, lon + 0.2, lat + 0.2)
nwis_kwds = {"hasDataTypeCd": "dv", "outputDataTypeCd": "dv", "parameterCd": "00060"}

station_map = gh.interactive_map(bbox, nwis_kwds=nwis_kwds)

station_map

Make this Notebook Trusted to load map: File -> Trust Notebook

River network data

Basin

import pynhd as nhd

nldi = nhd.NLDI()
cfile = Path(root, "basin.feather")
if cfile.exists():
    basin = gpd.read_feather(cfile)
else:
    basin = nldi.get_basins(station_id)
    basin.to_feather(cfile)

ax = basin.to_crs(CRS).plot(**BASE_PLOT)
ax.axis("off")
ax.margins(0)

import folium

folium.GeoJson(basin.geometry).add_to(station_map)
station_map

Make this Notebook Trusted to load map: File -> Trust Notebook

Main river network

cfile = Path(root, "flowline_main.feather")
if cfile.exists():
    flw_main = gpd.read_feather(cfile)
else:
    flw_main = nldi.navigate_byid(
        fsource="nwissite",
        fid=f"USGS-{station_id}",
        navigation="upstreamMain",
        source="flowlines",
        distance=4000,
    )
    flw_main.to_feather(cfile)

Tributaries

cfile = Path(root, "flowline_trib.feather")
if cfile.exists():
    flw_trib = gpd.read_feather(cfile)
else:
    flw_trib = nldi.navigate_byid(
        fsource="nwissite",
        fid=f"USGS-{station_id}",
        navigation="upstreamTributaries",
        source="flowlines",
        distance=4000,
    )
    flw_trib.to_feather(cfile)
flw_trib["nhdplus_comid"] = flw_trib["nhdplus_comid"].astype("float").astype("Int64")

BASE_PLOT["figsize"] = (8, 5)
ax = basin.plot(**BASE_PLOT)
flw_trib.plot(ax=ax)
flw_main.plot(ax=ax, lw=3, color="r")
ax.legend(["Tributaries", "Main"])
ax.axis("off")
ax.margins(0)

NHDPlus Value Added Attributes (VAA)

cfile = Path(root, "nhdplus_vaa.parquet")
nhdplus_vaa = nhd.nhdplus_vaa(cfile)

flw_trib = flw_trib.merge(nhdplus_vaa, left_on="nhdplus_comid", right_on="comid", how="left")
flw_trib.loc[flw_trib.slope < 0, "slope"] = np.nan

ax = basin.plot(**BASE_PLOT)
flw_trib.plot(
    ax=ax,
    column="slope",
    scheme="natural_breaks",
    k=6,
    cmap="plasma",
    legend=True,
    legend_kwds={"title": "Slope [-]"},
)
ax.axis("off")
ax.margins(0)

This page was generated from water_quality.ipynb. Interactive online version:

Water Quality Portal

import contextily as ctx
import geopandas as gpd
import matplotlib.pyplot as plt
from shapely.geometry import Point

from pygeohydro import WaterQuality

The WaterQuality has a number of convenience methods to retrieve data from the web service. Since there are many parameter combinations that can be used to retrieve data, a general method is provided to retrieve data from any of the valid endpoints. You can use get_json to retrieve stations info as a geopandas.GeoDataFrame or get_csv to retrieve stations data as a pandas.DataFrame. You can construct a dictionary of the parameters and pass it to one of these functions. For more information on the parameters, please consult the Water Quality Data documentation. For example, let’s find all the stations within a bounding box that have Caffeine data:

wq = WaterQuality()
stations = wq.station_bybbox((-92.8, 44.2, -88.9, 46.0), {"characteristicName": "Caffeine"})

Or the same criterion but within a 30 mile radius of a point:

stations = wq.station_bydistance(-92.8, 44.2, 30, {"characteristicName": "Caffeine"})

point = gpd.GeoSeries([Point(-92.8, 44.2)], crs="epsg:4326")
point = point.to_crs(point.estimate_utm_crs())
buff = point.buffer(30 * 1609.344)

fig, ax = plt.subplots(figsize=(6, 6))
buff.plot(ax=ax, facecolor="b", edgecolor="black", alpha=0.1)
point.plot(ax=ax, color="g", marker="x")
stations.to_crs(buff.crs).plot(ax=ax, color="r", marker="o")
ctx.add_basemap(ax, crs=buff.crs)
ax.set_axis_off()
ax.margins(0)
fig.savefig("_static/water_quality.png")

Then we can get for al these stations the data like this:

sids = stations.MonitoringLocationIdentifier.tolist()
caff = wq.data_bystation(sids, {"characteristicName": "Caffeine"})
caff.head()

	OrganizationIdentifier	OrganizationFormalName	ActivityIdentifier	ActivityTypeCode	ActivityMediaName	ActivityMediaSubdivisionName	ActivityStartDate	ActivityStartTime/Time	ActivityStartTime/TimeZoneCode	ActivityEndDate	...	ResultAnalyticalMethod/MethodName	MethodDescriptionText	LaboratoryName	AnalysisStartDate	ResultLaboratoryCommentText	DetectionQuantitationLimitTypeName	DetectionQuantitationLimitMeasure/MeasureValue	DetectionQuantitationLimitMeasure/MeasureUnitCode	PreparationStartDate	ProviderName
0	USGS-MN	USGS Minnesota Water Science Center	nwismn.01.00400395	Sample-Routine	Water	Surface Water	2004-09-03	11:30:00	CDT	NaN	...	NWQL Schedule 1433	USGS WRI 01-4186	NaN	2004-11-13	NaN	Estimated Detection Level	0.50	ug/l	2004-09-16	NWIS
1	USGS-IA	USGS Iowa Water Science Center	nwisia.01.00700543	Sample-Routine	Water	Groundwater	2007-08-27	15:00:00	CDT	NaN	...	Pest, polar, wf, spe, HPLC-MS	USGS WRI 01-4134	USGS-National Water Quality Lab, Denver, CO	2007-09-12	NaN	Laboratory Reporting Level	0.04	ug/l	2007-09-04	NWIS
2	USGS-MN	USGS Minnesota Water Science Center	nwismn.01.00800189	Sample-Routine	Water	Groundwater	2008-04-23	12:30:00	CDT	NaN	...	NWQL Schedule 4433	USGS TMR 5-B4	USGS-National Water Quality Lab, Denver, CO	2008-05-23	NaN	Estimated Detection Level	0.20	ug/l	2008-05-08	NWIS
3	USGS-MN	USGS Minnesota Water Science Center	nwismn.01.00800166	Sample-Routine	Water	Groundwater	2008-04-23	13:53:00	CDT	NaN	...	NWQL Schedule 4433	USGS TMR 5-B4	USGS-National Water Quality Lab, Denver, CO	2008-05-26	NaN	Estimated Detection Level	0.20	ug/l	2008-05-08	NWIS
4	USGS-MN	USGS Minnesota Water Science Center	nwismn.01.00800206	Sample-Routine	Water	Groundwater	2008-04-23	13:30:00	CDT	NaN	...	NWQL Schedule 4433	USGS TMR 5-B4	USGS-National Water Quality Lab, Denver, CO	2008-05-23	NaN	Estimated Detection Level	0.20	ug/l	2008-05-08	NWIS

5 rows × 63 columns

This page was generated from webservices.ipynb. Interactive online version:

Web Services Queries

import pygeoutils as geoutils
from pygeoogc import WFS, WMS, ArcGISRESTful, ServiceURL
from pynhd import NLDI

import warnings

warnings.filterwarnings("ignore", message=".*Content metadata for layer.*")

PyGeoOGC and PyGeoUtils can be used to access any web services that are based on ArcGIS RESTful, WMS, or WFS. It is noted that although, all these web service have limits on the number of objects (e.g., 1000 objectIDs for RESTful) or pixels (e.g., 8 million pixels) per requests, PyGeoOGC takes care of dividing the requests into smaller chunks under-the-hood and then merges them.

Let’s get started by retrieving a watershed geometry using NLDI and use it for subsetting the data.

basin = NLDI().get_basins("11092450")
basin_geom = basin.iloc[0].geometry

PyGeoOGC has a NamedTuple called ServiceURL that contains URLs of the some of the popular web services. Let’s use it to access NHDPlus HR Dataset RESTful service and get all the catchments that our basin contain and use pygeoutils.json2geodf to convert it into a GeoDataFrame.

hr = ArcGISRESTful(ServiceURL().restful.nhdplushr, 10, outformat="json")
oids = hr.oids_bygeom(basin_geom, "epsg:4326", spatial_relation="esriSpatialRelContains")
resp = hr.get_features(oids)
catchments = geoutils.json2geodf(resp)

catchments.plot(figsize=(8, 8))

Note oids_bygeom has an additional argument for passing any valid SQL WHERE clause to further filter the data on the server side. For example, let’s only keep the ones with an area of larger than 0.5 sqkm.

oids = hr.oids_bygeom(basin_geom, geo_crs=4326, sql_clause="AREASQKM > 0.5")
catchments = geoutils.json2geodf(hr.get_features(oids))

ax = catchments.plot(figsize=(8, 8))
ax.figure.savefig("_static/sql_clause.png", bbox_inches="tight", facecolor="w")

We can also submit a query based on IDs of any valid field in the database. If the measure property is desired you can pass return_m as True to the get_features class method:

oids = hr.oids_byfield("NHDPlusID", [5000500013223, 5000400039708, 5000500004825])
resp = hr.get_features(oids, return_m=True)
catchments = geoutils.json2geodf(resp)

catchments.plot(figsize=(8, 8))

Next, let’s get DEM using the 3D Elevation Program WMS service. First we need to connect to the service using WMS class.

wms = WMS(
    ServiceURL().wms.nm_3dep,
    layers="3DEPElevation:None",
    outformat="image/tiff",
    crs=5070,
)

Then we can get the data using the getmap_bybox function. Note that this function only accepts a bounding box, so we need to pass a bounding box and mask the returned data later on using pygeoutils.gtiff2xarray function.

bbox = basin_geom.bounds
r_dict = wms.getmap_bybox(
    bbox,
    100,
    box_crs=4326,
)
dem = geoutils.gtiff2xarray(r_dict, bbox, 4326)

dem.plot()

Next, let’s use WaterData service to get HUC8 using WFS and CQL filter.

layer = "wmadata:huc08"
wfs = WFS(
    ServiceURL().wfs.waterdata,
    layer=layer,
    outformat="application/json",
    version="2.0.0",
    crs=4269,
)
resp = wfs.getfeature_byfilter("huc8 LIKE '13030%'")
huc8 = geoutils.json2geodf(resp, 4269, 4326)

huc8.plot(figsize=(8, 8))

API References

AsyncRetriever

v0.16.0

PyGeoOGC

v0.16.1

PyGeoUtils

v0.16.1

PyNHD

v0.16.2

Py3DEP

v0.16.2

PyGeoHydro

v0.16.0

PyDaymet

v0.16.1

PyGridMET

v0.16.0

PyNLDAS2

v0.16.0

HydroSignatures

v0.16.0

pynhd

Top-level package for PyNHD.

Submodules

pynhd.core

Base classes for PyNHD functions.

Module Contents

class pynhd.core.AGRBase(base_url, layer=None, outfields='*', crs=4326, outformat='json')

Base class for getting geospatial data from a ArcGISRESTful service.

Parameters:

• base_url (str, optional) – The ArcGIS RESTful service url. The URL must either include a layer number after the last / in the url or the target layer must be passed as an argument.

• layer (str, optional) – A valid service layer. To see a list of available layers instantiate the class without passing any argument.

• outfields (str or list, optional) – Target field name(s), default to “*” i.e., all the fields.

• crs (str, int, or pyproj.CRS, optional) – Target spatial reference, default to EPSG:4326

• outformat (str, optional) – One of the output formats offered by the selected layer. If not correct a list of available formats is shown, defaults to json.

property service_info: ServiceInfo

Get the service information.

bygeom(geom, geo_crs=4326, sql_clause='', distance=None, return_m=False, return_geom=True)

Get feature within a geometry that can be combined with a SQL where clause.

Parameters:

• geom (Polygon or tuple) – A geometry (Polygon) or bounding box (tuple of length 4).

• geo_crs (str) – The spatial reference of the input geometry.

• sql_clause (str, optional) – A valid SQL 92 WHERE clause, defaults to an empty string.

• distance (int, optional) – The buffer distance for the input geometries in meters, default to None.

• return_m (bool, optional) – Whether to activate the Return M (measure) in the request, defaults to False.

• return_geom (bool, optional) – Whether to return the geometry of the feature, defaults to True.

Returns:

geopandas.GeoDataFrame – The requested features as a GeoDataFrame.

Return type:

geopandas.GeoDataFrame

byids(field, fids, return_m=False, return_geom=True)

Get features based on a list of field IDs.

Parameters:

• field (str) – Name of the target field that IDs belong to.

• fids (str or list) – A list of target field ID(s).

• return_m (bool) – Whether to activate the Return M (measure) in the request, defaults to False.

• return_geom (bool, optional) – Whether to return the geometry of the feature, defaults to True.

Returns:

geopandas.GeoDataFrame – The requested features as a GeoDataFrame.

Return type:

geopandas.GeoDataFrame

bysql(sql_clause, return_m=False, return_geom=True)

Get feature IDs using a valid SQL 92 WHERE clause.

Notes

Not all web services support this type of query. For more details look here

Parameters:

• sql_clause (str) – A valid SQL 92 WHERE clause.

• return_m (bool) – Whether to activate the measure in the request, defaults to False.

• return_geom (bool, optional) – Whether to return the geometry of the feature, defaults to True.

Returns:

geopandas.GeoDataFrame – The requested features as a GeoDataFrame.

Return type:

geopandas.GeoDataFrame

static get_validlayers(url)

Get a list of valid layers.

Parameters:

url (str) – The URL of the ArcGIS REST service.

Returns:

dict – A dictionary of valid layers.

Return type:

dict[str, int]

class pynhd.core.GeoConnex(item=None, dev=False, max_nfeatures=10000)

Access to the GeoConnex API.

Notes

The geometry field of the query can be a Polygon, MultiPolygon, or tuple/list of length 4 (bbox) in EPSG:4326 CRS. They should be within the extent of the GeoConnex endpoint.

Parameters:

• The item (service endpoint) to query. Valid endpoints are –

  • hu02 for Two-digit Hydrologic Regions

  • hu04 for Four-digit Hydrologic Subregion

  • hu06 for Six-digit Hydrologic Basins

  • hu08 for Eight-digit Hydrologic Subbasins

  • hu10 for Ten-digit Watersheds

  • nat_aq for National Aquifers of the United States from

    USGS National Water Information System National Aquifer code list.

  • principal_aq for Principal Aquifers of the United States from

    2003 USGS data release

  • sec_hydrg_reg for Secondary Hydrogeologic Regions of the

    Conterminous United States from 2018 USGS data release

  • gages for US Reference Stream Gauge Monitoring Locations

  • mainstems for US Reference Mainstem Rivers

  • states for U.S. States

  • counties for U.S. Counties

  • aiannh for Native American Lands

  • cbsa for U.S. Metropolitan and Micropolitan Statistical Areas

  • ua10 for Urbanized Areas and Urban Clusters (2010 Census)

  • places for U.S. legally incororated and Census designated places

  • pws for U.S. Public Water Systems

  • dams for US Reference Dams

• dev (bool, optional) – Whether to use the development endpoint, defaults to False.

• max_nfeatures (int, optional) – The maximum number of features to request from the service, defaults to 10000.

property dev: bool

Return the name of the endpoint.

property item: str | None

Return the name of the endpoint.

bycql(cql_dict: dict[str, Any], skip_geometry: Literal[False] = False) → geopandas.GeoDataFrame

bycql(cql_dict: dict[str, Any], skip_geometry: Literal[True]) → pandas.DataFrame

Query the GeoConnex endpoint.

Notes

GeoConnex only supports simple CQL queries. For more information and examples visit https://portal.ogc.org/files/96288#simple-cql-JSON. Use this for non-spatial queries, since there’s a dedicated method for spatial queries, bygeometry().

Parameters:

• cql_dict (dict) – A valid CQL dictionary (non-spatial queries).

• skip_geometry (bool, optional) – If True, no geometry will not be returned, by default False.

Returns:

geopandas.GeoDataFrame – The query result as a geopandas.GeoDataFrame.

bygeometry(geometry1: GTYPE, geometry2: GTYPE | None = ..., predicate: str = ..., crs: CRSTYPE = ..., skip_geometry: Literal[False] = False) → geopandas.GeoDataFrame

bygeometry(geometry1: GTYPE, geometry2: GTYPE | None = ..., predicate: str = ..., crs: CRSTYPE = ..., skip_geometry: Literal[True] = True) → pandas.DataFrame

Query the GeoConnex endpoint by geometry.

Parameters:

• geometry1 (Polygon or tuple of float) – The first geometry or bounding boxes to query. A bounding box is a tuple of length 4 in the form of (xmin, ymin, xmax, ymax). For example, an spatial query for a single geometry would be INTERSECTS(geom, geometry1).

• geometry2 (Polygon or tuple of float, optional) – The second geometry or bounding boxes to query. A bounding box is a tuple of length 4 in the form of (xmin, ymin, xmax, ymax). Default is None. For example, an spatial query for a two geometries would be CROSSES(geometry1, geometry2).

• predicate (str, optional) – The predicate to use, by default intersects. Supported predicates are intersects, within, contains, overlaps, crosses, disjoint, touches, and equals.

• crs (int or str or pyproj.CRS, optional) – The CRS of the polygon, by default EPSG:4326. If the input is a geopandas.GeoDataFrame or geopandas.GeoSeries, this argument will be ignored.

• skip_geometry (bool, optional) – If True, no geometry will not be returned.

Returns:

geopandas.GeoDataFrame – The query result as a geopandas.GeoDataFrame.

byid(feature_name: str, feature_ids: list[str] | str, skip_geometry: Literal[False] = False) → geopandas.GeoDataFrame

byid(feature_name: str, feature_ids: list[str] | str, skip_geometry: Literal[True]) → pandas.DataFrame

Query the GeoConnex endpoint.

class pynhd.core.ScienceBase

Access and explore items on USGS’s ScienceBase.

static get_children(item)

Get children items of an item.

static get_file_urls(item)

Get download and meta URLs of all the available files for an item.

pynhd.network_tools

Access NLDI and WaterData databases.

Module Contents

class pynhd.network_tools.NHDTools(flowlines)

Prepare NHDPlus data for downstream analysis.

Notes

Some of these tools are ported from nhdplusTools.

Parameters:

flowlines (geopandas.GeoDataFrame) – NHDPlus flowlines with at least the following columns: comid, lengthkm, ftype, terminalfl, fromnode, tonode, totdasqkm, startflag, streamorde, streamcalc, terminalpa, pathlength, divergence, hydroseq, and levelpathi.

add_tocomid()

Find the downstream comid(s) of each comid in NHDPlus flowline database.

Notes

This functions requires the following columns:

comid, terminalpa, fromnode, tonode

static check_requirements(reqs, cols)

Check for all the required data.

Parameters:

• reqs (iterable) – A list of required data names (str)

• cols (list) – A list of variable names (str)

clean_flowlines(use_enhd_attrs, terminal2nan)

Clean up flowlines.

Parameters:

• use_enhd_attrs (bool) – Use attributes from the ENHD database.

• terminal2nan (bool) – Convert terminal flowlines to NaN.

remove_isolated()

Remove isolated flowlines.

remove_tinynetworks(min_path_size, min_path_length, min_network_size)

Remove small paths in NHDPlus flowline database.

Notes

This functions requires the following columns: levelpathi, hydroseq, totdasqkm, terminalfl, startflag, pathlength, and terminalpa.

Parameters:

• min_network_size (float) – Minimum size of drainage network in sqkm.

• min_path_length (float) – Minimum length of terminal level path of a network in km.

• min_path_size (float) – Minimum size of outlet level path of a drainage basin in km. Drainage basins with an outlet drainage area smaller than this value will be removed.

to_linestring()

Convert flowlines to shapely LineString objects.

pynhd.network_tools.enhd_flowlines_nx()

Get a networkx.DiGraph of the entire NHD flowlines.

Changed in version 0.16.2: The function now replaces all 0 values in the tocomid column of ENHD with the negative of their corresponding comid values. This ensures all sinks are unique and treated accordingly for topological sorting and other network analysis. The difference are in the returned label2comid dictionary and onnetwork_sorted which will contain the negative values for the sinks.

Notes

The graph is directed and has the all the attributes of the flowlines in ENHD. Note that COMIDs are based on the 2020 snapshot of the NHDPlusV2.1.

Returns:

• graph (networkx.DiGraph) – The generated directed graph

• label2comid (dict) – A mapping of COMIDs to the node IDs in the graph

• onnetwork_sorted (list) – A topologically sorted list of the COMIDs.

Return type:

tuple[networkx.DiGraph, dict[int, int], list[int]]

pynhd.network_tools.flowline_resample(flw, spacing, id_col='comid', smoothing=None)

Resample a flowline based on a given spacing.

Parameters:

• flw (geopandas.GeoDataFrame) – A dataframe with geometry and id_col columns and CRS attribute. The flowlines should be able to merged to a single LineString. Otherwise, you should use the network_resample() function.

• spacing (float) – Spacing between the sample points in meters.

• id_col (str, optional) – Name of the flowlines column containing IDs, defaults to comid.

• smoothing (float or None, optional) – Smoothing factor is used for determining the number of knots. This arg controls the tradeoff between closeness and smoothness of fit. Larger smoothing means more smoothing while smaller values of smoothing indicates less smoothing. If None (default), smoothing is done with all points.

Returns:

geopandas.GeoDataFrame – Resampled flowline.

Return type:

geopandas.GeoDataFrame

pynhd.network_tools.flowline_xsection(flw, distance, width, id_col='comid', smoothing=None)

Get cross-section of a river network at a given spacing.

Parameters:

• flw (geopandas.GeoDataFrame) – A dataframe with geometry and, id_col, and levelpathi columns and a projected CRS attribute.

• distance (float) – The distance between two consecutive cross-sections.

• width (float) – The width of the cross-section.

• id_col (str, optional) – Name of the flowlines column containing IDs, defaults to comid.

• smoothing (float or None, optional) – Smoothing factor is used for determining the number of knots. This arg controls the tradeoff between closeness and smoothness of fit. Larger smoothing means more smoothing while smaller values of smoothing indicates less smoothing. If None (default), smoothing is done with all points.

Returns:

geopandas.GeoDataFrame – A dataframe with two columns: geometry and comid. The geometry column contains the cross-section of the river network and the comid column contains the corresponding comid from the input dataframe. Note that each comid can have multiple cross-sections depending on the given spacing distance.

Return type:

geopandas.GeoDataFrame

pynhd.network_tools.mainstem_huc12_nx()

Get a networkx.DiGraph of the entire mainstem HUC12s.

Notes

The directed graph is generated from the nhdplusv2wbd.csv file with all attributes that can be found in Mainstem. Note that HUC12s are based on the 2020 snapshot of the NHDPlusV2.1.

Returns:

• networkx.DiGraph – The mainstem as a networkx.DiGraph with all the attributes of the mainstems.

• dict – A mapping of the HUC12s to the node IDs in the graph.

• list – A topologically sorted list of the HUC12s which strings of length 12.

Return type:

tuple[networkx.DiGraph, dict[int, str], list[str]]

pynhd.network_tools.network_resample(flw, spacing, id_col='comid', smoothing=None)

Resample a network flowline based on a given spacing.

Parameters:

• flw (geopandas.GeoDataFrame) – A dataframe with geometry and, id_col, and levelpathi columns and a projected CRS attribute.

• spacing (float) – Target spacing between the sample points in the length unit of the flw’s CRS.

• id_col (str, optional) – Name of the flowlines column containing IDs, defaults to comid.

• smoothing (float or None, optional) – Smoothing factor is used for determining the number of knots. This arg controls the tradeoff between closeness and smoothness of fit. Larger smoothing means more smoothing while smaller values of smoothing indicates less smoothing. If None (default), smoothing is done with all points.

Returns:

geopandas.GeoDataFrame – Resampled flowlines.

Return type:

geopandas.GeoDataFrame

pynhd.network_tools.network_xsection(flw, distance, width, id_col='comid', smoothing=None)

Get cross-section of a river network at a given spacing.

Parameters:

• flw (geopandas.GeoDataFrame) – A dataframe with geometry and, id_col, and levelpathi columns and a projected CRS attribute.

• distance (float) – The distance between two consecutive cross-sections.

• width (float) – The width of the cross-section.

• id_col (str, optional) – Name of the flowlines column containing IDs, defaults to comid.

• smoothing (float or None, optional) – Smoothing factor is used for determining the number of knots. This arg controls the tradeoff between closeness and smoothness of fit. Larger smoothing means more smoothing while smaller values of smoothing indicates less smoothing. If None (default), smoothing is done with all points.

Returns:

geopandas.GeoDataFrame – A dataframe with two columns: geometry and comid. The geometry column contains the cross-section of the river network and the comid column contains the corresponding comid from the input dataframe. Note that each comid can have multiple cross-sections depending on the given spacing distance.

Return type:

geopandas.GeoDataFrame

pynhd.network_tools.nhdflw2nx(flowlines, id_col='comid', toid_col='tocomid', edge_attr=None)

Convert NHDPlus flowline database to networkx graph.

Parameters:

• flowlines (geopandas.GeoDataFrame) – NHDPlus flowlines.

• id_col (str, optional) – Name of the column containing the node ID, defaults to “comid”.

• toid_col (str, optional) – Name of the column containing the downstream node ID, defaults to “tocomid”.

• edge_attr (str, optional) – Name of the column containing the edge attributes, defaults to None. If True, all remaining columns will be used as edge attributes.

Returns:

nx.DiGraph – Networkx directed graph of the NHDPlus flowlines. Note that all elements of the toid_col are replaced with negative values of their corresponding id_cl values if they are NaN or 0. This is to ensure that the generated nodes in the graph are unique.

Return type:

networkx.DiGraph

pynhd.network_tools.nhdplus_l48(layer=None, data_dir='cache', **kwargs)

Get the entire NHDPlus dataset.

Notes

The entire NHDPlus dataset for CONUS (Lower 48) is downloaded from here. This 7.3 GB file will take a while to download, depending on your internet connection. The first time you run this function, the file will be downloaded and stored in the ./cache directory. Subsequent calls will use the cached file. Moreover, there are two additional dependencies required to read the file: pyogrio and py7zr. These dependencies can be installed using pip install pyogrio py7zr or conda install -c conda-forge pyogrio py7zr.

Parameters:

• layer (str, optional) – The layer name to be returned. Either layer should be provided or sql. Defaults to None. The available layers are:

  • Gage

  • BurnAddLine

  • BurnAddWaterbody

  • LandSea

  • Sink

  • Wall

  • Catchment

  • CatchmentSP

  • NHDArea

  • NHDWaterbody

  • HUC12

  • NHDPlusComponentVersions

  • PlusARPointEvent

  • PlusFlowAR

  • NHDFCode

  • DivFracMP

  • BurnLineEvent

  • NHDFlowline_Network

  • NHDFlowline_NonNetwork

  • GeoNetwork_Junctions

  • PlusFlow

  • N_1_Desc

  • N_1_EDesc

  • N_1_EStatus

  • N_1_ETopo

  • N_1_FloDir

  • N_1_JDesc

  • N_1_JStatus

  • N_1_JTopo

  • N_1_JTopo2

  • N_1_Props

• data_dire (str or pathlib.Pathlib.Path) – Directory to store the downloaded file and use in subsequent calls, defaults to ./cache.

• **kwargs – Keyword arguments are passed to pyogrio.read_dataframe. For more information, visit pyogrio.

Returns:

geopandas.GeoDataFrame – A dataframe with all the NHDPlus data.

Return type:

geopandas.GeoDataFrame

pynhd.network_tools.prepare_nhdplus(flowlines, min_network_size, min_path_length, min_path_size=0, purge_non_dendritic=False, remove_isolated=False, use_enhd_attrs=False, terminal2nan=True)

Clean up and fix common issues of NHDPlus MR and HR flowlines.

Ported from nhdplusTools.

Parameters:

• flowlines (geopandas.GeoDataFrame) – NHDPlus flowlines with at least the following columns: comid, lengthkm, ftype, terminalfl, fromnode, tonode, totdasqkm, startflag, streamorde, streamcalc, terminalpa, pathlength, divergence, hydroseq, levelpathi.

• min_network_size (float) – Minimum size of drainage network in sqkm

• min_path_length (float) – Minimum length of terminal level path of a network in km.

• min_path_size (float, optional) – Minimum size of outlet level path of a drainage basin in km. Drainage basins with an outlet drainage area smaller than this value will be removed. Defaults to 0.

• purge_non_dendritic (bool, optional) – Whether to remove non dendritic paths, defaults to False.

• remove_isolated (bool, optional) – Whether to remove isolated flowlines, i.e., keep only the largest connected component of the flowlines. Defaults to False.

• use_enhd_attrs (bool, optional) – Whether to replace the attributes with the ENHD attributes, defaults to False. Note that this only works for NHDPlus mid-resolution (MR) and does not work for NHDPlus high-resolution (HR). For more information, see this.

• terminal2nan (bool, optional) – Whether to replace the COMID of the terminal flowline of the network with NaN, defaults to True. If False, the terminal COMID will be set from the ENHD attributes i.e. use_enhd_attrs will be set to True which is only applicable to NHDPlus mid-resolution (MR).

Returns:

geopandas.GeoDataFrame – Cleaned up flowlines. Note that all column names are converted to lower case.

Return type:

geopandas.GeoDataFrame

pynhd.network_tools.topoogical_sort(flowlines, edge_attr=None, largest_only=False, id_col='ID', toid_col='toID')

Topological sorting of a river network.

Parameters:

• flowlines (pandas.DataFrame) – A dataframe with columns ID and toID

• edge_attr (str or list, optional) – Names of the columns in the dataframe to be used as edge attributes, defaults to None.

• largest_only (bool, optional) – Whether to return only the largest network, defaults to False.

• id_col (str, optional) – Name of the column containing the node ID, defaults to ID.

• toid_col (str, optional) – Name of the column containing the downstream node ID, defaults to toID.

Returns:

(list, dict , networkx.DiGraph) – A list of topologically sorted IDs, a dictionary with keys as IDs and values as a list of its upstream nodes, and the generated networkx.DiGraph object. Note that node IDs are associated with the input flow line IDs, but there might be some negative IDs in the output garph that are not present in the input flow line IDs. These “artificial” nodes are used to represent the graph outlet (the most downstream nodes) in the graph.

Return type:

tuple[list[numpy.int64 | pandas._libs.missing.NAType], dict[int, list[int]], networkx.DiGraph]

pynhd.network_tools.vector_accumulation(flowlines, func, attr_col, arg_cols, id_col='comid', toid_col='tocomid')

Flow accumulation using vector river network data.

Parameters:

• flowlines (pandas.DataFrame) – A dataframe containing comid, tocomid, attr_col and all the columns that ara required for passing to func.

• func (function) – The function that routes the flow in a single river segment. Positions of the arguments in the function should be as follows: func(qin, *arg_cols) qin is computed in this function and the rest are in the order of the arg_cols. For example, if arg_cols = ["slope", "roughness"] then the functions is called this way: func(qin, slope, roughness) where slope and roughness are elemental values read from the flowlines.

• attr_col (str) – The column name of the attribute being accumulated in the network. The column should contain the initial condition for the attribute for each river segment. It can be a scalar or an array (e.g., time series).

• arg_cols (list of strs) – List of the flowlines columns that contain all the required data for a routing a single river segment such as slope, length, lateral flow, etc.

• id_col (str, optional) – Name of the flowlines column containing IDs, defaults to comid

• toid_col (str, optional) – Name of the flowlines column containing toIDs, defaults to tocomid

Returns:

pandas.Series – Accumulated flow for all the nodes. The dataframe is sorted from upstream to downstream (topological sorting). Depending on the given initial condition in the attr_col, the outflow for each river segment can be a scalar or an array.

Return type:

pandas.Series

pynhd.nhdplus_derived

Access NLDI and WaterData databases.

Module Contents

class pynhd.nhdplus_derived.StreamCat

Get StreamCat API’s properties.

base_url

The base URL of the API.

Type:

str

valid_names

The valid names of the metrics.

Type:

list of str

alt_names

The alternative names of some metrics.

Type:

dict of str

valid_regions

The valid hydro regions.

Type:

list of str

valid_states

The valid two letter states’ abbreviations.

Type:

pandas.DataFrame

valid_counties

The valid counties’ FIPS codes.

Type:

pandas.DataFrame

valid_aois

The valid types of areas of interest.

Type:

list of str

metrics_df

The metrics’ metadata such as description and units.

Type:

pandas.DataFrame

valid_years

A dictionary of the valid years for annual metrics.

Type:

dict

pynhd.nhdplus_derived.enhd_attrs(parquet_path=None)

Get updated NHDPlus attributes from ENHD V2.0.

Notes

This function downloads a 160 MB parquet file from here. Although this dataframe does not include geometry, it can be linked to other geospatial NHDPlus dataframes through ComIDs.

Parameters:

parquet_path (str or pathlib.Pathlib.Path, optional) – Path to a file with .parquet extension for storing the file, defaults to ./cache/enhd_attrs.parquet.

Returns:

pandas.DataFrame – A dataframe that includes ComID-level attributes for 2.7 million NHDPlus flowlines.

Return type:

pandas.DataFrame

pynhd.nhdplus_derived.epa_nhd_catchments(comids, feature)

Get NHDPlus catchment-scale data from EPA’s HMS REST API.

Notes

For more information about curve number please refer to the project’s webpage on the EPA’s website.

Parameters:

• comids (int or list of int) – ComID(s) of NHDPlus catchments.

• feature (str) – The feature of interest. Available options are:

  • catchment_metrics: 414 catchment-scale metrics.

  • curve_number: 16-day average Curve Number.

  • comid_info: ComID information.

Returns:

dict of pandas.DataFrame or geopandas.GeoDataFrame – A dict of the requested dataframes. A comid_info dataframe is always returned.

Return type:

dict[str, pandas.DataFrame]

Examples

>>> import pynhd
>>> data = nhd.epa_nhd_catchments(1440291, "catchment_metrics")
>>> data["catchment_metrics"].loc[1440291, "AvgWetIndxCat"]
579.532

pynhd.nhdplus_derived.nhd_fcode()

Get all the NHDPlus FCodes.

pynhd.nhdplus_derived.nhdplus_attrs(attr_name=None)

Stage the NHDPlus Attributes database and save to nhdplus_attrs.parquet.

Notes

More info can be found here.

Parameters:

attr_names (str , *optional*) – Name of NHDPlus attribute to return, defaults to None, i.e., only return a metadata dataframe that includes the attribute names and their description and units.

Returns:

pandas.DataFrame – The staged data as a DataFrame.

Return type:

pandas.DataFrame

pynhd.nhdplus_derived.nhdplus_attrs_s3(attr_names=None, nodata=False)

Access NHDPlus V2.1 derived attributes over CONUS.

Notes

More info can be found here.

Parameters:

• attr_names (str or list of str, optional) – Names of NHDPlus attribute(s) to return, defaults to None, i.e., only return a metadata dataframe that includes the attribute names and their description and units.

• nodata (bool) – Whether to include NODATA percentages, default is False.

Returns:

pandas.DataFrame – A dataframe of requested NHDPlus attributes.

Return type:

pandas.DataFrame

pynhd.nhdplus_derived.nhdplus_h12pp(gpkg_path=None)

Access HUC12 Pour Points for NHDPlus V2.1 L48 (CONUS).

Notes

More info can be found here.

Parameters:

gpkg_path (str or pathlib.Pathlib.Path, optional) – Path to the geopackage file, defaults to None, i.e., download the file to the cache directory as 102020wbd_outlets.gpkg.

Returns:

geopandas.GeoDataFrame – A geodataframe of HUC12 pour points.

Return type:

pandas.DataFrame

pynhd.nhdplus_derived.nhdplus_vaa(parquet_path=None)

Get NHDPlus Value Added Attributes including roughness.

Notes

This function downloads a 245 MB parquet file from here. Although this dataframe does not include geometry, it can be linked to other geospatial NHDPlus dataframes through ComIDs.

Parameters:

parquet_path (str or pathlib.Pathlib.Path, optional) – Path to a file with .parquet extension for storing the file, defaults to ./cache/nldplus_vaa.parquet.

Returns:

pandas.DataFrame – A dataframe that includes ComID-level attributes for 2.7 million NHDPlus flowlines.

Return type:

pandas.DataFrame

pynhd.nhdplus_derived.streamcat(metric_names, metric_areas=None, comids=None, regions=None, states=None, counties=None, conus=False, percent_full=False, area_sqkm=False)

Get various metrics for NHDPlusV2 catchments from EPA’s StreamCat.

Notes

For more information about the service check its webpage at https://www.epa.gov/national-aquatic-resource-surveys/streamcat-dataset.

Parameters:

• metric_names (str or list of str) – Metric name(s) to retrieve. There are 567 metrics available. to get a full list check out StreamCat.valid_names(). To get a description of each metric, check out StreamCat.metrics_df(). Some metrics require year and/or slope to be specified, which have [Year] and/or [Slope] in their name. For convenience all these variables and their years/slopes are converted to a dict that can be accessed via StreamCat.valid_years() and StreamCat.valid_slopes().

• metric_areas (str or list of str, optional) – Areas to return the metrics for, defaults to None, i.e. all areas. Valid options are: catchment, watershed, riparian_catchment, riparian_watershed, other.

• comids (int or list of int, optional) – NHDPlus COMID(s), defaults to None. Either comids, regions, states, counties, or conus must be passed. They are mutually exclusive.

• regions (str or list of str, optional) – Hydro region(s) to retrieve metrics for, defaults to None. For a full list of valid regions check out StreamCat.valid_regions() Either comids, regions, states, counties, or conus must be passed. They are mutually exclusive.

• states (str or list of str, optional) – Two letter state abbreviation(s) to retrieve metrics for, defaults to None. For a full list of valid states check out StreamCat.valid_states() Either comids, regions, states, counties, or conus must be passed. They are mutually exclusive.

• counties (str or list of str, optional) – County FIPS codes(s) to retrieve metrics for, defaults to None. For a full list of valid county codes check out StreamCat.valid_counties() Either comids, regions, states, counties, or conus must be passed. They are mutually exclusive.

• conus (bool, optional) – If True, metric_names of all NHDPlus COMIDs are retrieved, defaults False. Either comids, regions, states, counties, or conus must be passed. They are mutually exclusive.

• percent_full (bool, optional) – If True, return the percent of each area of interest covered by the metric.

• area_sqkm (bool, optional) – If True, return the area in square kilometers.

Returns:

pandas.DataFrame – A dataframe with the requested metrics.

Return type:

pandas.DataFrame

pynhd.pynhd

Access NLDI and WaterData databases.

Module Contents

class pynhd.pynhd.HP3D(layer, outfields='*', crs=4326)

Access USGS 3D Hydrography Program (3DHP) service.

Notes

For more info visit: https://hydro.nationalmap.gov/arcgis/rest/services/3DHP_all/MapServer

Parameters:

• layer (str, optional) – A valid service layer. Layer names with _hr are high resolution and _mr are medium resolution. Also, layer names with _nonconus are for non-conus areas, i.e., Alaska, Hawaii, Puerto Rico, the Virgin Islands , and the Pacific Islands. Valid layers are:

  • hydrolocation

  • flowline

  • waterbody

  • drainage_area

  • catchment

• outfields (str or list, optional) – Target field name(s), default to “*” i.e., all the fields.

• crs (str, int, or pyproj.CRS, optional) – Target spatial reference, default to EPSG:4326.

bygeom(geom, geo_crs=4326, sql_clause='', distance=None, return_m=False, return_geom=True)

Get features within a geometry that can be combined with a SQL where clause.

byids(field, fids, return_m=False, return_geom=True)

Get features by object IDs.

bysql(sql_clause, return_m=False, return_geom=True)

Get features using a valid SQL 92 WHERE clause.

class pynhd.pynhd.NHD(layer, outfields='*', crs=4326)

Access National Hydrography Dataset (NHD), both meduim and high resolution.

Notes

For more info visit: https://hydro.nationalmap.gov/arcgis/rest/services/nhd/MapServer

Parameters:

• layer (str, optional) – A valid service layer. Layer names with _hr are high resolution and _mr are medium resolution. Also, layer names with _nonconus are for non-conus areas, i.e., Alaska, Hawaii, Puerto Rico, the Virgin Islands , and the Pacific Islands. Valid layers are:

  • point

  • point_event

  • line_hr

  • flow_direction

  • flowline_mr

  • flowline_hr_nonconus

  • flowline_hr

  • area_mr

  • area_hr_nonconus

  • area_hr

  • waterbody_mr

  • waterbody_hr_nonconus

  • waterbody_hr

• outfields (str or list, optional) – Target field name(s), default to “*” i.e., all the fields.

• crs (str, int, or pyproj.CRS, optional) – Target spatial reference, default to EPSG:4326.

bygeom(geom, geo_crs=4326, sql_clause='', distance=None, return_m=False, return_geom=True)

Get features within a geometry that can be combined with a SQL where clause.

byids(field, fids, return_m=False, return_geom=True)

Get features by object IDs.

bysql(sql_clause, return_m=False, return_geom=True)

Get features using a valid SQL 92 WHERE clause.

class pynhd.pynhd.NHDPlusHR(layer, outfields='*', crs=4326)

Access National Hydrography Dataset (NHD) Plus high resolution.

Notes

For more info visit: https://hydro.nationalmap.gov/arcgis/rest/services/NHDPlus_HR/MapServer

Parameters:

• layer (str, optional) – A valid service layer. Valid layers are:

  • gage for NHDPlusGage layer

  • sink for NHDPlusSink layer

  • point for NHDPoint layer

  • flowline for NetworkNHDFlowline layer

  • non_network_flowline for NonNetworkNHDFlowline layer

  • flow_direction for FlowDirection layer

  • wall for NHDPlusWall layer

  • line for NHDLine layer

  • area for NHDArea layer

  • waterbody for NHDWaterbody layer

  • catchment for NHDPlusCatchment layer

  • boundary_unit for NHDPlusBoundaryUnit layer

  • huc12 for WBDHU12 layer

• outfields (str or list, optional) – Target field name(s), default to “*” i.e., all the fields.

• crs (str, int, or pyproj.CRS, optional) – Target spatial reference, default to EPSG:4326.

bygeom(geom, geo_crs=4326, sql_clause='', distance=None, return_m=False, return_geom=True)

Get features within a geometry that can be combined with a SQL where clause.

byids(field, fids, return_m=False, return_geom=True)

Get features by object IDs.

bysql(sql_clause, return_m=False, return_geom=True)

Get features using a valid SQL 92 WHERE clause.

class pynhd.pynhd.NLDI

Access the Hydro Network-Linked Data Index (NLDI) service.

comid_byloc(coords, loc_crs=4326)

Get the closest ComID based on coordinates using hydrolocation endpoint.

Notes

This function tries to find the closest ComID based on flowline grid cells. If such a cell is not found, it will return the closest ComID using the flowtrace endpoint of the PyGeoAPI service to find the closest downstream ComID. The returned dataframe has a measure column that indicates the location of the input coordinate on the flowline as a percentage of the total flowline length.

Parameters:

• coords (tuple or list of tuples) – A tuple of length two (x, y) or a list of them.

• loc_crs (str, int, or pyproj.CRS, optional) – The spatial reference of the input coordinate, defaults to EPSG:4326.

Returns:

geopandas.GeoDataFrame or (geopandas.GeoDataFrame, list) – NLDI indexed ComID(s) and points in EPSG:4326. If some coords don’t return any ComID a list of missing coords are returned as well.

Return type:

geopandas.GeoDataFrame

feature_byloc(coords, loc_crs=4326)

Get the closest feature ID(s) based on coordinates using position endpoint.

Parameters:

• coords (tuple or list) – A tuple of length two (x, y) or a list of them.

• loc_crs (str, int, or pyproj.CRS, optional) – The spatial reference of the input coordinate, defaults to EPSG:4326.

Returns:

geopandas.GeoDataFrame or (geopandas.GeoDataFrame, list) – NLDI indexed feature ID(s) and flowlines in EPSG:4326. If some coords don’t return any IDs a list of missing coords are returned as well.

Return type:

geopandas.GeoDataFrame

get_basins(feature_ids, fsource='nwissite', split_catchment=False, simplified=True)

Get basins for a list of station IDs.

Parameters:

• feature_ids (str or list) – Target feature ID(s).

• fsource (str) – The name of feature(s) source, defaults to nwissite. The valid sources are:

  • ‘comid’ for NHDPlus comid.

  • ‘ca_gages’ for Streamgage catalog for CA SB19

  • ‘gfv11_pois’ for USGS Geospatial Fabric V1.1 Points of Interest

  • ‘huc12pp’ for HUC12 Pour Points

  • ‘nmwdi-st’ for New Mexico Water Data Initiative Sites

  • ‘nwisgw’ for NWIS Groundwater Sites

  • ‘nwissite’ for NWIS Surface Water Sites

  • ‘ref_gage’ for geoconnex.us reference gages

  • ‘vigil’ for Vigil Network Data

  • ‘wade’ for Water Data Exchange 2.0 Sites

  • ‘WQP’ for Water Quality Portal

• split_catchment (bool, optional) – If True, split basins at their outlet locations. Default to False.

• simplified (bool, optional) – If True, return a simplified version of basin geometries. Default to True.

Returns:

geopandas.GeoDataFrame or (geopandas.GeoDataFrame, list) – NLDI indexed basins in EPSG:4326. If some IDs don’t return any features a list of missing ID(s) are returned as well.

Return type:

geopandas.GeoDataFrame

getcharacteristic_byid(feature_ids: str | int | Sequence[str | int], char_type: str, fsource: str = ..., char_ids: str | list[str] = ..., values_only: Literal[True] = ...) → pandas.DataFrame

getcharacteristic_byid(feature_ids: str | int | Sequence[str | int], char_type: str, fsource: str = ..., char_ids: str | list[str] = ..., values_only: Literal[False] = ...) → tuple[pandas.DataFrame, pandas.DataFrame]

Get characteristics using a list ComIDs.

Parameters:

• feature_ids (str or list) – Target feature ID(s).

• char_type (str) – Type of the characteristic. Valid values are local for individual reach catchments, tot for network-accumulated values using total cumulative drainage area and div for network-accumulated values using divergence-routed.

• fsource (str, optional) – The name of feature(s) source, defaults to comid. The valid sources are:

  • ‘comid’ for NHDPlus comid.

  • ‘ca_gages’ for Streamgage catalog for CA SB19

  • ‘gfv11_pois’ for USGS Geospatial Fabric V1.1 Points of Interest

  • ‘huc12pp’ for HUC12 Pour Points

  • ‘nmwdi-st’ for New Mexico Water Data Initiative Sites

  • ‘nwisgw’ for NWIS Groundwater Sites

  • ‘nwissite’ for NWIS Surface Water Sites

  • ‘ref_gage’ for geoconnex.us reference gages

  • ‘vigil’ for Vigil Network Data

  • ‘wade’ for Water Data Exchange 2.0 Sites

  • ‘WQP’ for Water Quality Portal

• char_ids (str or list, optional) – Name(s) of the target characteristics, default to all.

• values_only (bool, optional) – Whether to return only characteristic_value as a series, default to True. If is set to False, percent_nodata is returned as well.

Returns:

pandas.DataFrame or tuple of pandas.DataFrame – Either only characteristic_value as a dataframe or or if values_only is Fale return percent_nodata as well.

getfeature_byid(fsource, fids)

Get feature(s) based ID(s).

Parameters:

• fsource (str) – The name of feature(s) source. The valid sources are:

  • ‘comid’ for NHDPlus comid.

  • ‘ca_gages’ for Streamgage catalog for CA SB19

  • ‘gfv11_pois’ for USGS Geospatial Fabric V1.1 Points of Interest

  • ‘huc12pp’ for HUC12 Pour Points

  • ‘nmwdi-st’ for New Mexico Water Data Initiative Sites

  • ‘nwisgw’ for NWIS Groundwater Sites

  • ‘nwissite’ for NWIS Surface Water Sites

  • ‘ref_gage’ for geoconnex.us reference gages

  • ‘vigil’ for Vigil Network Data

  • ‘wade’ for Water Data Exchange 2.0 Sites

  • ‘WQP’ for Water Quality Portal

• fid (str or list of str) – Feature ID(s).

Returns:

geopandas.GeoDataFrame or (geopandas.GeoDataFrame, list) – NLDI indexed features in EPSG:4326. If some IDs don’t return any features a list of missing ID(s) are returned as well.

Return type:

geopandas.GeoDataFrame

navigate_byid(fsource, fid, navigation, source, distance=500, trim_start=False, stop_comid=None)

Navigate the NHDPlus database from a single feature id up to a distance.

Parameters:

• fsource (str) – The name of feature(s) source. The valid sources are:

  • ‘comid’ for NHDPlus comid.

  • ‘ca_gages’ for Streamgage catalog for CA SB19

  • ‘gfv11_pois’ for USGS Geospatial Fabric V1.1 Points of Interest

  • ‘huc12pp’ for HUC12 Pour Points

  • ‘nmwdi-st’ for New Mexico Water Data Initiative Sites

  • ‘nwisgw’ for NWIS Groundwater Sites

  • ‘nwissite’ for NWIS Surface Water Sites

  • ‘ref_gage’ for geoconnex.us reference gages

  • ‘vigil’ for Vigil Network Data

  • ‘wade’ for Water Data Exchange 2.0 Sites

  • ‘WQP’ for Water Quality Portal

• fid (str or int) – The ID of the feature.

• navigation (str) – The navigation method.

• source (str) – Return the data from another source after navigating features from fsource.

• distance (int, optional) – Limit the search for navigation up to a distance in km, defaults is 500 km. Note that this is an expensive request so you have be mindful of the value that you provide. The value must be between 1 to 9999 km.

• trim_start (bool, optional) – If True, trim the starting flowline at the source feature, defaults to False.

• stop_comid (str or int, optional) – The ComID to stop the navigationation, defaults to None.

Returns:

geopandas.GeoDataFrame – NLDI indexed features in EPSG:4326.

Return type:

geopandas.GeoDataFrame

navigate_byloc(coords, navigation=None, source=None, loc_crs=4326, distance=500, trim_start=False, stop_comid=None)

Navigate the NHDPlus database from a coordinate.

Notes

This function first calls the feature_byloc function to get the comid of the nearest flowline and then calls the navigate_byid function to get the features from the obtained comid.

Parameters:

• coords (tuple) – A tuple of length two (x, y).

• navigation (str, optional) – The navigation method, defaults to None which throws an exception if comid_only is False.

• source (str, optional) – Return the data from another source after navigating the features based on comid, defaults to None which throws an exception if comid_only is False.

• loc_crs (str, int, or pyproj.CRS, optional) – The spatial reference of the input coordinate, defaults to EPSG:4326.

• distance (int, optional) – Limit the search for navigation up to a distance in km, defaults to 500 km. Note that this is an expensive request so you have be mindful of the value that you provide.

• trim_start (bool, optional) – If True, trim the starting flowline at the source feature, defaults to False.

• stop_comid (str or int, optional) – The ComID to stop the navigationation, defaults to None.

Returns:

geopandas.GeoDataFrame – NLDI indexed features in EPSG:4326.

Return type:

geopandas.GeoDataFrame

class pynhd.pynhd.PyGeoAPI

Access PyGeoAPI service.

cross_section(coord, width, numpts, crs=4326)

Return a GeoDataFrame from the xsatpoint service.

Parameters:

• coord (tuple) – The coordinate of the point to extract the cross-section as a tuple,e.g., (lon, lat).

• width (float) – The width of the cross-section in meters.

• numpts (int) – The number of points to extract the cross-section from the DEM.

• crs (str, int, or pyproj.CRS, optional) – The coordinate reference system of the coordinates, defaults to EPSG:4326.

Returns:

geopandas.GeoDataFrame – A GeoDataFrame containing the cross-section at the requested point.

Return type:

geopandas.GeoDataFrame

Examples

>>> from pynhd import PyGeoAPI
>>> pga = PyGeoAPI()
>>> gdf = pga.cross_section((-103.80119, 40.2684), width=1000.0, numpts=101, crs=4326)  
>>> print(gdf.iloc[-1, 1])  
1000.0

elevation_profile(line, numpts, dem_res, crs=4326)

Return a GeoDataFrame from the xsatpathpts service.

Parameters:

• line (shapely.LineString or shapely.MultiLineString) – The line to extract the elevation profile for.

• numpts (int) – The number of points to extract the elevation profile from the DEM.

• dem_res (int) – The target resolution for requesting the DEM from 3DEP service.

• crs (str, int, or pyproj.CRS, optional) – The coordinate reference system of the coordinates, defaults to EPSG:4326.

Returns:

geopandas.GeoDataFrame – A GeoDataFrame containing the elevation profile along the requested endpoints.

Return type:

geopandas.GeoDataFrame

Examples

>>> from pynhd import PyGeoAPI
>>> from shapely import LineString
>>> pga = PyGeoAPI()
>>> line = LineString([(-103.801086, 40.26772), (-103.80097, 40.270568)])
>>> gdf = pga.elevation_profile(line, 101, 1, 4326)  
>>> print(gdf.iloc[-1, 2])  
1299.8727

endpoints_profile(coords, numpts, dem_res, crs=4326)

Return a GeoDataFrame from the xsatendpts service.

Parameters:

• coords (list) – A list of two coordinates to trace as a list of tuples, e.g., [(x1, y1), (x2, y2)].

• numpts (int) – The number of points to extract the elevation profile from the DEM.

• dem_res (int) – The target resolution for requesting the DEM from 3DEP service.

• crs (str, int, or pyproj.CRS, optional) – The coordinate reference system of the coordinates, defaults to EPSG:4326.

Returns:

geopandas.GeoDataFrame – A GeoDataFrame containing the elevation profile along the requested endpoints.

Return type:

geopandas.GeoDataFrame

Examples

>>> from pynhd import PyGeoAPI
>>> pga = PyGeoAPI()
>>> gdf = pga.endpoints_profile(
...     [(-103.801086, 40.26772), (-103.80097, 40.270568)], numpts=101, dem_res=1, crs=4326
... )  
>>> print(gdf.iloc[-1, 1])  
411.5906

flow_trace(coord, crs=4326, direction='none')

Return a GeoDataFrame from the flowtrace service.

Parameters:

• coord (tuple) – The coordinate of the point to trace as a tuple,e.g., (lon, lat).

• crs (str) – The coordinate reference system of the coordinates, defaults to EPSG:4326.

• direction (str, optional) – The direction of flowpaths, either down, up, or none. Defaults to none.

Returns:

geopandas.GeoDataFrame – A GeoDataFrame containing the traced flowline.

Return type:

geopandas.GeoDataFrame

Examples

>>> from pynhd import PyGeoAPI
>>> pga = PyGeoAPI()
>>> gdf = pga.flow_trace(
...     (1774209.63, 856381.68), crs="ESRI:102003", direction="none"
... )  
>>> print(gdf.comid.iloc[0])  
22294818

split_catchment(coord, crs=4326, upstream=False)

Return a GeoDataFrame from the splitcatchment service.

Parameters:

• coord (tuple) – The coordinate of the point to trace as a tuple,e.g., (lon, lat).

• crs (str, int, or pyproj.CRS, optional) – The coordinate reference system of the coordinates, defaults to EPSG:4326.

• upstream (bool, optional) – If True, return all upstream catchments rather than just the local catchment, defaults to False.

Returns:

geopandas.GeoDataFrame – A GeoDataFrame containing the local catchment or the entire upstream catchments.

Return type:

geopandas.GeoDataFrame

Examples

>>> from pynhd import PyGeoAPI
>>> pga = PyGeoAPI()
>>> gdf = pga.split_catchment((-73.82705, 43.29139), crs=4326, upstream=False)  
>>> print(gdf.catchmentID.iloc[0])  
22294818

class pynhd.pynhd.WaterData(layer, crs=4326)

Access to WaterData service.

Parameters:

• layer (str) – A valid layer from the WaterData service. Valid layers are:

  • catchmentsp

  • gagesii

  • gagesii_basins

  • nhdarea

  • nhdflowline_network

  • nhdflowline_nonnetwork

  • nhdwaterbody

  • wbd02

  • wbd04

  • wbd06

  • wbd08

  • wbd10

  • wbd12

  Note that the layers’ namespace for the WaterData service is wmadata and will be added to the given layer argument if it is not provided.

• crs (str, int, or pyproj.CRS, optional) – The target spatial reference system, defaults to epsg:4326.

• validation (bool, optional) – Whether to validate the input data, defaults to True.

bybox(bbox, box_crs=4326, sort_attr=None)

Get features within a bounding box.

Parameters:

• bbox (tuple of floats) – A bounding box in the form of (minx, miny, maxx, maxy).

• box_crs (str, int, or pyproj.CRS, optional) – The spatial reference system of the bounding box, defaults to epsg:4326.

• sort_attr (str, optional) – The column name in the database to sort request by, defaults to the first attribute in the schema that contains id in its name.

Returns:

geopandas.GeoDataFrame – The requested features in a GeoDataFrames.

Return type:

geopandas.GeoDataFrame

bydistance(coords, distance, loc_crs=4326, sort_attr=None)

Get features within a radius (in meters) of a point.

Parameters:

• coords (tuple of float) – The x, y coordinates of the point.

• distance (int) – The radius (in meters) to search within.

• loc_crs (str, int, or pyproj.CRS, optional) – The CRS of the input coordinates, default to epsg:4326.

• sort_attr (str, optional) – The column name in the database to sort request by, defaults to the first attribute in the schema that contains id in its name.

Returns:

geopandas.GeoDataFrame – Requested features as a GeoDataFrame.

Return type:

geopandas.GeoDataFrame

byfilter(cql_filter, method='GET', sort_attr=None)

Get features based on a CQL filter.

Parameters:

• cql_filter (str) – The CQL filter to use for requesting the data.

• method (str, optional) – The HTTP method to use for requesting the data, defaults to GET. Allowed methods are GET and POST.

• sort_attr (str, optional) – The column name in the database to sort request by, defaults to the first attribute in the schema that contains id in its name.

Returns:

geopandas.GeoDataFrame – The requested features as a GeoDataFrames.

Return type:

geopandas.GeoDataFrame

bygeom(geometry, geo_crs=4326, xy=True, predicate='intersects', sort_attr=None)

Get features within a geometry.

Parameters:

• geometry (shapely.Polygon or shapely.MultiPolygon) – The input (multi)polygon to request the data.

• geo_crs (str, int, or pyproj.CRS, optional) – The CRS of the input geometry, default to epsg:4326.

• xy (bool, optional) – Whether axis order of the input geometry is xy or yx.

• predicate (str, optional) – The geometric prediacte to use for requesting the data, defaults to INTERSECTS. Valid predicates are:

  • equals

  • disjoint

  • intersects

  • touches

  • crosses

  • within

  • contains

  • overlaps

  • relate

  • beyond

• sort_attr (str, optional) – The column name in the database to sort request by, defaults to the first attribute in the schema that contains id in its name.

Returns:

geopandas.GeoDataFrame – The requested features in the given geometry.

Return type:

geopandas.GeoDataFrame

byid(featurename, featureids)

Get features based on IDs.

pynhd.pynhd.pygeoapi(geodf, service)

Return a GeoDataFrame from the flowtrace service.

Parameters:

• geodf (geopandas.GeoDataFrame) – A GeoDataFrame containing geometries to query. The required columns for each service are:

  • flow_trace: direction that indicates the direction of the flow trace. It can be up, down, or none (both directions).

  • split_catchment: upstream that indicates whether to return all upstream catchments or just the local catchment.

  • elevation_profile: numpts that indicates the number of points to extract along the flowpath and 3dep_res that indicates the target resolution for requesting the DEM from 3DEP service.

  • endpoints_profile: numpts that indicates the number of points to extract along the flowpath and 3dep_res that indicates the target resolution for requesting the DEM from 3DEP service.

  • cross_section: numpts that indicates the number of points to extract along the flowpath and width that indicates the width of the cross-section in meters.

• service (str) – The service to query, can be flow_trace, split_catchment, elevation_profile, endpoints_profile, or cross_section.

Returns:

geopandas.GeoDataFrame – A GeoDataFrame containing the results of requested service.

Return type:

geopandas.GeoDataFrame

Examples

>>> from shapely import Point
>>> import geopandas as gpd
>>> gdf = gpd.GeoDataFrame(
...     {
...         "direction": [
...             "none",
...         ]
...     },
...     geometry=[Point((1774209.63, 856381.68))],
...     crs="ESRI:102003",
... )
>>> trace = nhd.pygeoapi(gdf, "flow_trace")
>>> print(trace.comid.iloc[0])
22294818

Package Contents

pygeohydro

Top-level package for PyGeoHydro.

Submodules

pygeohydro.helpers

Some helper function for PyGeoHydro.

Module Contents

pygeohydro.helpers.get_us_states(subset_key=None)

Get US states as a GeoDataFrame from Census’ TIGERLine 2023 database.

Parameters:

subset_key (str or list of str, optional) – Key to subset the geometries instead of returning all states, by default all states are returned. Valid keys are:

• contiguous or conus

• continental

• commonwealths

• territories

• Two letter state codes, e.g., ["TX", "CA", "FL", ...]

Returns:

geopandas.GeoDataFrame – GeoDataFrame of requested US states.

Return type:

geopandas.GeoDataFrame

pygeohydro.helpers.nlcd_helper()

Get legends and properties of the NLCD cover dataset.

Notes

The following references have been used:

• jzmiller1/nlcd

• https://www.mrlc.gov/data-services-page

• https://www.mrlc.gov/data/legends/national-land-cover-database-2016-nlcd2016-legend

• https://doi.org/10.1111/jfr3.12347

Returns:

dict – Years where data is available and cover classes and categories, and roughness estimations.

Return type:

dict[str, Any]

pygeohydro.helpers.nwis_errors()

Get error code lookup table for USGS sites that have daily values.

pygeohydro.helpers.states_lookup_table()

Get codes and names of US states and their counties.

Notes

This function is based on a file prepared by developers of an R package called dataRetrieval.

Returns:

pandas.DataFrame – State codes and name as a dataframe.

Return type:

dict[str, StateCounties]

pygeohydro.nfhl

Accessing National Flood Hazard Layers (NFHL) through web services.

Module Contents

class pygeohydro.nfhl.NFHL(service, layer, outfields='*', crs=4326)

Access National Flood Hazard Layers (NFHL).

Parameters:

• service (str) – The service type. Valid services are:

  • NFHL: Effective National Flood Hazard Layers

  • Prelim_CSLF: Preliminary Changes Since Last Firm (CSLF)

  • Draft_CSLF: Draft Changes Since Last Firm (CSLF)

  • Prelim_NFHL: Preliminary National Flood Hazard Layers

  • Pending_NFHL: Pending National Flood Hazard Layers

  • Draft_NFHL: Draft National Flood Hazard Layers

• layer (str) – A valid service layer. Valid layers are service specific:

  • NFHL: nfhl availability, firm panels, lomrs, lomas,

    political jurisdictions, profile baselines, water lines, cross-sections, base flood elevations, levees, seclusion boundaries, coastal transects, transect baselines, general structures, river mile markers, water areas, plss, limit of moderate wave action, flood hazard boundaries, flood hazard zones, primary frontal dunes, base index, topographic low confidence areas, datum conversion points, coastal gages, gages, nodes, high water marks, station start points, hydrologic reaches, alluvial fans, and subbasins

  • Prelim_CSLF: preliminary, coastal high hazard area change,

    floodway change, special flood hazard area change, and non-special flood hazard area change

  • Draft_CSLF: draft, coastal high hazard area change,

    floodway change, special flood hazard area change, and non-special flood hazard area change

  • Prelim_NFHL: preliminary data availability,

    preliminary firm panel index, preliminary plss, preliminary topographic low confidence areas, preliminary river mile markers, preliminary datum conversion points, preliminary coastal gages, preliminary gages, preliminary nodes, preliminary high water marks, preliminary station start points, preliminary cross-sections, preliminary coastal transects, preliminary base flood elevations, preliminary profile baselines, preliminary transect baselines, preliminary limit of moderate wave action, preliminary water lines, preliminary political jurisdictions, preliminary levees, preliminary general structures, preliminary primary frontal dunes, preliminary hydrologic reaches, preliminary flood hazard boundaries, preliminary flood hazard zones, preliminary submittal information, preliminary alluvial fans, preliminary subbasins, and preliminary water areas

  • Pending_NFHL: pending submittal information, pending water areas,

    pending firm panel index, pending data availability, pending firm panels, pending political jurisdictions, pending profile baselines, pending water lines, pending cross-sections, pending base flood elevations, pending levees, pending seclusion boundaries, pending coastal transects, pending transect baselines, pending general structures, pending river mile markers, pending plss, pending limit of moderate wave action, pending flood hazard boundaries, pending flood hazard zones, pending primary frontal dunes, pending topographic low confidence areas, pending datum conversion points, pending coastal gages, pending gages, pending nodes, pending high water marks, pending station start points, pending hydrologic reaches, pending alluvial fans, and pending subbasins

  • Draft_NFHL: draft data availability, draft firm panels,

    draft political jurisdictions, draft profile baselines, draft water lines, draft cross-sections, draft base flood elevations, draft levees, draft submittal info, draft coastal transects, draft transect baselines, draft general structures, draft limit of moderate wave action, draft flood hazard boundaries, and draft flood hazard zones

• outfields (str or list, optional) – Target field name(s), default to “*” i.e., all the fields.

• crs (str, int, or pyproj.CRS, optional) – Target spatial reference of output, default to EPSG:4326.

Examples

>>> from pygeohydro import NFHL
>>> nfhl = NFHL("NFHL", "cross-sections")
>>> gdf_xs = nfhl.bygeom((-73.42, 43.28, -72.9, 43.52), geo_crs=4269)

References

• National Flood Hazard Layer

bygeom(geom, geo_crs=4326, sql_clause='', distance=None, return_m=False, return_geom=True)

Get features within a geometry that can be combined with a SQL where clause.

byids(field, fids, return_m=False, return_geom=True)

Get features by object IDs.

bysql(sql_clause, return_m=False, return_geom=True)

Get features using a valid SQL 92 WHERE clause.

property valid_services: dict[str, str]

A dictionary of valid services and their URLs.

pygeohydro.nlcd

Accessing data from the supported databases through their APIs.

Module Contents

pygeohydro.nlcd.cover_statistics(cover_da)

Percentages of the categorical NLCD cover data.

Parameters:

cover_da (xarray.DataArray) – Land cover DataArray from a LULC Dataset from the nlcd_bygeom function.

Returns:

Stats – A named tuple with the percentages of the cover classes and categories.

Return type:

pygeohydro.helpers.Stats

pygeohydro.nlcd.nlcd_area_percent(geo_df, year=2019, region='L48')

Compute the area percentages of the natural, developed, and impervious areas.

Notes

This function uses imperviousness and land use/land cover data from NLCD to compute the area percentages of the natural, developed, and impervious areas. It considers land cover classes of 21 to 24 as urban and the rest as natural. Then, uses imperviousness percentage to partition the urban area into developed and impervious areas. So, urban = developed + impervious and always natural + urban = natural + developed + impervious = 100.

Parameters:

• geometry (geopandas.GeoDataFrame or geopandas.GeoSeries) – A GeoDataFrame or GeoSeries with the geometry to query. The indices are used as keys in the output dictionary.

• year (int, optional) – Year of the NLCD data, defaults to 2019. Available years are 2021, 2019, 2016, 2013, 2011, 2008, 2006, 2004, and 2001.

• region (str, optional) – Region in the US that the input geometries are located, defaults to L48. Valid values are L48 (for CONUS), HI (for Hawaii), AK (for Alaska), and PR (for Puerto Rico). Both lower and upper cases are acceptable.

Returns:

pandas.DataFrame – A dataframe with the same index as input geo_df and columns are the area percentages of the natural, developed, impervious, and urban (sum of developed and impervious) areas. Sum of urban and natural percentages is always 100, as well as the sume of natural, developed, and impervious percentages.

Return type:

pandas.DataFrame

pygeohydro.nlcd.nlcd_bycoords(coords, years=None, region='L48', ssl=True)

Get data from NLCD database (2019).

Parameters:

• coords (list of tuple) – List of coordinates in the form of (longitude, latitude).

• years (dict, optional) – The years for NLCD layers as a dictionary, defaults to {'impervious': [2019], 'cover': [2019], 'canopy': [2019], "descriptor": [2019]}. Layers that are not in years are ignored, e.g., {'cover': [2016, 2019]} returns land cover data for 2016 and 2019.

• region (str, optional) – Region in the US that the input geometries are located, defaults to L48. Valid values are L48 (for CONUS), HI (for Hawaii), AK (for Alaska), and PR (for Puerto Rico). Both lower and upper cases are acceptable.

• ssl (bool, optional) – Whether to use SSL for the connection, defaults to True.

Returns:

geopandas.GeoDataFrame – A GeoDataFrame with the NLCD data and the coordinates.

Return type:

geopandas.GeoDataFrame

pygeohydro.nlcd.nlcd_bygeom(geometry, resolution=30, years=None, region='L48', crs=4326, ssl=True)

Get data from NLCD database (2019).

Parameters:

• geometry (geopandas.GeoDataFrame or geopandas.GeoSeries) – A GeoDataFrame or GeoSeries with the geometry to query. The indices are used as keys in the output dictionary.

• resolution (float, optional) – The data resolution in meters. The width and height of the output are computed in pixel based on the geometry bounds and the given resolution. The default is 30 m which is the native resolution of NLCD data.

• years (dict, optional) – The years for NLCD layers as a dictionary, defaults to {'impervious': [2019], 'cover': [2019], 'canopy': [2019], "descriptor": [2019]}. Layers that are not in years are ignored, e.g., {'cover': [2016, 2019]} returns land cover data for 2016 and 2019.

• region (str, optional) – Region in the US that the input geometries are located, defaults to L48. Valid values are L48 (for CONUS), HI (for Hawaii), AK (for Alaska), and PR (for Puerto Rico). Both lower and upper cases are acceptable.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference system to be used for requesting the data, defaults to epsg:4326.

• ssl (bool, optional) – Whether to use SSL for the connection, defaults to True.

Returns:

dict of xarray.Dataset or xarray.Dataset – A single or a dict of NLCD datasets. If dict, the keys are indices of the input GeoDataFrame.

Return type:

dict[int | str, xarray.Dataset]

pygeohydro.nlcd.overland_roughness(cover_da)

Estimate overland roughness from land cover data.

Parameters:

cover_da (xarray.DataArray) – Land cover DataArray from a LULC Dataset from the nlcd_bygeom function.

Returns:

xarray.DataArray – Overland roughness

Return type:

xarray.DataArray

pygeohydro.nwis

Accessing NWIS.

Module Contents

class pygeohydro.nwis.NWIS

Access NWIS web service.

Notes

More information about query parameters and codes that NWIS accepts can be found at its help webpage.

classmethod get_info(queries, expanded=False, fix_names=True, nhd_info=False)

Send multiple queries to USGS Site Web Service.

Parameters:

• queries (dict or list of dict) – A single or a list of valid queries.

• expanded (bool, optional) – Whether to get expanded site information for example drainage area, default to False.

• fix_names (bool, optional) – If True, reformat station names and some small annoyances, defaults to True.

• nhd_info (bool, optional) – If True, get NHD information for each site, defaults to False. This will add four new columns: nhd_comid, nhd_areasqkm, nhd_reachcode, and nhd_measure. Where nhd_id is the NHD COMID of the flowline that the site is located in, nhd_reachcode is the NHD Reach Code that the site is located in, and nhd_measure is the measure along the flowline that the site is located at.

Returns:

geopandas.GeoDataFrame – A correctly typed GeoDataFrame containing site(s) information.

Return type:

geopandas.GeoDataFrame

classmethod get_parameter_codes(keyword)

Search for parameter codes by name or number.

Notes

NWIS guideline for keywords is as follows:

By default an exact search is made. To make a partial search the term should be prefixed and suffixed with a % sign. The % sign matches zero or more characters at the location. For example, to find all with “discharge” enter %discharge% in the field. % will match any number of characters (including zero characters) at the location.

Parameters:

keyword (str) – Keyword to search for parameters by name of number.

Returns:

pandas.DataFrame – Matched parameter codes as a dataframe with their description.

Return type:

pandas.DataFrame

Examples

>>> from pygeohydro import NWIS
>>> nwis = NWIS()
>>> codes = nwis.get_parameter_codes("%discharge%")
>>> codes.loc[codes.parameter_cd == "00060", "parm_nm"].iloc[0]
'Discharge, cubic feet per second'

classmethod get_streamflow(station_ids: Sequence[str] | str, dates: tuple[str, str], freq: str = 'dv', mmd: bool = False, to_xarray: Literal[False] = ...) → pandas.DataFrame

classmethod get_streamflow(station_ids: Sequence[str] | str, dates: tuple[str, str], freq: str = 'dv', mmd: bool = False, to_xarray: Literal[True] = ...) → xarray.Dataset

Get mean daily streamflow observations from USGS.

Parameters:

• station_ids (str, list) – The gage ID(s) of the USGS station.

• dates (tuple) – Start and end dates as a tuple (start, end).

• freq (str, optional) – The frequency of the streamflow data, defaults to dv (daily values). Valid frequencies are dv (daily values), iv (instantaneous values). Note that for iv the time zone for the input dates is assumed to be UTC.

• mmd (bool, optional) – Convert cms to mm/day based on the contributing drainage area of the stations. Defaults to False.

• to_xarray (bool, optional) – Whether to return a xarray.Dataset. Defaults to False.

Returns:

pandas.DataFrame or xarray.Dataset – Streamflow data observations in cubic meter per second (cms). The stations that don’t provide the requested discharge data in the target period will be dropped. Note that when frequency is set to iv the time zone is converted to UTC.

static retrieve_rdb(url, payloads)

Retrieve and process requests with RDB format.

Parameters:

• url (str) – Name of USGS REST service, valid values are site, dv, iv, gwlevels, and stat. Please consult USGS documentation here for more information.

• payloads (list of dict) – List of target payloads.

Returns:

pandas.DataFrame – Requested features as a pandas’s DataFrame.

Return type:

pandas.DataFrame

pygeohydro.nwis.streamflow_fillna(streamflow, missing_max=5)

Fill missing data (NAN) in daily streamflow observations.

It drops stations with more than missing_max days missing data per year. Missing data in the remaining stations, are filled with day-of-year average over the entire dataset.

Parameters:

• discharge (xarray.DataArray or pandas.DataFrame or pandas.Series) – Daily streamflow observations with at least 10 years of daily data.

• missing_max (int) – Maximum allowed number of missing daily data per year for filling, defaults to 5.

Returns:

xarray.DataArray or pandas.DataFrame or pandas.Series – Streamflow observations with missing data filled for stations with less than missing_max days of missing data.

Return type:

ArrayLike

pygeohydro.plot

Plot hydrological signatures.

Plots include daily, monthly and annual hydrograph as well as regime curve (monthly mean) and flow duration curve.

Module Contents

pygeohydro.plot.prepare_plot_data(daily)

Generate a structured data for plotting hydrologic signatures.

Parameters:

daily (pandas.Series or pandas.DataFrame) – The data to be processed

Returns:

PlotDataType – Containing daily, ``mean_monthly, ranked, titles, and units fields.

Return type:

PlotDataType

pygeohydro.plot.signatures(discharge, precipitation=None, title=None, figsize=None, output=None, close=False)

Plot hydrological signatures w/ or w/o precipitation.

Plots includes daily hydrograph, regime curve (mean monthly) and flow duration curve. The input discharges are converted from cms to mm/day based on the watershed area, if provided.

Parameters:

• discharge (pd.DataFrame or pd.Series) – The streamflows in mm/day. The column names are used as labels on the plot and the column values should be daily streamflow.

• precipitation (pd.Series, optional) – Daily precipitation time series in mm/day. If given, the data is plotted on the second x-axis at the top.

• title (str, optional) – The plot supertitle.

• figsize (tuple, optional) – The figure size in inches, defaults to (9, 5).

• output (str, optional) – Path to save the plot as png, defaults to None which means the plot is not saved to a file.

• close (bool, optional) – Whether to close the figure.

pygeohydro.pygeohydro

Accessing data from the supported databases through their APIs.

Module Contents

class pygeohydro.pygeohydro.EHydro(data_type='points')

Access USACE Hydrographic Surveys (eHydro).

Notes

For more info visit: https://navigation.usace.army.mil/Survey/Hydro

bygeom(geom, geo_crs=4326, sql_clause='', distance=None, return_m=False, return_geom=True)

Get features within a geometry that can be combined with a SQL where clause.

byids(field, fids, return_m=False, return_geom=True)

Get features by object IDs.

bysql(sql_clause, return_m=False, return_geom=True)

Get features using a valid SQL 92 WHERE clause.

Parameters:

data_type (str, optional) – Type of the survey data to retrieve, defaults to points. Note that the points data type gets the best available point cloud data, i.e., if SurveyPointHD is available, it will be returned, otherwise SurveyPoint will be returned. Available types are:

• points: Point clouds

• outlines: Polygons of survey outlines

• contours: Depth contours

• bathymetry: Bathymetry data

Note that point clouds are not available for all surveys.

property survey_grid: geopandas.GeoDataFrame

Full survey availability on hexagonal grid cells of 35 km resolution.

class pygeohydro.pygeohydro.NID

Retrieve data from the National Inventory of Dams web service.

property df

Entire NID inventory (csv version) as a pandas.DataFrame.

property gdf

Entire NID inventory (gpkg version) as a geopandas.GeoDataFrame.

property nid_inventory_path: pathlib.Path

Path to the NID inventory feather file.

get_byfilter(query_list)

Query dams by filters from the National Inventory of Dams web service.

Parameters:

query_list (list of dict) – List of dictionary of query parameters. For an exhaustive list of the parameters, use the advanced fields dataframe that can be accessed via NID().fields_meta. Some filter require min/max values such as damHeight and drainageArea. For such filters, the min/max values should be passed like so: {filter_key: ["[min1 max1]", "[min2 max2]"]}.

Returns:

list of geopandas.GeoDataFrame – Query results in the same order as the input query list.

Return type:

list[geopandas.GeoDataFrame]

Examples

>>> from pygeohydro import NID
>>> nid = NID()
>>> query_list = [
...    {"drainageArea": ["[200 500]"]},
...    {"nidId": ["CA01222"]},
... ]
>>> dam_dfs = nid.get_byfilter(query_list)

get_bygeom(geometry, geo_crs)

Retrieve NID data within a geometry.

Parameters:

• geometry (Polygon, MultiPolygon, or tuple of length 4) – Geometry or bounding box (west, south, east, north) for extracting the data.

• geo_crs (list of str) – The CRS of the input geometry.

Returns:

geopandas.GeoDataFrame – GeoDataFrame of NID data

Return type:

geopandas.GeoDataFrame

Examples

>>> from pygeohydro import NID
>>> nid = NID()
>>> dams = nid.get_bygeom((-69.77, 45.07, -69.31, 45.45), 4326)

get_suggestions(text, context_key=None)

Get suggestions from the National Inventory of Dams web service.

Notes

This function is useful for exploring and/or narrowing down the filter fields that are needed to query the dams using get_byfilter.

Parameters:

• text (str) – Text to query for suggestions.

• context_key (str, optional) – Suggestion context, defaults to empty string, i.e., all context keys. For a list of valid context keys, see NID().fields_meta.

Returns:

tuple of pandas.DataFrame – The suggestions for the requested text as two DataFrames: First, is suggestions found in the dams properties and second, those found in the query fields such as states, huc6, etc.

Return type:

tuple[pandas.DataFrame, pandas.DataFrame]

Examples

>>> from pygeohydro import NID
>>> nid = NID()
>>> dams, contexts = nid.get_suggestions("houston", "city")

inventory_byid(federal_ids)

Get extra attributes for dams based on their dam ID.

Notes

This function is meant to be used for getting extra attributes for dams. For example, first you need to use either get_bygeom or get_byfilter to get basic attributes of the target dams. Then you can use this function to get extra attributes using the id column of the GeoDataFrame that get_bygeom or get_byfilter returns.

Parameters:

federal_ids (list of str) – List of the target dam Federal IDs.

Returns:

pandas.DataFrame – Dams with extra attributes in addition to the standard NID fields that other NID methods return.

Return type:

geopandas.GeoDataFrame

Examples

>>> from pygeohydro import NID
>>> nid = NID()
>>> dams = nid.inventory_byid(['KY01232', 'GA02400', 'NE04081', 'IL55070', 'TN05345'])

stage_nid_inventory(fname=None)

Download the entire NID inventory data and save to a feather file.

Parameters:

fname (str, pathlib.Path, optional) – The path to the file to save the data to, defaults to ./cache/nid_inventory.feather.

pygeohydro.pygeohydro.get_camels()

Get streaflow and basin attributes of all 671 stations in CAMELS dataset.

Notes

For more info on CAMELS visit: https://ral.ucar.edu/solutions/products/camels

Returns:

tuple of geopandas.GeoDataFrame and xarray.Dataset – The first is basin attributes as a geopandas.GeoDataFrame and the second is streamflow data and basin attributes as an xarray.Dataset.

Return type:

tuple[geopandas.GeoDataFrame, xarray.Dataset]

pygeohydro.pygeohydro.soil_gnatsgo(layers, geometry, crs=4326)

Get US soil data from the gNATSGO dataset.

Notes

This function uses Microsoft’s Planetary Computer service to get the data. The dataset’s description and its supported soil properties can be found at: https://planetarycomputer.microsoft.com/dataset/gnatsgo-rasters

Parameters:

• layers (list of str or str) – Target layer(s). Available layers can be found at the dataset’s website here.

• geometry (Polygon, MultiPolygon, or tuple of length 4) – Geometry or bounding box of the region of interest.

• crs (int, str, or pyproj.CRS, optional) – The input geometry CRS, defaults to epsg:4326.

Returns:

xarray.Dataset – Requested soil properties.

Return type:

xarray.Dataset

pygeohydro.pygeohydro.soil_properties(properties='*', soil_dir='cache')

Get soil properties dataset in the United States from ScienceBase.

Notes

This function downloads the source zip files from ScienceBase , extracts the included .tif files, and return them as an xarray.Dataset.

Parameters:

• properties (list of str or str, optional) – Soil properties to extract, default to “*”, i.e., all the properties. Available properties are awc for available water capacity, fc for field capacity, and por for porosity.

• soil_dir (str or pathlib.Pathlib.Path) – Directory to store zip files or if exists read from them, defaults to ./cache.

pygeohydro.pygeohydro.ssebopeta_bycoords(coords, dates, crs=4326)

Daily actual ET for a dataframe of coords from SSEBop database in mm/day.

Parameters:

• coords (pandas.DataFrame) – A dataframe with id, x, y columns.

• dates (tuple or list, optional) – Start and end dates as a tuple (start, end) or a list of years [2001, 2010, …].

• crs (str, int, or pyproj.CRS, optional) – The CRS of the input coordinates, defaults to epsg:4326.

Returns:

xarray.Dataset – Daily actual ET in mm/day as a dataset with time and location_id dimensions. The location_id dimension is the same as the id column in the input dataframe.

Return type:

xarray.Dataset

pygeohydro.pygeohydro.ssebopeta_bygeom(geometry, dates, geo_crs=4326)

Get daily actual ET for a region from SSEBop database.

Notes

Since there’s still no web service available for subsetting SSEBop, the data first needs to be downloaded for the requested period then it is masked by the region of interest locally. Therefore, it’s not as fast as other functions and the bottleneck could be the download speed.

Parameters:

• geometry (shapely.Polygon or tuple) – The geometry for downloading clipping the data. For a tuple bbox, the order should be (west, south, east, north).

• dates (tuple or list, optional) – Start and end dates as a tuple (start, end) or a list of years [2001, 2010, …].

• geo_crs (str, int, or pyproj.CRS, optional) – The CRS of the input geometry, defaults to epsg:4326.

Returns:

xarray.DataArray – Daily actual ET within a geometry in mm/day at 1 km resolution

Return type:

xarray.DataArray

pygeohydro.stnfloodevents

Access USGS Short-Term Network (STN) via Restful API.

Module Contents

class pygeohydro.stnfloodevents.STNFloodEventData

Client for STN Flood Event Data’s RESTFUL Service API.

Advantages of using this client are:

• The user does not need to know the details of RESTFUL in general and of this API specifically.

• Parses the data and returns Python objects (e.g., pandas.DataFrame, geopandas.GeoDataFrame) instead of JSON.

• Convenience functions are offered for data dictionaries.

• Geo-references the data where applicable.

service_url

The service url of the STN Flood Event Data RESTFUL Service API.

Type:

str

data_dictionary_url

The data dictionary url of the STN Flood Event Data RESTFUL Service API.

Type:

str

service_crs

The CRS of the data from the service which is EPSG:4326.

Type:

int

instruments_query_params

The accepted query parameters for the instruments data type. Accepted values are SensorType, CurrentStatus, States, Event, County, DeploymentType, EventType, EventStatus, and CollectionCondition.

Type:

set

peaks_query_params

The accepted query parameters for the peaks data type. Accepted values are EndDate, States, Event, StartDate, County, EventType, and EventStatus.

Type:

set

hwms_query_params

The accepted query parameters for the hwms data type. Accepted values are EndDate, States, Event, StartDate, County, EventType, and EventStatus.

Type:

set

sites_query_params

The accepted query parameters for the sites data type. Accepted values are OPDefined, HousingTypeOne, NetworkName, HousingTypeSeven, RDGOnly, HWMOnly, Event, SensorOnly, State, SensorType, and HWMSurveyed.

Type:

set

Notes

Point data from the service is assumed to be in the WGS84 coordinate reference system (EPSG:4326).

References

• USGS Short-Term Network (STN)

• All Sensors API Documentation

• All Peak Summary API Documentation

• All HWM API Documentation

• All Sites API Documentation

• USGS Flood Event Viewer: Providing Hurricane and Flood Response Data

• A USGS guide for finding and interpreting high-water marks

• High-Water Marks and Flooding

• Identifying and preserving high-water mark data

classmethod data_dictionary(data_type: str, as_dict: Literal[False] = False, async_retriever_kwargs: dict[str, Any] | None = ...) → pandas.DataFrame

classmethod data_dictionary(data_type: str, as_dict: Literal[True] = True, async_retriever_kwargs: dict[str, Any] | None = ...) → dict[str, Any]

Retrieve data dictionaries from the STN Flood Event Data API.

Parameters:

• data_type (str) – The data source from STN Flood Event Data API. It can be instruments, peaks, hwms, or sites.

• as_dict (bool, default = False) – If True, return the data dictionary as a dictionary. Otherwise, it returns as pandas.DataFrame.

• async_retriever_kwargs (dict, optional) – Additional keyword arguments to pass to async_retriever.retrieve_json(). The url and request_kwds options are already set.

Returns:

pandas.DataFrame or dict – The retrieved data dictionary as pandas.DataFrame or dict.

See also

get_all_data()

Retrieves all data for a given data type.

get_filtered_data()

Retrieves filtered data for a given data type.

Examples

>>> from pygeohydro.stnfloodevents import STNFloodEventData
>>> data = STNFloodEventData.data_dictionary(data_type="instruments", as_dict=False)
>>> data.shape[1]
2
>>> data.columns
Index(['Field', 'Definition'], dtype='object')

classmethod get_all_data(data_type: str, as_list: Literal[False] = False, crs: CRSTYPE = ..., async_retriever_kwargs: dict[str, Any] | None = ...) → geopandas.GeoDataFrame | pandas.DataFrame

classmethod get_all_data(data_type: str, as_list: Literal[True] = True, crs: CRSTYPE = ..., async_retriever_kwargs: dict[str, Any] | None = ...) → list[dict[str, Any]]

Retrieve all data from the STN Flood Event Data API.

Parameters:

• data_type (str) – The data source from STN Flood Event Data API. It can be instruments, peaks, hwms, or sites.

• as_list (bool, optional) – If True, return the data as a list, defaults to False.

• crs (int, str, or pyproj.CRS, optional) – Desired Coordinate reference system (CRS) of output. Only used for GeoDataFrames with hwms and sites data types.

• async_retriever_kwargs (dict, optional) – Additional keyword arguments to pass to async_retriever.retrieve_json(). The url and request_kwds options are already set.

Returns:

geopandas.GeoDataFrame or pandas.DataFrame or list of dict – The retrieved data as a GeoDataFrame, DataFrame, or a list of dictionaries.

Raises:

InputValueError – If the input data_type is not one of instruments, peaks, hwms, or sites

See also

get_filtered_data()

Retrieves filtered data for a given data type.

data_dictionary()

Retrieves the data dictionary for a given data type.

Notes

Notice schema differences between the data dictionaries, filtered data queries, and all data queries. This is a known issue and is being addressed by USGS.

Examples

>>> from pygeohydro.stnfloodevents import STNFloodEventData
>>> data = STNFloodEventData.get_all_data(data_type="instruments")
>>> data.shape[1]
18
>>> data.columns
Index(['instrument_id', 'sensor_type_id', 'deployment_type_id',
       'location_description', 'serial_number', 'interval', 'site_id',
       'event_id', 'inst_collection_id', 'housing_type_id', 'sensor_brand_id',
       'vented', 'instrument_status', 'data_files', 'files', 'last_updated',
       'last_updated_by', 'housing_serial_number'],
       dtype='object')

classmethod get_filtered_data(data_type: str, query_params: dict[str, Any] | None = ..., as_list: Literal[False] = False, crs: CRSTYPE = ..., async_retriever_kwargs: dict[str, Any] | None = ...) → geopandas.GeoDataFrame | pandas.DataFrame

classmethod get_filtered_data(data_type: str, query_params: dict[str, Any] | None = ..., as_list: Literal[True] = True, crs: CRSTYPE = ..., async_retriever_kwargs: dict[str, Any] | None = ...) → list[dict[str, Any]]

Retrieve filtered data from the STN Flood Event Data API.

Parameters:

• data_type (str) – The data source from STN Flood Event Data API. It can be instruments, peaks, hwms, or sites.

• query_params (dict, optional) – RESTFUL API query parameters. For accepted values, see the STNFloodEventData class attributes instruments_query_params, peaks_query_params, hwms_query_params, and sites_query_params for available values.

  Also, see the API documentation for each data type for more information:

  • instruments

  • peaks

  • hwms

  • sites

• as_list (bool, optional) – If True, return the data as a list, defaults to False.

• crs (int, str, or pyproj.CRS, optional) – Desired Coordinate reference system (CRS) of output. Only used for GeoDataFrames outputs.

• async_retriever_kwargs (dict, optional) – Additional keyword arguments to pass to async_retriever.retrieve_json(). The url and request_kwds options are already set.

Returns:

geopandas.GeoDataFrame or pandas.DataFrame or list of dict – The retrieved data as a GeoDataFrame, DataFrame, or a list of dictionaries.

Raises:

• InputValueError – If the input data_type is not one of instruments, peaks, hwms, or sites

• InputValueError – If any of the input query_params are not in accepted parameters (See instruments_query_params, peaks_query_params, hwms_query_params, or sites_query_params).

See also

get_all_data()

Retrieves all data for a given data type.

data_dictionary()

Retrieves the data dictionary for a given data type.

Notes

Notice schema differences between the data dictionaries, filtered data queries, and all data queries. This is a known issue and is being addressed by USGS.

Examples

>>> from pygeohydro.stnfloodevents import STNFloodEventData
>>> query_params = {"States": "SC, CA"}
>>> data = STNFloodEventData.get_filtered_data(data_type="instruments", query_params=query_params)
>>> data.shape[1]
34
>>> data.columns
Index(['sensorType', 'deploymentType', 'eventName', 'collectionCondition',
    'housingType', 'sensorBrand', 'statusId', 'timeStamp', 'site_no',
    'latitude', 'longitude', 'siteDescription', 'networkNames', 'stateName',
    'countyName', 'siteWaterbody', 'siteHDatum', 'sitePriorityName',
    'siteZone', 'siteHCollectMethod', 'sitePermHousing', 'instrument_id',
    'sensor_type_id', 'deployment_type_id', 'location_description',
    'serial_number', 'housing_serial_number', 'interval', 'site_id',
    'vented', 'instrument_status', 'data_files', 'files', 'geometry'],
    dtype='object')

pygeohydro.stnfloodevents.stn_flood_event(data_type, query_params=None)

Retrieve data from the STN Flood Event Data API.

Parameters:

• data_type (str) – The data source from STN Flood Event Data API. It can be instruments, peaks, hwms, or sites.

• query_params (dict, optional) – RESTFUL API query parameters, defaults to None which returns a pandas.DataFrame of information about the given data_type. For accepted values, see the STNFloodEventData class attributes instruments_query_params, peaks_query_params, hwms_query_params, and sites_query_params for available values.

  Also, see the API documentation for each data type for more information:

  • instruments

  • peaks

  • hwms

  • sites

Returns:

geopandas.GeoDataFrame or pandas.DataFrame – The retrieved data as a GeoDataFrame or DataFrame (if query_params is not passed).

Raises:

• InputValueError – If the input data_type is not one of instruments, peaks, hwms, or sites

• InputValueError – If any of the input query_params are not in accepted parameters.

Return type:

geopandas.GeoDataFrame | pandas.DataFrame

References

• USGS Short-Term Network (STN)

• Filtered Sensors API Documentation

• Peak Summary API Documentation

• Filtered HWM API Documentation

• Filtered Sites API Documentation

• USGS Flood Event Viewer: Providing Hurricane and Flood Response Data

• A USGS guide for finding and interpreting high-water marks

• High-Water Marks and Flooding

• Identifying and preserving high-water mark data

Notes

Notice schema differences between the data dictionaries, filtered data queries, and all data queries. This is a known issue and is being addressed by USGS.

Examples

>>> query_params = {"States": "SC, CA"}
>>> data = stn_flood_event("instruments", query_params=query_params)
>>> data.shape[1]
34
>>> data.columns
Index(['sensorType', 'deploymentType', 'eventName', 'collectionCondition',
    'housingType', 'sensorBrand', 'statusId', 'timeStamp', 'site_no',
    'latitude', 'longitude', 'siteDescription', 'networkNames', 'stateName',
    'countyName', 'siteWaterbody', 'siteHDatum', 'sitePriorityName',
    'siteZone', 'siteHCollectMethod', 'sitePermHousing', 'instrument_id',
    'sensor_type_id', 'deployment_type_id', 'location_description',
    'serial_number', 'housing_serial_number', 'interval', 'site_id',
    'vented', 'instrument_status', 'data_files', 'files', 'geometry'],
    dtype='object')

pygeohydro.us_abbrs

US states and territories Abbreviations from us package.

pygeohydro.waterdata

Accessing WaterData related APIs.

Module Contents

class pygeohydro.waterdata.SensorThings

Class for interacting with SensorThings API.

static odata_helper(columns=None, conditionals=None, expand=None, max_count=None, extra_params=None)

Generate Odata filters for SensorThings API.

Parameters:

• columns (list of str, optional) – Columns to be selected from the database, defaults to None.

• conditionals (str, optional) – Conditionals to be applied to the database, defaults to None. Note that the conditionals should have the form of cond1 operator 'value' and/or cond2 operator 'value. For example: properties/monitoringLocationType eq 'Stream' and ...

• expand (dict of dict, optional) – Expand the properties of the selected columns, defaults to None. Note that the expand should have the form of {Property: {func: value, ...}}. For example: {"Locations": {"select": "location", "filter": "ObservedProperty/@iot.id eq '00060'"}}

• max_count (int, optional) – Maximum number of items to be returned, defaults to None.

• extra_params (dict, optional) – Extra parameters to be added to the Odata filter, defaults to None.

Returns:

odata (dict) – Odata filter for the SensorThings API.

Return type:

dict[str, str]

query_byodata(odata, outformat='json')

Query the SensorThings API by Odata filter.

Parameters:

• odata (str) – Odata filter for the SensorThings API.

• outformat (str, optional) – Format of the response, defaults to json. Valid values are json and geojson.

Returns:

pandas.DataFrame or geopandas.GeoDataFrame – Requested data.

Return type:

geopandas.GeoDataFrame | pandas.DataFrame

sensor_info(sensor_ids)

Query the SensorThings API by a sensor ID.

Parameters:

sensor_ids (str or list of str) – A single or list of sensor IDs, e.g., USGS-09380000.

Returns:

pandas.DataFrame – Requested sensor data.

Return type:

pandas.DataFrame

sensor_property(sensor_property, sensor_ids)

Query a sensor property.

Parameters:

• sensor_property (str or list of str) – A sensor property, Valid properties are Datastreams, MultiDatastreams, Locations, HistoricalLocations, TaskingCapabilities.

• sensor_ids (str or list of str) – A single or list of sensor IDs, e.g., USGS-09380000.

Returns:

pandas.DataFrame – A dataframe containing the requested property.

Return type:

pandas.DataFrame

class pygeohydro.waterdata.WaterQuality

Water Quality Web Service https://www.waterqualitydata.us.

Notes

This class has a number of convenience methods to retrieve data from the Water Quality Data. Since there are many parameter combinations that can be used to retrieve data, a general method is also provided to retrieve data from any of the valid endpoints. You can use get_json to retrieve stations info as a geopandas.GeoDataFrame or get_csv to retrieve stations data as a pandas.DataFrame. You can construct a dictionary of the parameters and pass it to one of these functions. For more information on the parameters, please consult the Water Quality Data documentation.

data_bystation(station_ids, wq_kwds)

Retrieve data for a single station.

Parameters:

• station_ids (str or list of str) – Station ID(s). The IDs should have the format “Agency code-Station ID”.

• wq_kwds (dict, optional) – Water Quality Web Service keyword arguments. Default to None.

Returns:

pandas.DataFrame – DataFrame of data for the stations.

Return type:

pandas.DataFrame

get_csv(endpoint, kwds, request_method='GET')

Get the CSV response from the Water Quality Web Service.

Parameters:

• endpoint (str) – Endpoint of the Water Quality Web Service.

• kwds (dict) – Water Quality Web Service keyword arguments.

• request_method (str, optional) – HTTP request method. Default to GET.

Returns:

pandas.DataFrame – The web service response as a DataFrame.

Return type:

pandas.DataFrame

get_json(endpoint, kwds, request_method='GET')

Get the JSON response from the Water Quality Web Service.

Parameters:

• endpoint (str) – Endpoint of the Water Quality Web Service.

• kwds (dict) – Water Quality Web Service keyword arguments.

• request_method (str, optional) – HTTP request method. Default to GET.

Returns:

geopandas.GeoDataFrame – The web service response as a GeoDataFrame.

Return type:

geopandas.GeoDataFrame

get_param_table()

Get the parameter table from the USGS Water Quality Web Service.

lookup_domain_values(endpoint)

Get the domain values for the target endpoint.

station_bybbox(bbox, wq_kwds)

Retrieve station info within bounding box.

Parameters:

• bbox (tuple of float) – Bounding box coordinates (west, south, east, north) in epsg:4326.

• wq_kwds (dict, optional) – Water Quality Web Service keyword arguments. Default to None.

Returns:

geopandas.GeoDataFrame – GeoDataFrame of station info within the bounding box.

Return type:

geopandas.GeoDataFrame

station_bydistance(lon, lat, radius, wq_kwds)

Retrieve station within a radius (decimal miles) of a point.

Parameters:

• lon (float) – Longitude of point.

• lat (float) – Latitude of point.

• radius (float) – Radius (decimal miles) of search.

• wq_kwds (dict, optional) – Water Quality Web Service keyword arguments. Default to None.

Returns:

geopandas.GeoDataFrame – GeoDataFrame of station info within the radius of the point.

Return type:

geopandas.GeoDataFrame

pygeohydro.watershed

Accessing watershed boundary-level data through web services.

Module Contents

class pygeohydro.watershed.WBD(layer, outfields='*', crs=4326)

Access Watershed Boundary Dataset (WBD).

Notes

This web service offers Hydrologic Unit (HU) polygon boundaries for the United States, Puerto Rico, and the U.S. Virgin Islands. For more info visit: https://hydro.nationalmap.gov/arcgis/rest/services/wbd/MapServer

Parameters:

• layer (str, optional) – A valid service layer. Valid layers are:

  • wbdline

  • huc2

  • huc4

  • huc6

  • huc8

  • huc10

  • huc12

  • huc14

  • huc16

• outfields (str or list, optional) – Target field name(s), default to “*” i.e., all the fields.

• crs (str, int, or pyproj.CRS, optional) – Target spatial reference, default to EPSG:4326.

pygeohydro.watershed.huc_wb_full(huc_lvl)

Get the full watershed boundary for a given HUC level.

Notes

This function is designed for cases where the full watershed boundary is needed for a given HUC level. If only a subset of the HUCs is needed, then use the pygeohydro.WBD class. The full dataset is downloaded from the National Maps’ WBD staged products.

Parameters:

huc_lvl (int) – HUC level, must be even numbers between 2 and 16.

Returns:

geopandas.GeoDataFrame – The full watershed boundary for the given HUC level.

Return type:

geopandas.GeoDataFrame

pygeohydro.watershed.irrigation_withdrawals()

Get monthly water use for irrigation at HUC12-level for CONUS.

Notes

Dataset is retrieved from https://doi.org/10.5066/P9FDLY8P.

Package Contents

py3dep

Top-level package for Py3DEP.

Submodules

py3dep.py3dep

Get data from 3DEP database.

Module Contents

py3dep.py3dep.add_elevation(ds, resolution=None, x_dim='x', y_dim='y', mask=None)

Add elevation data to a dataset as a new variable.

Parameters:

• ds (xarray.DataArray or xarray.Dataset) – The dataset to add elevation data to. It must contain CRS information.

• resolution (float, optional) – Target DEM source resolution in meters, defaults None, i.e., the resolution of the input ds will be used.

• x_dim (str, optional) – Name of the x-coordinate dimension in ds, defaults to x.

• y_dim (str, optional) – Name of the y-coordinate dimension in ds, defaults to y.

• mask (xarray.DataArray, optional) – A mask to apply to the elevation data, defaults to None.

Returns:

xarray.Dataset – The dataset with elevation variable added.

Return type:

xarray.Dataset

py3dep.py3dep.check_3dep_availability(bbox, crs=4326)

Query 3DEP’s resolution availability within a bounding box.

This function checks availability of 3DEP’s at the following resolutions: 1 m, 3 m, 5 m, 10 m, 30 m, 60 m, and topobathy (integrated topobathymetry).

Parameters:

• bbox (tuple) – Bounding box as tuple of (min_x, min_y, max_x, max_y).

• crs (str, int, or pyproj.CRS or pyproj.CRS, optional) – Spatial reference (CRS) of bbox, defaults to EPSG:4326.

Returns:

dict – True if bbox intersects 3DEP elevation for each available resolution. Keys are the supported resolutions and values are their availability. If the query fails due to any reason, the value will be Failed. If necessary, you can try again later until there is no Failed value.

Return type:

dict[str, bool | str]

Examples

>>> import py3dep
>>> bbox = (-69.77, 45.07, -69.31, 45.45)
>>> py3dep.check_3dep_availability(bbox)
{'1m': True, '3m': False, '5m': False, '10m': True, '30m': True, '60m': False, 'topobathy': False}

py3dep.py3dep.elevation_bycoords(coords: tuple[float, float], crs: CRSTYPE = ..., source: Literal[tep, tnm] = ...) → float

py3dep.py3dep.elevation_bycoords(coords: list[tuple[float, float]], crs: CRSTYPE = ..., source: Literal[tep, tnm] = ...) → list[float]

Get elevation for a list of coordinates.

Parameters:

• coords (tuple or list of tuple) – Coordinates of target location(s), e.g., [(x, y), ...].

• crs (str, int, or pyproj.CRS or pyproj.CRS, optional) – Spatial reference (CRS) of coords, defaults to EPSG:4326.

• source (str, optional) – Data source to be used, default to tep. Supported sources are tnm (using The National Map’s Bulk Point Query Service with 10 m resolution) and tep (using 3DEP’s static DEM VRTs at 10 m resolution). The tnm and tep sources are more accurate since they use the 1/3 arc-second DEM layer from 3DEP service but it is limited to the US. Note that tnm is bit unstable. It’s recommended to use tep unless 10-m resolution accuracy is not necessary.

Returns:

float or list of float – Elevation in meter.

py3dep.py3dep.elevation_bygrid(xcoords, ycoords, crs, resolution, depression_filling=False)

Get elevation from DEM data for a grid.

This function is intended for getting elevations for a gridded dataset.

Parameters:

• xcoords (list) – List of x-coordinates of a grid.

• ycoords (list) – List of y-coordinates of a grid.

• crs (str, int, or pyproj.CRS or pyproj.CRS) – The spatial reference system of the input grid, defaults to EPSG:4326.

• resolution (int) – The accuracy of the output, defaults to 10 m which is the highest available resolution that covers CONUS. Note that higher resolution increases computation time so chose this value with caution.

• depression_filling (bool, optional) – Fill depressions before sampling using pyflwdir package, defaults to False.

Returns:

xarray.DataArray – Elevations of the input coordinates as a xarray.DataArray.

Return type:

xarray.DataArray

py3dep.py3dep.elevation_profile(lines, spacing, crs=4326)

Get the elevation profile along a line at a given uniform spacing.

Note

This function converts the line to a spline and then calculates the elevation along the spline at a given uniform spacing using 10-m resolution DEM from 3DEP.

Parameters:

• lines (LineString or MultiLineString) – Line segment(s) to be profiled. If its type is MultiLineString, it will be converted to a single LineString and if this operation fails, an InputTypeError will be raised.

• spacing (float) – Spacing between the sample points along the line in meters.

• crs (str, int, or pyproj.CRS, optional) – Spatial reference System (CRS) of lines, defaults to EPSG:4326.

Returns:

xarray.DataArray – Elevation profile with dimension z and three coordinates: x, y, and distance. The distance coordinate is the distance from the start of the line in meters.

Return type:

xarray.DataArray

py3dep.py3dep.get_dem(geometry, resolution, crs=4326)

Get DEM data at any resolution from 3DEP.

Notes

This function is a wrapper of static_3dep_dem and get_map functions. Since static_3dep_dem is much faster, if the requested resolution is 10 m, 30 m, or 60 m, static_3dep_dem will be used. Otherwise, get_map will be used.

Parameters:

• geometry (Polygon, MultiPolygon, or tuple of length 4) – Geometry to get DEM within. It can be a polygon or a boundong box of form (xmin, ymin, xmax, ymax).

• resolution (int) – Target DEM source resolution in meters.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference system of the input geometry, defaults to EPSG:4326.

Returns:

xarray.DataArray – DEM at the specified resolution in meters and 4326 CRS.

Return type:

xarray.DataArray

py3dep.py3dep.get_dem_vrt(bbox, resolution, vrt_path, tiff_dir='cache', crs=4326)

Get DEM data at any resolution from 3DEP and save it as a VRT file.

Parameters:

• bbox (tuple of length 4) – The boundong box of form (xmin, ymin, xmax, ymax).

• resolution (int) – Target DEM source resolution in meters.

• vrt_path (str or pathlib.Path) – Path to the output VRT file.

• tiff_dir (str or pathlib.Path, optional) – Path to the directory to save the downloaded TIFF file, defaults to ./cache.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference system of bbox, defaults to EPSG:4326.

py3dep.py3dep.get_map(layers: str, geometry: shapely.Polygon | shapely.MultiPolygon | tuple[float, float, float, float], resolution: int, geo_crs: CRSTYPE = ..., crs: CRSTYPE = ...) → xarray.DataArray

py3dep.py3dep.get_map(layers: list[str], geometry: shapely.Polygon | shapely.MultiPolygon | tuple[float, float, float, float], resolution: int, geo_crs: CRSTYPE = ..., crs: CRSTYPE = ...) → xarray.Dataset

Access dynamic layer of 3DEP.

The 3DEP service has multi-resolution sources, so depending on the user provided resolution the data is resampled on server-side based on all the available data sources. The following layers are available:

• DEM

• Hillshade Gray

• Aspect Degrees

• Aspect Map

• GreyHillshade_elevationFill

• Hillshade Multidirectional

• Slope Map

• Slope Degrees

• Hillshade Elevation Tinted

• Height Ellipsoidal

• Contour 25

• Contour Smoothed 25

Parameters:

• layers (str or list of str) – A valid 3DEP layer or a list of them.

• geometry (Polygon, MultiPolygon, or tuple) – A shapely Polygon or a bounding box of the form (west, south, east, north).

• resolution (int) – The target resolution in meters. The width and height of the output are computed in pixels based on the geometry bounds and the given resolution.

• geo_crs (str, int, or pyproj.CRS, optional) – The spatial reference system of the input geometry, defaults to EPSG:4326.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference system to be used for requesting the data, defaults to EPSG:4326. Valid values are EPSG:4326, EPSG:3576, EPSG:3571, EPSG:3575, EPSG:3857, EPSG:3572, CRS:84, EPSG:3573, and EPSG:3574.

Returns:

xarray.DataArray or xarray.Dataset – The requested topographic data as an xarray.DataArray or xarray.Dataset.

py3dep.py3dep.query_3dep_sources(bbox, crs=4326, res=None)

Query 3DEP’s data sources within a bounding box.

This function queries the availability of the underlying data that 3DEP uses at the following resolutions: 1 m, 3 m, 5 m, 10 m, 30 m, 60 m, and topobathy (integrated topobathymetry).

Parameters:

• bbox (tuple) – Bounding box as tuple of (min_x, min_y, max_x, max_y).

• crs (str, int, or pyproj.CRS or pyproj.CRS, optional) – Spatial reference (CRS) of bbox, defaults to EPSG:4326.

• res (str, list of str, optional) – Resolution to query, defaults to None, i.e., all resolutions. Available resolutions are: 1m, 3m, 5m, 10m, 30m, 60m, and topobathy.

Returns:

geopandas.GeoDataFrame – Polygon(s) representing the 3DEP data sources at each resolution. Resolutions are given in the dem_res column.

Return type:

geopandas.GeoDataFrame

Examples

>>> import py3dep
>>> bbox = (-69.77, 45.07, -69.31, 45.45)
>>> src = py3dep.query_3dep_sources(bbox)
>>> src.groupby("dem_res")["OBJECTID"].count().to_dict()
{'10m': 8, '1m': 3, '30m': 8}
>>> src = py3dep.query_3dep_sources(bbox, res="1m")
>>> src.groupby("dem_res")["OBJECTID"].count().to_dict()
{'1m': 3}

py3dep.py3dep.static_3dep_dem(geometry, crs, resolution=10)

Get DEM data at specific resolution from 3DEP.

Notes

In contrast to get_map function, this function only gets DEM data at specific resolution, namely 10 m, 30 m, and 60 m. However, this function is faster. This function is intended for cases where only need DEM at a specific resolution is required and for the other requests get_map should be used.

Parameters:

• geometry (Polygon, MultiPolygon, or tuple of length 4) – Geometry to get DEM within. It can be a polygon or a boundong box of form (xmin, ymin, xmax, ymax).

• crs (int, str, of pyproj.CRS) – CRS of the input geometry.

• resolution (int, optional) – Target DEM source resolution in meters, defaults to 10 m which is the highest resolution available over the US. Available options are 10, 30, and 60.

Returns:

xarray.DataArray – The request DEM at the specified resolution.

Return type:

xarray.DataArray

py3dep.utils

Utilities for Py3DEP.

Module Contents

py3dep.utils.deg2mpm(slope)

Convert slope from degrees to meter/meter.

Parameters:

slope (xarray.DataArray) – Slope in degrees.

Returns:

xarray.DataArray – Slope in meter/meter. The name is set to slope and the units attribute is set to m/m.

Return type:

xarray.DataArray

py3dep.utils.fill_depressions(elevtn, outlets='min', idxs_pit=None, nodata=np.nan, max_depth=-1.0, elv_max=None, connectivity=8)

Fill local depressions in elevation data based on Wang and Liu (2006).

Note

This function is based on the fill_depressions function from the pyflwdir package. This function improves the performance of the original function by a factor of up to 2 and adds more input checks. Additionally, it works with xarray.DataArray objects.

Outlets are assumed to occur at the edge of valid elevation cells outlets='edge'; at the lowest valid edge cell to create one single outlet outlets='min'; or at user provided outlet cells idxs_pit.

Depressions elsewhere are filled based on its lowest pour point elevation. If the pour point depth is larger than the maximum pour point depth max_depth a pit is set at the depression local minimum elevation.

Wang, L., & Liu, H. (2006). https://doi.org/10.1080/13658810500433453

Parameters:

• elevtn (numpy.ndarray or xarray.DataArray) – elevation raster as a 2D numpy.ndarray or xarray.DataArray.

• outlets ({"edge", "min}, optional) – Initial basin outlet(s) at the edge of all cells (‘edge’) or only the minimum elevation edge cell (‘min’; default)

• idxs_pit (1D array of int, optional) – Linear indices of outlet cells, in any, defaults to None.

• nodata (float, optional) – nodata value, defaults to numpy.nan.

• max_depth (float, optional) – Maximum pour point depth. Depressions with a larger pour point depth are set as pit. A negative value (default) equals an infitely large pour point depth causing all depressions to be filled. Defaults to -1.0.

• elv_max, float, optional – Maximum elevation for outlets, only in combination with outlets='edge'. By default None.

• connectivity ({4, 8}, optional) – Number of neighboring cells to consider, defaults to 8.

Returns:

elevtn_out (numpy.ndarray) – Depression filled elevation with type float32.

Return type:

DataArray

Package Contents

pydaymet

Top-level package for PyDaymet.

Submodules

pydaymet.core

Core class for the Daymet functions.

Module Contents

class pydaymet.core.Daymet(variables=None, pet=None, snow=False, time_scale='daily', region='na')

Base class for Daymet requests.

Parameters:

• variables (str or list or tuple, optional) – List of variables to be downloaded. The acceptable variables are: tmin, tmax, prcp, srad, vp, swe, dayl Descriptions can be found here. Defaults to None i.e., all the variables are downloaded.

• pet (str, optional) – Method for computing PET. Supported methods are penman_monteith, priestley_taylor, hargreaves_samani, and None (don’t compute PET). The penman_monteith method is based on Allen et al.[1] assuming that soil heat flux density is zero. The priestley_taylor method is based on Priestley and TAYLOR[2] assuming that soil heat flux density is zero. The hargreaves_samani method is based on Hargreaves and Samani[3]. Defaults to None.

• snow (bool, optional) – Compute snowfall from precipitation and minimum temperature. Defaults to False.

• time_scale (str, optional) – Data time scale which can be daily, monthly (monthly summaries), or annual (annual summaries). Defaults to daily.

• region (str, optional) – Region in the US, defaults to na. Acceptable values are:

  • na: Continental North America

  • hi: Hawaii

  • pr: Puerto Rico

References

[1]

Richard G Allen, Luis S Pereira, Dirk Raes, Martin Smith, and others. Crop evapotranspiration-guidelines for computing crop water requirements-fao irrigation and drainage paper 56. Fao, Rome, 300(9):D05109, 1998.

[2]

Charles Henry Brian Priestley and Robert Joseph TAYLOR. On the assessment of surface heat flux and evaporation using large-scale parameters. Monthly weather review, 100(2):81–92, 1972.

[3]

George H. Hargreaves and Zohrab A. Samani. Estimating potential evapotranspiration. Journal of the Irrigation and Drainage Division, 108(3):225–230, sep 1982. URL: https://doi.org/10.1061%2Fjrcea4.0001390, doi:10.1061/jrcea4.0001390.

static check_dates(dates)

Check if input dates are in correct format and valid.

dates_todict(dates)

Set dates by start and end dates as a tuple, (start, end).

dates_tolist(dates)

Correct dates for Daymet accounting for leap years.

Daymet doesn’t account for leap years and removes Dec 31 when it’s leap year.

Parameters:

dates (tuple) – Target start and end dates.

Returns:

list – All the dates in the Daymet database within the provided date range.

Return type:

list[tuple[pandas.Timestamp, pandas.Timestamp]]

years_todict(years)

Set date by list of year(s).

years_tolist(years)

Correct dates for Daymet accounting for leap years.

Daymet doesn’t account for leap years and removes Dec 31 when it’s leap year.

Parameters:

years (list) – A list of target years.

Returns:

list – All the dates in the Daymet database within the provided date range.

Return type:

list[tuple[pandas.Timestamp, pandas.Timestamp]]

pydaymet.core.separate_snow(clm, t_rain=T_RAIN, t_snow=T_SNOW)

Separate snow based on Martinez and Gupta[4].

Parameters:

• clm (pandas.DataFrame or xarray.Dataset) – Climate data that should include prcp and tmin.

• t_rain (float, optional) – Threshold for temperature for considering rain, defaults to 2.5 degrees C.

• t_snow (float, optional) – Threshold for temperature for considering snow, defaults to 0.6 degrees C.

Returns:

pandas.DataFrame or xarray.Dataset – Input data with snow (mm/day) column if input is a pandas.DataFrame, or snow variable if input is an xarray.Dataset.

Return type:

DF

References

[4]

Guillermo F. Martinez and Hoshin V. Gupta. Toward improved identification of hydrological models: a diagnostic evaluation of the “abcd” monthly water balance model for the conterminous united states. Water Resources Research, 2010. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2009WR008294, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2009WR008294, doi:https://doi.org/10.1029/2009WR008294.

pydaymet.pet

Core class for the Daymet functions.

Module Contents

pydaymet.pet.potential_et(clm: pandas.DataFrame, coords: tuple[float, float], crs: CRSTYPE, method: Literal[penman_monteith, priestley_taylor, hargreaves_samani] = ..., params: dict[str, float] | None = ...) → pandas.DataFrame

pydaymet.pet.potential_et(clm: xarray.Dataset, coords: None = None, crs: None = None, method: Literal[penman_monteith, priestley_taylor, hargreaves_samani] = ..., params: dict[str, float] | None = ...) → xarray.Dataset

Compute Potential EvapoTranspiration for both gridded and a single location.

Parameters:

• clm (pandas.DataFrame or xarray.Dataset) – The dataset must include at least the following variables:

  • Minimum temperature in degree celsius

  • Maximum temperature in degree celsius

  • Solar radiation in in W/m2

  • Daylight duration in seconds

  Optionally, for penman_monteith, wind speed at 2-m level will be used if available, otherwise, default value of 2 m/s will be assumed. Table below shows the variable names that the function looks for in the input data.

  pandas.DataFrame	xarray.Dataset
  tmin (degrees C)	tmin
  tmax (degrees C)	tmax
  srad (W/m2)	srad
  dayl (s)	dayl
  u2m (m/s)	u2m

• coords (tuple of floats, optional) – Coordinates of the daymet data location as a tuple, (x, y). This is required when clm is a DataFrame.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference of the input coordinate, defaults to EPSG:4326. This is only used when clm is a DataFrame.

• method (str, optional) – Method for computing PET. Supported methods are penman_monteith, priestley_taylor, and hargreaves_samani. The penman_monteith method is based on Allen et al.[1] assuming that soil heat flux density is zero. The priestley_taylor method is based on Priestley and TAYLOR[2] assuming that soil heat flux density is zero. The hargreaves_samani method is based on Hargreaves and Samani[3]. Defaults to hargreaves_samani.

• params (dict, optional) – Model-specific parameters as a dictionary, defaults to None. Valid parameters are:

  • penman_monteith: soil_heat_flux, albedo, alpha, and arid_correction.

  • priestley_taylor: soil_heat_flux, albedo, and arid_correction.

  • hargreaves_samani: None.

  Default values for the parameters are: soil_heat_flux = 0, albedo = 0.23, alpha = 1.26, and arid_correction = False. An important parameter for priestley_taylor and penman_monteith methods is arid_correction which is used to correct the actual vapor pressure for arid regions. Since relative humidity is not provided by Daymet, the actual vapor pressure is computed assuming that the dewpoint temperature is equal to the minimum temperature. However, for arid regions, FAO 56 suggests subtracting minimum temperature by 2-3 °C to account for the fact that in arid regions, the air might not be saturated when its temperature is at its minimum. For such areas, you can pass {"arid_correction": True, ...} to subtract 2 °C from the minimum temperature for computing the actual vapor pressure.

Returns:

pandas.DataFrame or xarray.Dataset – The input DataFrame/Dataset with an additional variable named pet (mm/day) for pandas.DataFrame and pet for xarray.Dataset.

References

[1]

Richard G Allen, Luis S Pereira, Dirk Raes, Martin Smith, and others. Crop evapotranspiration-guidelines for computing crop water requirements-fao irrigation and drainage paper 56. Fao, Rome, 300(9):D05109, 1998.

[2]

Charles Henry Brian Priestley and Robert Joseph TAYLOR. On the assessment of surface heat flux and evaporation using large-scale parameters. Monthly weather review, 100(2):81–92, 1972.

[3]

George H. Hargreaves and Zohrab A. Samani. Estimating potential evapotranspiration. Journal of the Irrigation and Drainage Division, 108(3):225–230, sep 1982. URL: https://doi.org/10.1061%2Fjrcea4.0001390, doi:10.1061/jrcea4.0001390.

pydaymet.pydaymet

Access the Daymet database for both single single pixel and gridded queries.

Module Contents

pydaymet.pydaymet.get_bycoords(coords, dates, coords_id=None, crs=4326, variables=None, region='na', time_scale='daily', pet=None, pet_params=None, snow=False, snow_params=None, ssl=True, to_xarray=False)

Get point-data from the Daymet database at 1-km resolution.

This function uses THREDDS data service to get the coordinates and supports getting monthly and annual summaries of the climate data directly from the server.

Parameters:

• coords (tuple or list of tuples) – Coordinates of the location(s) of interest as a tuple (x, y)

• dates (tuple or list) – Start and end dates as a tuple (start, end) or a list of years [2001, 2010, ...].

• coords_id (list of int or str, optional) – A list of identifiers for the coordinates. This option only applies when to_xarray is set to True. If not provided, the coordinates will be enumerated.

• crs (str, int, or pyproj.CRS, optional) – The CRS of the input coordinates, defaults to EPSG:4326.

• variables (str or list) – List of variables to be downloaded. The acceptable variables are: tmin, tmax, prcp, srad, vp, swe, dayl Descriptions can be found here.

• region (str, optional) – Target region in the US, defaults to na. Acceptable values are:

  • na: Continental North America

  • hi: Hawaii

  • pr: Puerto Rico

• time_scale (str, optional) – Data time scale which can be daily, monthly (monthly summaries), or annual (annual summaries). Defaults to daily.

• pet (str, optional) – Method for computing PET. Supported methods are penman_monteith, priestley_taylor, hargreaves_samani, and None (don’t compute PET). The penman_monteith method is based on Allen et al.[1] assuming that soil heat flux density is zero. The priestley_taylor method is based on Priestley and TAYLOR[2] assuming that soil heat flux density is zero. The hargreaves_samani method is based on Hargreaves and Samani[3]. Defaults to None.

• pet_params (dict, optional) – Model-specific parameters as a dictionary, defaults to None. An important parameter for priestley_taylor and penman_monteith methods is arid_correction which is used to correct the actual vapor pressure for arid regions. Since relative humidity is not provided by Daymet, the actual vapor pressure is computed assuming that the dewpoint temperature is equal to the minimum temperature. However, for arid regions, FAO 56 suggests subtracting the minimum temperature by 2-3 °C to account for aridity, since in arid regions, the air might not be saturated when its temperature is at its minimum. For such areas, you can pass {"arid_correction": True, ...} to subtract 2 °C from the minimum temperature before computing the actual vapor pressure.

• snow (bool, optional) – Compute snowfall from precipitation and minimum temperature. Defaults to False.

• snow_params (dict, optional) – Model-specific parameters as a dictionary that is passed to the snowfall function. These parameters are only used if snow is True. Two parameters are required: t_rain (deg C) which is the threshold for temperature for considering rain and t_snow (deg C) which is the threshold for temperature for considering snow. The default values are {'t_rain': 2.5, 't_snow': 0.6} that are adopted from https://doi.org/10.5194/gmd-11-1077-2018.

• ssl (bool, optional) – Whether to verify SSL certification, defaults to True.

• to_xarray (bool, optional) – Return the data as an xarray.Dataset. Defaults to False.

Returns:

pandas.DataFrame or xarray.Dataset – Daily climate data for a single or list of locations.

Return type:

pandas.DataFrame | xarray.Dataset

Examples

>>> import pydaymet as daymet
>>> coords = (-1431147.7928, 318483.4618)
>>> dates = ("2000-01-01", "2000-12-31")
>>> clm = daymet.get_bycoords(
...     coords,
...     dates,
...     crs=3542,
...     pet="hargreaves_samani",
... )
>>> clm["pet (mm/day)"].mean()
3.713

References

[1] (1,2,3)

Richard G Allen, Luis S Pereira, Dirk Raes, Martin Smith, and others. Crop evapotranspiration-guidelines for computing crop water requirements-fao irrigation and drainage paper 56. Fao, Rome, 300(9):D05109, 1998.

[2] (1,2,3)

Charles Henry Brian Priestley and Robert Joseph TAYLOR. On the assessment of surface heat flux and evaporation using large-scale parameters. Monthly weather review, 100(2):81–92, 1972.

[3] (1,2,3)

George H. Hargreaves and Zohrab A. Samani. Estimating potential evapotranspiration. Journal of the Irrigation and Drainage Division, 108(3):225–230, sep 1982. URL: https://doi.org/10.1061%2Fjrcea4.0001390, doi:10.1061/jrcea4.0001390.

pydaymet.pydaymet.get_bygeom(geometry, dates, crs=4326, variables=None, region='na', time_scale='daily', pet=None, pet_params=None, snow=False, snow_params=None, ssl=True)

Get gridded data from the Daymet database at 1-km resolution.

Parameters:

• geometry (Polygon, MultiPolygon, or bbox) – The geometry of the region of interest.

• dates (tuple or list) – Start and end dates as a tuple (start, end) or a list of years [2001, 2010, …].

• crs (str, int, or pyproj.CRS, optional) – The CRS of the input geometry, defaults to epsg:4326.

• variables (str or list) – List of variables to be downloaded. The acceptable variables are: tmin, tmax, prcp, srad, vp, swe, dayl Descriptions can be found here.

• region (str, optional) – Region in the US, defaults to na. Acceptable values are:

  • na: Continental North America

  • hi: Hawaii

  • pr: Puerto Rico

• time_scale (str, optional) – Data time scale which can be daily, monthly (monthly average), or annual (annual average). Defaults to daily.

• pet (str, optional) – Method for computing PET. Supported methods are penman_monteith, priestley_taylor, hargreaves_samani, and None (don’t compute PET). The penman_monteith method is based on Allen et al.[1] assuming that soil heat flux density is zero. The priestley_taylor method is based on Priestley and TAYLOR[2] assuming that soil heat flux density is zero. The hargreaves_samani method is based on Hargreaves and Samani[3]. Defaults to None.

• pet_params (dict, optional) – Model-specific parameters as a dictionary, defaults to None. Valid parameters are:

  • penman_monteith: soil_heat_flux, albedo, alpha, and arid_correction.

  • priestley_taylor: soil_heat_flux, albedo, and arid_correction.

  • hargreaves_samani: None.

  Default values for the parameters are: soil_heat_flux = 0, albedo = 0.23, alpha = 1.26, and arid_correction = False. An important parameter for priestley_taylor and penman_monteith methods is arid_correction which is used to correct the actual vapor pressure for arid regions. Since relative humidity is not provided by Daymet, the actual vapor pressure is computed assuming that the dewpoint temperature is equal to the minimum temperature. However, for arid regions, FAO 56 suggests subtracting the minimum temperature by 2-3 °C to account for aridity, since in arid regions, the air might not be saturated when its temperature is at its minimum. For such areas, you can pass {"arid_correction": True, ...} to subtract 2 °C from the minimum temperature before computing the actual vapor pressure.

• snow (bool, optional) – Compute snowfall from precipitation and minimum temperature. Defaults to False.

• snow_params (dict, optional) – Model-specific parameters as a dictionary that is passed to the snowfall function. These parameters are only used if snow is True. Two parameters are required: t_rain (deg C) which is the threshold for temperature for considering rain and t_snow (deg C) which is the threshold for temperature for considering snow. The default values are {'t_rain': 2.5, 't_snow': 0.6} that are adopted from https://doi.org/10.5194/gmd-11-1077-2018.

• ssl (bool, optional) – Whether to verify SSL certification, defaults to True.

Returns:

xarray.Dataset – Daily climate data within the target geometry.

Return type:

xarray.Dataset

Examples

>>> from shapely import Polygon
>>> import pydaymet as daymet
>>> geometry = Polygon(
...     [[-69.77, 45.07], [-69.31, 45.07], [-69.31, 45.45], [-69.77, 45.45], [-69.77, 45.07]]
... )
>>> clm = daymet.get_bygeom(geometry, 2010, variables="tmin", time_scale="annual")
>>> clm["tmin"].mean().item()
1.361

References

pydaymet.pydaymet.get_bystac(geometry, dates, crs=4326, variables=None, region='na', time_scale='daily', res_km=1, pet=None, pet_params=None, snow=False, snow_params=None)

Get gridded Daymet from STAC.

New in version 0.16.1.

Note

This function provides access to the Daymet data from Microsoft’s the Planetary Computer: https://planetarycomputer.microsoft.com/dataset/group/daymet. Although this function can be much faster than get_bygeom(), currently, it gives access to Daymet v4.2 from 1980 to 2020. For accessing the latest version of Daymet (v4.5) you need to use get_bygeom().

Also, this function requires fsspec, dask, zarr, and pystac-client packages. They can be installed using pip install fsspec dask zarr pystac-client or conda install fsspec dask-core zarr pystac-client.

Parameters:

• geometry (Polygon, MultiPolygon, or bbox) – The geometry of the region of interest.

• dates (tuple) – Start and end dates as a tuple (start, end) or a list of years [2001, 2010, …].

• crs (str, int, or pyproj.CRS, optional) – The CRS of the input geometry, defaults to epsg:4326.

• variables (str or list) – List of variables to be downloaded. The acceptable variables are: tmin, tmax, prcp, srad, vp, swe, dayl Descriptions can be found here.

• region (str, optional) – Region in the US, defaults to na. Acceptable values are:

  • na: Continental North America

  • hi: Hawaii

  • pr: Puerto Rico

• time_scale (str, optional) – Data time scale which can be daily, monthly (monthly average), or annual (annual average). Defaults to daily.

• res_km (int, optional) – Spatial resolution in kilometers, defaults to 1. For values greater than 1, the data will be aggregated (coarsend) using mean.

• pet (str, optional) – Method for computing PET. Supported methods are penman_monteith, priestley_taylor, hargreaves_samani, and None (don’t compute PET). The penman_monteith method is based on Allen et al.[1] assuming that soil heat flux density is zero. The priestley_taylor method is based on Priestley and TAYLOR[2] assuming that soil heat flux density is zero. The hargreaves_samani method is based on Hargreaves and Samani[3]. Defaults to None.

• pet_params (dict, optional) – Model-specific parameters as a dictionary, defaults to None. Valid parameters are:

  • penman_monteith: soil_heat_flux, albedo, alpha, and arid_correction.

  • priestley_taylor: soil_heat_flux, albedo, and arid_correction.

  • hargreaves_samani: None.

  Default values for the parameters are: soil_heat_flux = 0, albedo = 0.23, alpha = 1.26, and arid_correction = False. An important parameter for priestley_taylor and penman_monteith methods is arid_correction which is used to correct the actual vapor pressure for arid regions. Since relative humidity is not provided by Daymet, the actual vapor pressure is computed assuming that the dewpoint temperature is equal to the minimum temperature. However, for arid regions, FAO 56 suggests subtracting the minimum temperature by 2-3 °C to account for aridity, since in arid regions, the air might not be saturated when its temperature is at its minimum. For such areas, you can pass {"arid_correction": True, ...} to subtract 2 °C from the minimum temperature before computing the actual vapor pressure.

• snow (bool, optional) – Compute snowfall from precipitation and minimum temperature. Defaults to False.

• snow_params (dict, optional) – Model-specific parameters as a dictionary that is passed to the snowfall function. These parameters are only used if snow is True. Two parameters are required: t_rain (deg C) which is the threshold for temperature for considering rain and t_snow (deg C) which is the threshold for temperature for considering snow. The default values are {'t_rain': 2.5, 't_snow': 0.6} that are adopted from https://doi.org/10.5194/gmd-11-1077-2018.

• ssl (bool, optional) – Whether to verify SSL certification, defaults to True.

Returns:

xarray.Dataset – Daily climate data within the target geometry.

Return type:

xarray.Dataset

Examples

>>> from shapely import Polygon
>>> geometry = Polygon(
...     [[-69.77, 45.07], [-69.70, 45.07], [-69.70, 45.15], [-69.77, 45.15], [-69.77, 45.07]]
... )
>>> clm = daymet.get_bystac(
...     geometry,
...     ("2010-01-01", "2010-01-02"),
...     variables="tmin",
...     res_km=4,
...     snow=True,
...     pet="hargreaves_samani",
... )
>>> clm["pet"].mean().item()
0.3

References

Package Contents

pygridmet

Top-level package for PyGridMET.

Submodules

pygridmet.core

Core class for the GridMET functions.

Module Contents

class pygridmet.core.GridMET(dates=2000, variables=None, snow=False)

Base class for GridMET requests.

Parameters:

• dates (tuple or int or list, optional) – Start and end dates as a tuple, (start, end), or a list of years. Defaults to 2000 so the class can be initialized without any arguments.

• variables (str or list or tuple, optional) – List of variables to be downloaded. The acceptable variables are: pr, rmax, rmin, sph, srad, th, tmmn, tmmx, vs, bi, fm100, fm1000, erc, etr, pet, and vpd. Descriptions can be found here. Defaults to None, i.e., all the variables are downloaded.

• snow (bool, optional) – Compute snowfall from precipitation and minimum temperature. Defaults to False.

References

static check_dates(dates)

Check if input dates are in correct format and valid.

dates_todict(dates)

Set dates by start and end dates as a tuple, (start, end).

dates_tolist(dates)

Correct dates for GridMET accounting for leap years.

GridMET doesn’t account for leap years and removes Dec 31 when it’s leap year.

Parameters:

dates (tuple) – Target start and end dates.

Returns:

list – All the dates in the GridMET database within the provided date range.

Return type:

list[tuple[pandas.Timestamp, pandas.Timestamp]]

separate_snow(clm, t_rain=T_RAIN, t_snow=T_SNOW)

Separate snow based on Martinez and Gupta[1].

Parameters:

• clm (pandas.DataFrame or xarray.Dataset) – Climate data that should include pr and tmmn.

• t_rain (float, optional) – Threshold for temperature for considering rain, defaults to 2.5 K.

• t_snow (float, optional) – Threshold for temperature for considering snow, defaults to 0.6 K.

Returns:

pandas.DataFrame or xarray.Dataset – Input data with snow (mm) column if input is a pandas.DataFrame, or snow variable if input is an xarray.Dataset.

Return type:

DF

References

[1]

Guillermo F. Martinez and Hoshin V. Gupta. Toward improved identification of hydrological models: a diagnostic evaluation of the “abcd” monthly water balance model for the conterminous united states. Water Resources Research, 2010. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2009WR008294, arXiv:https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2009WR008294, doi:https://doi.org/10.1029/2009WR008294.

years_todict(years)

Set date by list of year(s).

years_tolist(years)

Correct dates for GridMET accounting for leap years.

GridMET doesn’t account for leap years and removes Dec 31 when it’s leap year.

Parameters:

years (list) – A list of target years.

Returns:

list – All the dates in the GridMET database within the provided date range.

Return type:

list[tuple[pandas.Timestamp, pandas.Timestamp]]

pygridmet.pygridmet

Access the GridMET database for both single single pixel and gridded queries.

Module Contents

pygridmet.pygridmet.get_bycoords(coords, dates, coords_id=None, crs=4326, variables=None, snow=False, snow_params=None, ssl=True, to_xarray=False)

Get point-data from the GridMET database at 1-km resolution.

Parameters:

• coords (tuple or list of tuples) – Coordinates of the location(s) of interest as a tuple (x, y)

• dates (tuple or list, optional) – Start and end dates as a tuple (start, end) or a list of years [2001, 2010, ...].

• coords_id (list of int or str, optional) – A list of identifiers for the coordinates. This option only applies when to_xarray is set to True. If not provided, the coordinates will be enumerated.

• crs (str, int, or pyproj.CRS, optional) – The CRS of the input coordinates, defaults to EPSG:4326.

• variables (str or list) – List of variables to be downloaded. The acceptable variables are: pr, rmax, rmin, sph, srad, th, tmmn, tmmx, vs, bi, fm100, fm1000, erc, etr, pet, and vpd. Descriptions can be found here. Defaults to None, i.e., all the variables are downloaded.

• snow (bool, optional) – Compute snowfall from precipitation and minimum temperature. Defaults to False.

• snow_params (dict, optional) – Model-specific parameters as a dictionary that is passed to the snowfall function. These parameters are only used if snow is True. Two parameters are required: t_rain (deg C) which is the threshold for temperature for considering rain and t_snow (deg C) which is the threshold for temperature for considering snow. The default values are {'t_rain': 2.5, 't_snow': 0.6} that are adopted from https://doi.org/10.5194/gmd-11-1077-2018.

• ssl (bool, optional) – Whether to verify SSL certification, defaults to True.

• to_xarray (bool, optional) – Return the data as an xarray.Dataset. Defaults to False.

Returns:

pandas.DataFrame or xarray.Dataset – Daily climate data for a single or list of locations.

Return type:

pandas.DataFrame | xarray.Dataset

Examples

>>> import pygridmet as gridmet
>>> coords = (-1431147.7928, 318483.4618)
>>> dates = ("2000-01-01", "2000-01-31")
>>> clm = gridmet.get_bycoords(
...     coords,
...     dates,
...     crs=3542,
... )
>>> clm["pr (mm)"].mean()
9.677

pygridmet.pygridmet.get_bygeom(geometry, dates, crs=4326, variables=None, snow=False, snow_params=None, ssl=True)

Get gridded data from the GridMET database at 1-km resolution.

Parameters:

• geometry (Polygon, MultiPolygon, or bbox) – The geometry of the region of interest.

• dates (tuple or list, optional) – Start and end dates as a tuple (start, end) or a list of years [2001, 2010, …].

• crs (str, int, or pyproj.CRS, optional) – The CRS of the input geometry, defaults to epsg:4326.

• variables (str or list) – List of variables to be downloaded. The acceptable variables are: pr, rmax, rmin, sph, srad, th, tmmn, tmmx, vs, bi, fm100, fm1000, erc, etr, pet, and vpd. Descriptions can be found here. Defaults to None, i.e., all the variables are downloaded.

• snow (bool, optional) – Compute snowfall from precipitation and minimum temperature. Defaults to False.

• snow_params (dict, optional) – Model-specific parameters as a dictionary that is passed to the snowfall function. These parameters are only used if snow is True. Two parameters are required: t_rain (deg C) which is the threshold for temperature for considering rain and t_snow (deg C) which is the threshold for temperature for considering snow. The default values are {'t_rain': 2.5, 't_snow': 0.6} that are adopted from https://doi.org/10.5194/gmd-11-1077-2018.

• ssl (bool, optional) – Whether to verify SSL certification, defaults to True.

Returns:

xarray.Dataset – Daily climate data within the target geometry.

Return type:

xarray.Dataset

Examples

>>> from shapely import Polygon
>>> import pygridmet as gridmet
>>> geometry = Polygon(
...     [[-69.77, 45.07], [-69.31, 45.07], [-69.31, 45.45], [-69.77, 45.45], [-69.77, 45.07]]
... )
>>> clm = gridmet.get_bygeom(geometry, 2010, variables="tmmn")
>>> clm["tmmn"].mean().item()
274.167

Package Contents

pynldas2

Top-level package.

Submodules

pynldas2.pynldas2

Get hourly NLDAS2 forcing data.

Module Contents

pynldas2.pynldas2.get_bycoords(coords, start_date, end_date, coords_id=None, crs=4326, variables=None, to_xarray=False, n_conn=4, snow=False, snow_params=None, source='grib')

Get NLDAS-2 climate forcing data for a list of coordinates.

Parameters:

• coords (list of tuples) – List of (lon, lat) coordinates.

• start_date (str) – Start date of the data.

• end_date (str) – End date of the data.

• crs (str, int, or pyproj.CRS, optional) – The CRS of the input coordinates, defaults to EPSG:4326.

• variables (str or list of str, optional) – Variables to download. If None, all variables are downloaded. Valid variables are: prcp, pet, temp, wind_u, wind_v, rlds, rsds, and humidity (and psurf if source=netcdf)

• to_xarray (bool, optional) – If True, the data is returned as an xarray dataset.

• n_conn (int, optional) – Number of parallel connections to use for retrieving data, defaults to 4. The maximum number of connections is 4, if more than 4 are requested, 4 connections will be used.

• snow (bool, optional) – Compute snowfall from precipitation and temperature. Defaults to False.

• snow_params (dict, optional) – Model-specific parameters as a dictionary that is passed to the snowfall function. These parameters are only used if snow is True. Two parameters are required: t_rain (deg C) which is the threshold for temperature for considering rain and t_snow (deg C) which is the threshold for temperature for considering snow. The default values are {'t_rain': 2.5, 't_snow': 0.6} that are adopted from https://doi.org/10.5194/gmd-11-1077-2018.

• source ({"grib", "netcdf"}, optional) – Source to pull data rods from. Valid sources are: grib and netcdf.

Returns:

pandas.DataFrame – The requested data as a dataframe.

Return type:

pandas.DataFrame | xarray.Dataset

pynldas2.pynldas2.get_bygeom(geometry, start_date, end_date, geo_crs, variables=None, n_conn=4, snow=False, snow_params=None, source='grib')

Get hourly NLDAS-2 climate forcing within a geometry at 0.125 resolution.

Parameters:

• geometry (shapely.Polygon, shapely.MultiPolygon, or tuple of length 4) – Input polygon or a bounding box like so (xmin, ymin, xmax, ymax).

• start_date (str) – Start date of the data.

• end_date (str) – End date of the data.

• geo_crs (int, str, or pyproj.CRS) – CRS of the input geometry

• variables (str or list of str, optional) – Variables to download. If None, all variables are downloaded. Valid variables are: prcp, pet, temp, wind_u, wind_v, rlds, rsds, and humidity (and psurf if source=netcdf)

• n_conn (int, optional) – Number of parallel connections to use for retrieving data, defaults to 4. It should be less than 4.

• snow (bool, optional) – Compute snowfall from precipitation and temperature. Defaults to False.

• snow_params (dict, optional) – Model-specific parameters as a dictionary that is passed to the snowfall function. These parameters are only used if snow is True. Two parameters are required: t_rain (deg C) which is the threshold for temperature for considering rain and t_snow (deg C) which is the threshold for temperature for considering snow. The default values are {'t_rain': 2.5, 't_snow': 0.6} that are adopted from https://doi.org/10.5194/gmd-11-1077-2018.

• source ({"grib", "netcdf"}, optional) – Source to pull data rods from. Valid sources are: grib and netcdf.

Returns:

xarray.Dataset – The requested forcing data.

Return type:

xarray.Dataset

pynldas2.pynldas2.get_grid_mask()

Get the NLDAS-2 grid that contains the land/water/soil/vegetation mask.

Returns:

xarray.Dataset – The grid mask.

Package Contents

hydrosignatures

Top-level package for HydroSignatures.

Submodules

hydrosignatures.hydrosignatures

Function for computing hydrologic signature.

Module Contents

class hydrosignatures.hydrosignatures.HydroSignatures

Hydrological signatures.

Parameters:

• q_mmpt (pandas.Series) – Discharge in mm per unit time (the same timescale as precipitation).

• p_mmpt (pandas.Series) – Precipitation in mm per unit time (the same timescale as discharge).

• si_method (str, optional) – Seasonality index method. Either walsh or markham. Default is walsh.

• fdc_slope_bins (tuple of int, optional) – The percentage bins between 1-100 to compute the slope of FDC within it, defaults to (33, 67).

• bfi_alpha (float, optional) – Alpha parameter for baseflow separation filter using Lyne and Hollick method. Default is 0.925.

property signature_names: dict[str, str]

Return a dictionary with the hydrological signatures.

property values: SignaturesFloat

Return a dictionary with the hydrological signatures.

bfi()

Compute Baseflow Index.

diff(other)

Compute absolute difference between two hydrological signatures.

fdc()

Compute exceedance probability (for flow duration curve).

fdc_slope()

Compute FDC slopes between a list of lower and upper percentiles.

isclose(other)

Check if the signatures are close between with a tolerance of 1e-3.

mean_annual_flood()

Compute mean annual flood.

mean_monthly()

Compute mean monthly flow (for regime curve).

runoff_ratio()

Compute total runoff ratio.

seasonality_index()

Compute seasonality index.

streamflow_elasticity()

Compute streamflow elasticity.

to_dict()

Return a dictionary with the hydrological signatures.

to_json()

Return a JSON string with the hydrological signatures.

hydrosignatures.hydrosignatures.aridity_index(pet: pandas.Series, prcp: pandas.Series) → numpy.float64

hydrosignatures.hydrosignatures.aridity_index(pet: pandas.DataFrame, prcp: pandas.DataFrame) → pandas.Series

hydrosignatures.hydrosignatures.aridity_index(pet: xarray.DataArray, prcp: xarray.DataArray) → xarray.DataArray

Compute (Budyko) aridity index (PET/Prcp).

Parameters:

• pet (pandas.DataFrame or pandas.Series or xarray.DataArray) – Potential evapotranspiration time series. Each column can correspond to PET a different location. Note that pet and prcp must have the same shape.

• prcp (pandas.DataFrame or pandas.Series or xarray.DataArray) – Precipitation time series. Each column can correspond to PET a different location. Note that pet and prcp must have the same shape.

Returns:

float or pandas.Series or xarray.DataArray – The aridity index.

hydrosignatures.hydrosignatures.baseflow(discharge, alpha=0.925, n_passes=3, pad_width=10)

Extract baseflow using the Lyne and Hollick filter (Ladson et al., 2013).

Parameters:

• discharge (numpy.ndarray or pandas.DataFrame or pandas.Series or xarray.DataArray) – Discharge time series that must not have any missing values. It can also be a 2D array where each row is a time series.

• n_passes (int, optional) – Number of filter passes, defaults to 3. It must be an odd number greater than 3.

• alpha (float, optional) – Filter parameter that must be between 0 and 1, defaults to 0.925.

• pad_width (int, optional) – Padding width for extending the data from both ends to address the warm up issue.

Returns:

numpy.ndarray or pandas.DataFrame or pandas.Series or xarray.DataArray – Same discharge input array-like but values replaced with computed baseflow values.

Return type:

ArrayVar

hydrosignatures.hydrosignatures.baseflow_index(discharge, alpha=0.925, n_passes=3, pad_width=10)

Compute the baseflow index using the Lyne and Hollick filter (Ladson et al., 2013).

Parameters:

• discharge (numpy.ndarray or pandas.DataFrame or pandas.Series or xarray.DataArray) – Discharge time series that must not have any missing values. It can also be a 2D array where each row is a time series.

• n_passes (int, optional) – Number of filter passes, defaults to 3. It must be an odd number greater than 3.

• alpha (float, optional) – Filter parameter that must be between 0 and 1, defaults to 0.925.

• pad_width (int, optional) – Padding width for extending the data from both ends to address the warm up issue.

Returns:

numpy.float64 – The baseflow index.

Return type:

numpy.float64

hydrosignatures.hydrosignatures.exceedance(daily, threshold=0.001)

Compute exceedance probability from daily data.

Parameters:

• daily (pandas.Series or pandas.DataFrame) – The data to be processed

• threshold (float, optional) – The threshold to compute exceedance probability, defaults to 1e-3.

Returns:

pandas.Series or pandas.DataFrame – Exceedance probability.

Return type:

pandas.DataFrame

hydrosignatures.hydrosignatures.extract_extrema(ts, var_name, n_pts)

Get local extrema in a time series.

Parameters:

• ts (pandas.Series) – Variable time series.

• var_name (str) – Variable name.

• n_pts (int) – Number of points to consider for detecting local extrema on both sides of each point.

Returns:

pandas.DataFrame – A dataframe with three columns: var_name, peak (bool) and trough (bool).

Return type:

pandas.DataFrame

hydrosignatures.hydrosignatures.flashiness_index(daily)

Compute flashiness index from daily data following Baker et al. (2004).

Parameters:

daily (pandas.Series or pandas.DataFrame or numpy.ndarray or xarray.DataArray) – The data to be processed

Returns:

numpy.ndarray – Flashiness index.

Return type:

FloatArray

References

Baker, D.B., Richards, R.P., Loftus, T.T. and Kramer, J.W., 2004. A new flashiness index: Characteristics and applications to midwestern rivers and streams 1. JAWRA Journal of the American Water Resources Association, 40(2), pp.503-522.

hydrosignatures.hydrosignatures.flood_moments(streamflow)

Compute flood moments (MAF, CV, CS) from streamflow.

Parameters:

streamflow (pandas.DataFrame) – The streamflow data to be processed

Returns:

pandas.DataFrame – Flood moments; mean annual flood (MAF), coefficient of variation (CV), and coefficient of skewness (CS).

Return type:

pandas.DataFrame

hydrosignatures.hydrosignatures.flow_duration_curve_slope(discharge, bins, log)

Compute FDC slopes between the given lower and upper percentiles.

Parameters:

• discharge (pandas.Series or pandas.DataFrame or numpy.ndarray or xarray.DataArray) – The discharge data to be processed.

• bins (tuple of int) – Percentile bins for computing FDC slopes between., e.g., (33, 67) returns the slope between the 33rd and 67th percentiles.

• log (bool) – Whether to use log-transformed data.

Returns:

numpy.ndarray – The slopes between the given percentiles.

Return type:

FloatArray

hydrosignatures.hydrosignatures.mean_monthly(daily, index_abbr=False, cms=False)

Compute mean monthly summary from daily data.

Parameters:

• daily (pandas.Series or pandas.DataFrame) – The data to be processed

• index_abbr (bool, optional) – Whether to use abbreviated month names as index instead of numbers, defaults to False.

• cms (bool, optional) – Whether the input data is in cubic meters per second (cms), defaults to False. If True, the mean monthly summary will be computed by taking the mean of the daily data, otherwise the sum of the daily data will be used.

Returns:

pandas.Series or pandas.DataFrame – Mean monthly summary.

Return type:

DF

hydrosignatures.hydrosignatures.rolling_mean_monthly(daily)

Compute rolling mean monthly.

hydrosignatures.hydrosignatures.seasonality_index_markham(data)

Compute seasonality index based on Markham, 1970.

hydrosignatures.hydrosignatures.seasonality_index_walsh(data)

Compute seasonality index based on Walsh and Lawler, 1981 method.

Package Contents

async_retriever

Top-level package.

Submodules

async_retriever.async_retriever

Core async functions.

Module Contents

async_retriever.async_retriever.delete_url_cache(url, request_method='GET', cache_name=None, **kwargs)

Delete cached response associated with url, along with its history (if applicable).

Parameters:

• url (str) – URL to be deleted from the cache

• request_method (str, optional) – HTTP request method to be deleted from the cache, defaults to GET.

• cache_name (str, optional) – Path to a file for caching the session, defaults to ./cache/aiohttp_cache.sqlite.

• kwargs (dict, optional) – Keywords to pass to the cache.delete_url().

async_retriever.async_retriever.retrieve(urls: Sequence[aiohttp.typedefs.StrOrURL], read_method: Literal[text], request_kwds: Sequence[dict[str, Any]] | None = ..., request_method: Literal[get, GET, post, POST] = ..., max_workers: int = ..., cache_name: pathlib.Path | str | None = ..., timeout: int = ..., expire_after: int = ..., ssl: ssl.SSLContext | bool | None = ..., disable: bool = ..., raise_status: bool = ...) → list[str]

async_retriever.async_retriever.retrieve(urls: Sequence[aiohttp.typedefs.StrOrURL], read_method: Literal[ujson], request_kwds: Sequence[dict[str, Any]] | None = ..., request_method: Literal[get, GET, post, POST] = ..., max_workers: int = ..., cache_name: pathlib.Path | str | None = ..., timeout: int = ..., expire_after: int = ..., ssl: ssl.SSLContext | bool | None = ..., disable: bool = ..., raise_status: bool = ...) → list[dict[str, Any]] | list[list[dict[str, Any]]]

async_retriever.async_retriever.retrieve(urls: Sequence[aiohttp.typedefs.StrOrURL], read_method: Literal[binary], request_kwds: Sequence[dict[str, Any]] | None = ..., request_method: Literal[get, GET, post, POST] = ..., max_workers: int = ..., cache_name: pathlib.Path | str | None = ..., timeout: int = ..., expire_after: int = ..., ssl: ssl.SSLContext | bool | None = ..., disable: bool = ..., raise_status: bool = ...) → list[bytes]

Send async requests.

Parameters:

• urls (list of str) – List of URLs.

• read_method (str) – Method for returning the request; binary, json, and text.

• request_kwds (list of dict, optional) – List of requests keywords corresponding to input URLs (1 on 1 mapping), defaults to None. For example, [{"params": {...}, "headers": {...}}, ...].

• request_method (str, optional) – Request type; GET (get) or POST (post). Defaults to GET.

• max_workers (int, optional) – Maximum number of async processes, defaults to 8.

• cache_name (str, optional) – Path to a file for caching the session, defaults to ./cache/aiohttp_cache.sqlite.

• timeout (int, optional) – Requests timeout in seconds, defaults to 5.

• expire_after (int, optional) – Expiration time for response caching in seconds, defaults to 2592000 (one week).

• ssl (bool or SSLContext, optional) – SSLContext to use for the connection, defaults to None. Set to False to disable SSL certification verification.

• disable (bool, optional) – If True temporarily disable caching requests and get new responses from the server, defaults to False.

• raise_status (bool, optional) – Raise an exception if the response status is not 200. If False return None. Defaults to True.

Returns:

list – List of responses in the order of input URLs.

Examples

>>> import async_retriever as ar
>>> stations = ["01646500", "08072300", "11073495"]
>>> url = "https://waterservices.usgs.gov/nwis/site"
>>> urls, kwds = zip(
...     *[
...         (url, {"params": {"format": "rdb", "sites": s, "siteStatus": "all"}})
...         for s in stations
...     ]
... )
>>> resp = ar.retrieve(urls, "text", request_kwds=kwds)
>>> resp[0].split("\n")[-2].split("\t")[1]
'01646500'

async_retriever.async_retriever.retrieve_binary(urls, request_kwds=None, request_method='GET', max_workers=8, cache_name=None, timeout=5, expire_after=EXPIRE_AFTER, ssl=None, disable=False, raise_status=True)

Send async requests and get the response as bytes.

Parameters:

• urls (list of str) – List of URLs.

• request_kwds (list of dict, optional) – List of requests keywords corresponding to input URLs (1 on 1 mapping), defaults to None. For example, [{"params": {...}, "headers": {...}}, ...].

• request_method (str, optional) – Request type; GET (get) or POST (post). Defaults to GET.

• max_workers (int, optional) – Maximum number of async processes, defaults to 8.

• cache_name (str, optional) – Path to a file for caching the session, defaults to ./cache/aiohttp_cache.sqlite.

• timeout (int, optional) – Requests timeout in seconds, defaults to 5.

• expire_after (int, optional) – Expiration time for response caching in seconds, defaults to 2592000 (one week).

• ssl (bool or SSLContext, optional) – SSLContext to use for the connection, defaults to None. Set to False to disable SSL certification verification.

• disable (bool, optional) – If True temporarily disable caching requests and get new responses from the server, defaults to False.

• raise_status (bool, optional) – Raise an exception if the response status is not 200. If False return None. Defaults to True.

Returns:

bytes – List of responses in the order of input URLs.

Return type:

list[bytes]

async_retriever.async_retriever.retrieve_json(urls, request_kwds=None, request_method='GET', max_workers=8, cache_name=None, timeout=5, expire_after=EXPIRE_AFTER, ssl=None, disable=False, raise_status=True)

Send async requests and get the response as json.

Parameters:

• urls (list of str) – List of URLs.

• request_kwds (list of dict, optional) – List of requests keywords corresponding to input URLs (1 on 1 mapping), defaults to None. For example, [{"params": {...}, "headers": {...}}, ...].

• request_method (str, optional) – Request type; GET (get) or POST (post). Defaults to GET.

• max_workers (int, optional) – Maximum number of async processes, defaults to 8.

• cache_name (str, optional) – Path to a file for caching the session, defaults to ./cache/aiohttp_cache.sqlite.

• timeout (int, optional) – Requests timeout in seconds, defaults to 5.

• expire_after (int, optional) – Expiration time for response caching in seconds, defaults to 2592000 (one week).

• ssl (bool or SSLContext, optional) – SSLContext to use for the connection, defaults to None. Set to False to disable SSL certification verification.

• disable (bool, optional) – If True temporarily disable caching requests and get new responses from the server, defaults to False.

• raise_status (bool, optional) – Raise an exception if the response status is not 200. If False return None. Defaults to True.

Returns:

dict – List of responses in the order of input URLs.

Return type:

list[dict[str, Any]] | list[list[dict[str, Any]]]

Examples

>>> import async_retriever as ar
>>> urls = ["https://labs.waterdata.usgs.gov/api/nldi/linked-data/comid/position"]
>>> kwds = [
...     {
...         "params": {
...             "f": "json",
...             "coords": "POINT(-68.325 45.0369)",
...         },
...     },
... ]
>>> r = ar.retrieve_json(urls, kwds)
>>> print(r[0]["features"][0]["properties"]["identifier"])
2675320

async_retriever.async_retriever.retrieve_text(urls, request_kwds=None, request_method='GET', max_workers=8, cache_name=None, timeout=5, expire_after=EXPIRE_AFTER, ssl=None, disable=False, raise_status=True)

Send async requests and get the response as text.

Parameters:

• urls (list of str) – List of URLs.

• request_kwds (list of dict, optional) – List of requests keywords corresponding to input URLs (1 on 1 mapping), defaults to None. For example, [{"params": {...}, "headers": {...}}, ...].

• request_method (str, optional) – Request type; GET (get) or POST (post). Defaults to GET.

• max_workers (int, optional) – Maximum number of async processes, defaults to 8.

• cache_name (str, optional) – Path to a file for caching the session, defaults to ./cache/aiohttp_cache.sqlite.

• timeout (int, optional) – Requests timeout in seconds in seconds, defaults to 5.

• expire_after (int, optional) – Expiration time for response caching in seconds, defaults to 2592000 (one week).

• ssl (bool or SSLContext, optional) – SSLContext to use for the connection, defaults to None. Set to False to disable SSL certification verification.

• disable (bool, optional) – If True temporarily disable caching requests and get new responses from the server, defaults to False.

• raise_status (bool, optional) – Raise an exception if the response status is not 200. If False return None. Defaults to True.

Returns:

list – List of responses in the order of input URLs.

Return type:

list[str]

Examples

>>> import async_retriever as ar
>>> stations = ["01646500", "08072300", "11073495"]
>>> url = "https://waterservices.usgs.gov/nwis/site"
>>> urls, kwds = zip(
...     *[
...         (url, {"params": {"format": "rdb", "sites": s, "siteStatus": "all"}})
...         for s in stations
...     ]
... )
>>> resp = ar.retrieve_text(urls, kwds)
>>> resp[0].split("\n")[-2].split("\t")[1]
'01646500'

async_retriever.async_retriever.stream_write(urls, file_paths, request_kwds=None, request_method='GET', max_workers=8, ssl=None, chunk_size=None)

Send async requests.

Parameters:

• urls (list of str) – List of URLs.

• file_paths (list of str or pathlib.Path) – List of file paths to write the response to.

• request_kwds (list of dict, optional) – List of requests keywords corresponding to input URLs (1 on 1 mapping), defaults to None. For example, [{"params": {...}, "headers": {...}}, ...].

• request_method (str, optional) – Request type; GET (get) or POST (post). Defaults to GET.

• max_workers (int, optional) – Maximum number of async processes, defaults to 8.

• ssl (bool or SSLContext, optional) – SSLContext to use for the connection, defaults to None. Set to False to disable SSL certification verification.

• chunk_size (int, optional) – The size of the chunks in bytes to be written to the file, defaults to None, which will iterates over data chunks and write them as received from the server.

Examples

>>> import async_retriever as ar
>>> import tempfile
>>> url = "https://freetestdata.com/wp-content/uploads/2021/09/Free_Test_Data_500KB_CSV-1.csv"
>>> with tempfile.NamedTemporaryFile() as temp:
...     ar.stream_write([url], [temp.name])

Package Contents

pygeoogc

Top-level package for PyGeoOGC.

Submodules

pygeoogc.cache_keys

Functions for creating unique keys based on web request parameters.

This module is based on the aiohttp-client-cache package, which is licensed under the MIT license. See the LICENSE file for more details.

Module Contents

pygeoogc.cache_keys.create_request_key(method, url, params=None, data=None, json=None)

Create a unique cache key based on request details.

pygeoogc.core

Base classes and function for REST, WMS, and WMF services.

Module Contents

class pygeoogc.core.ArcGISRESTfulBase(base_url, layer=None, outformat='geojson', outfields='*', crs=4326, max_workers=1, verbose=False, disable_retry=False)

Access to an ArcGIS REST service.

Parameters:

• base_url (str, optional) – The ArcGIS RESTful service url. The URL must either include a layer number after the last / in the url or the target layer must be passed as an argument.

• layer (int, optional) – Target layer number, defaults to None. If None layer number must be included as after the last / in base_url.

• outformat (str, optional) – One of the output formats offered by the selected layer. If not correct a list of available formats is shown, defaults to geojson. It defaults to esriSpatialRelIntersects.

• outfields (str or list) – The output fields to be requested. Setting * as outfields requests all the available fields which is the default setting.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference of the output data, defaults to epsg:4326

• max_workers (int, optional) – Max number of simultaneous requests, default to 2. Note that some services might face issues when several requests are sent simultaneously and will return the requests partially. It’s recommended to avoid using too many workers unless you are certain the web service can handle it.

• verbose (bool, optional) – If True, prints information about the requests and responses, defaults to False.

• disable_retry (bool, optional) – If True in case there are any failed queries, no retrying attempts is done and object IDs of the failed requests is saved to a text file which its ipath can be accessed via self.failed_path.

get_features(featureids, return_m=False, return_geom=True)

Get features based on the feature IDs.

Parameters:

• featureids (list) – List of feature IDs.

• return_m (bool, optional) – Whether to activate the Return M (measure) in the request, defaults to False.

• return_geom (bool, optional) – Whether to return the geometry of the feature, defaults to True.

Returns:

dict – (Geo)json response from the web service.

Return type:

list[dict[str, Any]]

get_response(url, payloads, method='GET')

Send payload and get the response.

initialize_service()

Initialize the RESTFul service.

partition_oids(oids)

Partition feature IDs based on self.max_nrecords.

retry_failed_requests()

Retry failed requests.

class pygeoogc.core.WFSBase(url, layer=None, outformat=None, version='2.0.0', crs=4326, read_method='json', max_nrecords=1000, validation=True)

Base class for WFS service.

Parameters:

• url (str) – The base url for the WFS service, for examples: https://hazards.fema.gov/nfhl/services/public/NFHL/MapServer/WFSServer

• layer (str) – The layer from the service to be downloaded, defaults to None which throws an error and includes all the available layers offered by the service.

• outformat (str) –

  The data format to request for data from the service, defaults to None which

  throws an error and includes all the available format offered by the service.

• version (str, optional) – The WFS service version which should be either 1.0.0, 1.1.0, or 2.0.0. Defaults to 2.0.0.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference system to be used for requesting the data, defaults to epsg:4326.

• read_method (str, optional) – Method for reading the retrieved data, defaults to json. Valid options are json, binary, and text.

• max_nrecords (int, optional) – The maximum number of records in a single request to be retrieved from the service, defaults to 1000. If the number of requested records is greater than this value, the query will be split into multiple requests.

• validation (bool, optional) – Validate the input arguments from the WFS service, defaults to True. Set this to False if you are sure all the WFS settings such as layer and crs are correct to avoid sending extra requests.

get_service_options()

Validate input arguments with the WFS service.

sort_params(sort_attr, nfeatures, start_index)

Get sort parameters for a WFS request.

validate_wfs()

Validate input arguments with the WFS service.

class pygeoogc.core.WMSBase(url, layers='', outformat='', version='1.3.0', crs=4326, validation=True)

Base class for accessing a WMS service.

Parameters:

• url (str) – The base url for the WMS service e.g., https://www.mrlc.gov/geoserver/mrlc_download/wms

• layers (str or list, optional) – A layer or a list of layers from the service to be downloaded. You can pass an empty string to get a list of available layers.

• outformat (str, optional) – The data format to request for data from the service. You can pass an empty string to get a list of available output formats.

• version (str, optional) – The WMS service version which should be either 1.1.1 or 1.3.0, defaults to 1.3.0.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference system to be used for requesting the data, defaults to epsg:4326.

• validation (bool, optional) – Validate the input arguments from the WMS service, defaults to True. Set this to False if you are sure all the WMS settings such as layer and crs are correct to avoid sending extra requests.

get_service_options()

Validate input arguments with the WMS service.

get_validlayers()

Get the layers supported by the WMS service.

validate_wms()

Validate input arguments with the WMS service.

pygeoogc.pygeoogc

Base classes and function for REST, WMS, and WMF services.

Module Contents

class pygeoogc.pygeoogc.ArcGISRESTful(base_url, layer=None, outformat='geojson', outfields='*', crs=4326, max_workers=1, verbose=False, disable_retry=False)

Access to an ArcGIS REST service.

Notes

By default, all retrieval methods retry to get the missing feature IDs, if there are any. You can disable this behavior by setting disable_retry to True. If there are any missing feature IDs after the retry, they are saved to a text file, ipath of which can be accessed by self.client.failed_path.

Parameters:

• base_url (str, optional) – The ArcGIS RESTful service url. The URL must either include a layer number after the last / in the url or the target layer must be passed as an argument.

• layer (int, optional) – Target layer number, defaults to None. If None layer number must be included as after the last / in base_url.

• outformat (str, optional) – One of the output formats offered by the selected layer. If not correct a list of available formats is shown, defaults to geojson.

• outfields (str or list) – The output fields to be requested. Setting * as outfields requests all the available fields which is the default behaviour.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference of the output data, defaults to epsg:4326.

• max_workers (int, optional) – Number of simultaneous download, default to 1, i.e., no threading. Note that some services might face issues when several requests are sent simultaneously and will return the requests partially. It’s recommended to avoid using too many workers unless you are certain the web service can handle it.

• verbose (bool, optional) – If True, prints information about the requests and responses, defaults to False.

• disable_retry (bool, optional) – If True in case there are any failed queries, no retrying attempts is done and object IDs of the failed requests is saved to a text file which its ipath can be accessed via self.client.failed_path.

get_features(featureids, return_m=False, return_geom=True)

Get features based on the feature IDs.

Parameters:

• featureids (list) – List of feature IDs.

• return_m (bool, optional) – Whether to activate the Return M (measure) in the request, defaults to False.

• return_geom (bool, optional) – Whether to return the geometry of the feature, defaults to True.

Returns:

dict – (Geo)json response from the web service.

Return type:

list[dict[str, Any]]

oids_byfield(field, ids)

Get Object IDs based on a list of field IDs.

Parameters:

• field (str) – Name of the target field that IDs belong to.

• ids (str or list) – A list of target ID(s).

Returns:

list of tuples – A list of feature IDs partitioned by self.max_nrecords.

Return type:

Iterator[tuple[str, Ellipsis]]

oids_bygeom(geom, geo_crs=4326, spatial_relation='esriSpatialRelIntersects', sql_clause=None, distance=None)

Get feature IDs within a geometry that can be combined with a SQL where clause.

Parameters:

• geom (LineString, Polygon, Point, MultiPoint, tuple, or list of tuples) – A geometry (LineString, Polygon, Point, MultiPoint), tuple of length two ((x, y)), a list of tuples of length 2 ([(x, y), ...]), or bounding box (tuple of length 4 ((xmin, ymin, xmax, ymax))).

• geo_crs (str, int, or pyproj.CRS, optional) – The spatial reference of the input geometry, defaults to epsg:4326.

• spatial_relation (str, optional) – The spatial relationship to be applied on the input geometry while performing the query. If not correct a list of available options is shown. It defaults to esriSpatialRelIntersects. Valid predicates are:

  • esriSpatialRelIntersects

  • esriSpatialRelContains

  • esriSpatialRelCrosses

  • esriSpatialRelEnvelopeIntersects

  • esriSpatialRelIndexIntersects

  • esriSpatialRelOverlaps

  • esriSpatialRelTouches

  • esriSpatialRelWithin

  • esriSpatialRelRelation

• sql_clause (str, optional) – Valid SQL 92 WHERE clause, default to None.

• distance (int, optional) – Buffer distance in meters for the input geometries, default to None.

Returns:

list of tuples – A list of feature IDs partitioned by self.max_nrecords.

Return type:

Iterator[tuple[str, Ellipsis]]

oids_bysql(sql_clause)

Get feature IDs using a valid SQL 92 WHERE clause.

Notes

Not all web services support this type of query. For more details look here.

Parameters:

sql_clause (str) – A valid SQL 92 WHERE clause.

Returns:

list of tuples – A list of feature IDs partitioned by self.max_nrecords.

Return type:

Iterator[tuple[str, Ellipsis]]

partition_oids(oids)

Partition feature IDs based on self.max_nrecords.

Parameters:

oids (list of int or int) – A list of feature ID(s).

Returns:

list of tuples – A list of feature IDs partitioned by self.max_nrecords.

Return type:

Iterator[tuple[str, Ellipsis]]

class pygeoogc.pygeoogc.HttpURLs

URLs of the supported HTTP services.

class pygeoogc.pygeoogc.RESTfulURLs

URLs of the supported RESTful services.

class pygeoogc.pygeoogc.ServiceURL

URLs of the supported services.

class pygeoogc.pygeoogc.WFS(url, layer=None, outformat=None, version='2.0.0', crs=4326, read_method='json', max_nrecords=1000, validation=True)

Data from any WFS service within a geometry or by featureid.

Parameters:

• url (str) – The base url for the WFS service, for examples: https://hazards.fema.gov/nfhl/services/public/NFHL/MapServer/WFSServer

• layer (str) – The layer from the service to be downloaded, defaults to None which throws an error and includes all the available layers offered by the service.

• outformat (str) –

  The data format to request for data from the service, defaults to None which

  throws an error and includes all the available format offered by the service.

• version (str, optional) – The WFS service version which should be either 1.0.0, 1.1.0, or 2.0.0. Defaults to 2.0.0.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference system to be used for requesting the data, defaults to epsg:4326.

• read_method (str, optional) – Method for reading the retrieved data, defaults to json. Valid options are json, binary, and text.

• max_nrecords (int, optional) – The maximum number of records in a single request to be retrieved from the service, defaults to 1000. If the number of records requested is greater than this value, it will be split into multiple requests.

• validation (bool, optional) – Validate the input arguments from the WFS service, defaults to True. Set this to False if you are sure all the WFS settings such as layer and crs are correct to avoid sending extra requests.

getfeature_bybox(bbox, box_crs=4326, always_xy=False, sort_attr=None)

Get data from a WFS service within a bounding box.

Parameters:

• bbox (tuple) – A bounding box for getting the data: [west, south, east, north]

• box_crs (str, or pyproj.CRS, optional) – The spatial reference system of the input bbox, defaults to epsg:4326.

• always_xy (bool, optional) – Whether to always use xy axis order, defaults to False. Some services change the axis order from xy to yx, following the latest WFS version specifications but some don’t. If the returned value does not have any geometry, it indicates that most probably the axis order does not match. You can set this to True in that case.

• sort_attr (str, optional) – The column name in the database to sort request by, defaults to the first attribute in the schema that contains id in its name.

Returns:

list of str or bytes or dict – WFS query response within a bounding box.

Return type:

RESPONSE

getfeature_byfilter(cql_filter, method='GET', sort_attr=None)

Get features based on a valid CQL filter.

Notes

The validity of the input CQL expression is user’s responsibility since the function does not perform any checks and just sends a request using the input filter.

Parameters:

• cql_filter (str) – A valid CQL filter expression.

• method (str) – The request method, could be GET or POST (for long filters).

• sort_attr (str, optional) – The column name in the database to sort request by, defaults to the first attribute in the schema that contains id in its name.

Returns:

str or bytes or dict – WFS query response

Return type:

RESPONSE

getfeature_bygeom(geometry, geo_crs=4326, always_xy=False, predicate='INTERSECTS', sort_attr=None)

Get features based on a geometry.

Parameters:

• geometry (shapely.Polygon or shapely.MultiPolygon) – The input geometry

• geo_crs (str, or pyproj.CRS, optional) – The CRS of the input geometry, default to epsg:4326.

• always_xy (bool, optional) – Whether to always use xy axis order, defaults to False. Some services change the axis order from xy to yx, following the latest WFS version specifications but some don’t. If the returned value does not have any geometry, it indicates that most probably the axis order does not match. You can set this to True in that case.

• predicate (str, optional) – The geometric predicate to use for requesting the data, defaults to INTERSECTS. Valid predicates are:

  • EQUALS

  • DISJOINT

  • INTERSECTS

  • TOUCHES

  • CROSSES

  • WITHIN

  • CONTAINS

  • OVERLAPS

  • RELATE

  • BEYOND

• sort_attr (str, optional) – The column name in the database to sort request by, defaults to the first attribute in the schema that contains id in its name.

Returns:

str or bytes or dict – WFS query response based on the given geometry.

Return type:

RESPONSE

getfeature_byid(featurename, featureids)

Get features based on feature IDs.

Parameters:

• featurename (str) – The name of the column for searching for feature IDs.

• featureids (int, str, or list of them) – The feature ID(s).

Returns:

str or bytes or dict – WMS query response.

Return type:

RESPONSE

class pygeoogc.pygeoogc.WFSURLs

URLs of the supported WFS services.

class pygeoogc.pygeoogc.WMS(url, layers, outformat, version='1.3.0', crs=4326, validation=True, ssl=True)

Get data from a WMS service within a geometry or bounding box.

Parameters:

• url (str) – The base url for the WMS service e.g., https://www.mrlc.gov/geoserver/mrlc_download/wms

• layers (str or list) – A layer or a list of layers from the service to be downloaded. You can pass an empty string to get a list of available layers.

• outformat (str) – The data format to request for data from the service. You can pass an empty string to get a list of available output formats.

• crs (str, int, or pyproj.CRS, optional) – The spatial reference system to be used for requesting the data, defaults to epsg:4326.

• version (str, optional) – The WMS service version which should be either 1.1.1 or 1.3.0, defaults to 1.3.0.

• validation (bool, optional) – Validate the input arguments from the WMS service, defaults to True. Set this to False if you are sure all the WMS settings such as layer and crs are correct to avoid sending extra requests.

• ssl (bool, optional) – Whether to use SSL for the connection, defaults to True.

get_validlayers()

Get the layers supported by the WMS service.

getmap_bybox(bbox: tuple[float, float, float, float], resolution: float, box_crs: CRSTYPE = ..., always_xy: bool = ..., max_px: int = ..., kwargs: dict[str, Any] | None = ..., tiff_dir: Literal[None] = None) → dict[str, bytes]

getmap_bybox(bbox: tuple[float, float, float, float], resolution: float, box_crs: CRSTYPE = ..., always_xy: bool = ..., max_px: int = ..., kwargs: dict[str, Any] | None = ..., tiff_dir: str | pathlib.Path = ...) → list[pathlib.Path]

Get data from a WMS service within a geometry or bounding box.

Parameters:

• bbox (tuple) – A bounding box for getting the data.

• resolution (float) – The output resolution in meters. The width and height of output are computed in pixel based on the geometry bounds and the given resolution.

• box_crs (str, int, or pyproj.CRS, optional) – The spatial reference system of the input bbox, defaults to epsg:4326.

• always_xy (bool, optional) – Whether to always use xy axis order, defaults to False. Some services change the axis order from xy to yx, following the latest WFS version specifications but some don’t. If the returned value does not have any geometry, it indicates that most probably the axis order does not match. You can set this to True in that case.

• max_px (int, optional) – The maximum allowable number of pixels (width x height) for a WMS requests, defaults to 8 million based on some trial-and-error.

• kwargs (dict, optional) – Optional additional keywords passed as payload, defaults to None. For example, {"styles": "default"}.

• tiff_dir (str or pathlib.Path, optional) – If given, the retrieved data will be stored on disk instead of returning it, defaults to None, i.e., saving to memory and returning the data.

Returns:

dict of bytes or list of pathlib.Path – If to_disk=False, a dict where the keys are the layer name and values are the returned response from the WMS service as bytes. If to_disk=True, a list of pathlib.Path objects to the saved files.

class pygeoogc.pygeoogc.WMSURLs

URLs of the supported WMS services.

pygeoogc.utils

Some utilities for PyGeoOGC.

Module Contents

class pygeoogc.utils.RetrySession(retries=3, backoff_factor=0.3, status_to_retry=(500, 502, 504), prefixes=('https://',), cache_name=None, expire_after=EXPIRE_AFTER, disable=False, ssl=True)

Configures the passed-in session to retry on failed requests.

Notes

The fails can be due to connection errors, specific HTTP response codes and 30X redirections. The code was originally based on: bustawin/retry-requests

Parameters:

• retries (int, optional) – The number of maximum retries before raising an exception, defaults to 5.

• backoff_factor (float, optional) – A factor used to compute the waiting time between retries, defaults to 0.5.

• status_to_retry (tuple, optional) – A tuple of status codes that trigger the reply behaviour, defaults to (500, 502, 504).

• prefixes (tuple, optional) – The prefixes to consider, defaults to (”http://”, “https://”)

• cache_name (str, optional) – Path to a folder for caching the session, default to None which uses system’s temp directory.

• expire_after (int, optional) – Expiration time for the cache in seconds, defaults to -1 (never expire).

• disable (bool, optional) – If True temporarily disable caching request/responses, defaults to False.

• ssl (bool, optional) – If True verify SSL certificates, defaults to True.

property disable: bool

Disable caching request/responses.

close()

Close the session.

get(url, payload=None, params=None, headers=None, stream=None)

Retrieve data from a url by GET and return the Response.

head(url, params=None, data=None, json=None, headers=None)

Retrieve data from a url by POST and return the Response.

post(url, payload=None, data=None, json=None, headers=None, stream=None)

Retrieve data from a url by POST and return the Response.

pygeoogc.utils.match_crs(geom, in_crs, out_crs)

Reproject a geometry to another CRS.

Parameters:

• geom (list or tuple or geometry) – Input geometry which could be a list of coordinates such as [(x1, y1), ...], a bounding box like so (xmin, ymin, xmax, ymax), or any valid shapely’s geometry such as Polygon, MultiPolygon, etc..

• in_crs (str, int, or pyproj.CRS) – Spatial reference of the input geometry

• out_crs (str, int, or pyproj.CRS) – Target spatial reference

Returns:

same type as the input geometry – Transformed geometry in the target CRS.

Return type:

GEOM

Examples

>>> from shapely import Point
>>> point = Point(-7766049.665, 5691929.739)
>>> match_crs(point, 3857, 4326).xy
(array('d', [-69.7636111130079]), array('d', [45.44549114818127]))
>>> bbox = (-7766049.665, 5691929.739, -7763049.665, 5696929.739)
>>> match_crs(bbox, 3857, 4326)
(-69.7636111130079, 45.44549114818127, -69.73666165448431, 45.47699468552394)
>>> coords = [(-7766049.665, 5691929.739)]
>>> match_crs(coords, 3857, 4326)
[(-69.7636111130079, 45.44549114818127)]

pygeoogc.utils.streaming_download(urls: str, kwds: dict[str, dict[Any, Any]] | None = None, fnames: str | pathlib.Path | None = None, root_dir: str | pathlib.Path | None = None, file_prefix: str = '', file_extention: str = '', method: str = 'GET', ssl: bool = True, chunk_size: int = CHUNK_SIZE, n_jobs: int = MAX_CONN) → pathlib.Path

pygeoogc.utils.streaming_download(urls: list[str], kwds: list[dict[str, dict[Any, Any]]] | None = None, fnames: Sequence[str | pathlib.Path] | None = None, root_dir: str | pathlib.Path | None = None, file_prefix: str = '', file_extention: str = '', method: str = 'GET', ssl: bool = True, chunk_size: int = CHUNK_SIZE, n_jobs: int = MAX_CONN) → list[pathlib.Path]

Download and store files in parallel from a list of URLs/Keywords.

Notes

This function runs asynchronously in parallel using n_jobs threads.

Parameters:

• urls (tuple or list) – A list of URLs to download.

• kwds (tuple or list, optional) – A list of keywords associated with each URL, e.g., ({“params”: …, “headers”: …}, …). Defaults to None.

• fnames (tuple or list, optional) – A list of filenames associated with each URL, e.g., (“file1.zip”, …). Defaults to None. If not provided, random unique filenames will be generated based on URL and keyword pairs.

• root_dir (str or Path, optional) – Root directory to store the files, defaults to None which uses HyRiver’s cache directory. Note that you should either provide root_dir or fnames. If both are provided, root_dir will be ignored.

• file_prefix (str, optional) – Prefix to add to filenames when storing the files, defaults to None, i.e., no prefix. This argument will be only be used if fnames is not passed.

• file_extention (str, optional) – Extension to use for storing the files, defaults to None, i.e., no extension if fnames is not provided otherwise. This argument will be only be used if fnames is not passed.

• method (str, optional) – HTTP method to use, i.e, GET or POST, by default “GET”.

• ssl (bool, optional) – Whether to use SSL verification, defaults to True.

• chunk_size (int, optional) – Chunk size to use when downloading, defaults to 100 * 1024 * 1024 i.e., 100 MB.

• n_jobs (int, optional) – The maximum number of concurrent downloads, defaults to 10.

Returns:

list – A list of pathlib.Path objects associated with URLs in the same order.

pygeoogc.utils.traverse_json(json_data, ipath)

Extract an element from a JSON-like object along a specified ipath.

This function is based on bcmullins.

Parameters:

• json_data (dict or list of dicts) – The input json dictionary.

• ipath (list) – The ipath to the requested element.

Returns:

list – The sub-items founds in the JSON.

Return type:

list[Any]

Examples

>>> data = [
...     {"employees": [
...         {"name": "Alice", "role": "dev", "nbr": 1},
...         {"name": "Bob", "role": "dev", "nbr": 2},
...         ],},
...     {"firm": {"name": "Charlie's Waffle Emporium", "location": "CA"}},
... ]
>>> traverse_json(data, ["employees", "name"])
[['Alice', 'Bob'], [None]]

pygeoogc.utils.validate_crs(crs)

Validate a CRS.

Parameters:

crs (str, int, or pyproj.CRS) – Input CRS.

Returns:

str – Validated CRS as a string.

Return type:

str

Package Contents

pygeoutils

Top-level package for PyGeoUtils.

Submodules

pygeoutils.geotools

Some utilities for manipulating GeoSpatial data.

Module Contents

class pygeoutils.geotools.Coordinates

Generate validated and normalized coordinates in WGS84.

Parameters:

• lon (float or list of floats) – Longitude(s) in decimal degrees.

• lat (float or list of floats) – Latitude(s) in decimal degrees.

• bounds (tuple of length 4, optional) – The bounding box to check of the input coordinates fall within. Defaults to WGS84 bounds.

Examples

>>> c = Coordinates([460, 20, -30], [80, 200, 10])
>>> c.points.x.tolist()
[100.0, -30.0]

property points: geopandas.GeoSeries

Get validate coordinate as a geopandas.GeoSeries.

class pygeoutils.geotools.GeoSpline(points, n_pts, degree=3, smoothing=None)

Create a parametric spline from a GeoDataFrame of points.

Parameters:

• points (geopandas.GeoDataFrame or geopandas.GeoSeries) – Input points as a GeoDataFrame or GeoSeries. The results will be more accurate if the CRS is projected.

• npts_sp (int) – Number of points in the output spline curve.

• degree (int, optional) – Degree of the smoothing spline. Must be 1 <= degree <= 5. Default to 3 which is a cubic spline.

• smoothing (float or None, optional) – Smoothing factor is used for determining the number of knots. This arg controls the tradeoff between closeness and smoothness of fit. Larger smoothing means more smoothing while smaller values of smoothing indicates less smoothing. If None (default), smoothing is done with all points.

Examples

>>> import geopandas as gpd
>>> xl, yl = zip(
...     *[
...         (-97.06138, 32.837),
...         (-97.06133, 32.836),
...         (-97.06124, 32.834),
...         (-97.06127, 32.832),
...     ]
... )
>>> pts = gpd.GeoSeries(gpd.points_from_xy(xl, yl, crs=4326))
>>> sp = GeoSpline(pts.to_crs(3857), 5).spline
>>> pts_sp = gpd.GeoSeries(gpd.points_from_xy(sp.x, sp.y, crs=3857))
>>> pts_sp = pts_sp.to_crs(4326)
>>> list(zip(pts_sp.x, pts_sp.y))
[(-97.06138, 32.837),
(-97.06132, 32.83575),
(-97.06126, 32.83450),
(-97.06123, 32.83325),
(-97.06127, 32.83200)]

property spline: Spline

Get the spline as a Spline object.

pygeoutils.geotools.break_lines(lines, points, tol=0.0)

Break lines at specified points at given direction.

Parameters:

• lines (geopandas.GeoDataFrame) – Lines to break at intersection points.

• points (geopandas.GeoDataFrame) – Points to break lines at. It must contain a column named direction with values up or down. This column is used to determine which part of the lines to keep, i.e., upstream or downstream of points.

• tol (float, optional) – Tolerance for snapping points to the nearest lines in meters. The default is 0.0.

Returns:

geopandas.GeoDataFrame – Original lines except for the parts that have been broken at the specified points.

Return type:

GDFTYPE

pygeoutils.geotools.coords_list(coords)

Convert a single coordinate or list of coordinates to a list of coordinates.

Parameters:

coords (tuple of list of tuple) – Input coordinates

Returns:

list of tuple – List of coordinates as [(x1, y1), ...].

Return type:

list[tuple[float, float]]

pygeoutils.geotools.geo2polygon(geometry, geo_crs=None, crs=None)

Convert a geometry to a Shapely’s Polygon and transform to any CRS.

Parameters:

• geometry (Polygon or tuple of length 4) – Polygon or bounding box (west, south, east, north).

• geo_crs (int, str, or pyproj.CRS, optional) – Spatial reference of the input geometry, defaults to None.

• crs (int, str, or pyproj.CRS) – Target spatial reference, defaults to None.

Returns:

shapely.Polygon or shapely.MultiPolygon – A (Multi)Polygon in the target CRS, if different from the input CRS.

Return type:

shapely.Polygon | shapely.MultiPolygon

pygeoutils.geotools.geometry_list(geometry)

Convert input geometry to a list of Polygons, Points, or LineStrings.

Parameters:

geometry (Polygon or MultiPolygon or tuple of length 4 or list of tuples of length 2 or 3) – Input geometry could be a (Multi)Polygon, (Multi)LineString, (Multi)Point, a tuple/list of length 4 (west, south, east, north), or a list of tuples of length 2 or 3.

Returns:

list – A list of Polygons, Points, or LineStrings.

Return type:

list[shapely.Polygon] | list[shapely.Point] | list[shapely.LineString]

pygeoutils.geotools.geometry_reproject(geom, in_crs, out_crs)

Reproject a geometry to another CRS.

Parameters:

• geom (list or tuple or any shapely.GeometryType) – Input geometry could be a list of coordinates such as [(x1, y1), ...], a bounding box like so (xmin, ymin, xmax, ymax), or any valid shapely’s geometry such as Polygon, MultiPolygon, etc..

• in_crs (str, int, or pyproj.CRS) – Spatial reference of the input geometry

• out_crs (str, int, or pyproj.CRS) – Target spatial reference

Returns:

same type as the input geometry – Transformed geometry in the target CRS.

Return type:

GEOM

Examples

>>> from shapely import Point
>>> point = Point(-7766049.665, 5691929.739)
>>> geometry_reproject(point, 3857, 4326).xy
(array('d', [-69.7636111130079]), array('d', [45.44549114818127]))
>>> bbox = (-7766049.665, 5691929.739, -7763049.665, 5696929.739)
>>> geometry_reproject(bbox, 3857, 4326)
(-69.7636111130079, 45.44549114818127, -69.73666165448431, 45.47699468552394)
>>> coords = [(-7766049.665, 5691929.739)]
>>> geometry_reproject(coords, 3857, 4326)
[(-69.7636111130079, 45.44549114818127)]

pygeoutils.geotools.line_curvature(line)

Compute the curvature of a Spline curve.

Notes

The formula for the curvature of a Spline curve is:

\[\kappa = \frac{\dot{x}\ddot{y} - \ddot{x}\dot{y}}{(\dot{x}^2 + \dot{y}^2)^{3/2}}\]

where \(\dot{x}\) and \(\dot{y}\) are the first derivatives of the Spline curve and \(\ddot{x}\) and \(\ddot{y}\) are the second derivatives of the Spline curve. Also, the radius of curvature is:

\[\rho = \frac{1}{|\kappa|}\]

Parameters:

line (shapely.LineString) – Line to compute the curvature at.

Returns:

• phi (numpy.ndarray) – Angle of the tangent of the Spline curve.

• curvature (numpy.ndarray) – Curvature of the Spline curve.

• radius (numpy.ndarray) – Radius of curvature of the Spline curve.

Return type:

tuple[FloatArray, FloatArray, FloatArray]

pygeoutils.geotools.make_spline(x, y, n_pts, k=3, s=None)

Create a parametric spline from a set of points.

Parameters:

• x (numpy.ndarray) – x-coordinates of the points.

• y (numpy.ndarray) – y-coordinates of the points.

• n_pts (int) – Number of points in the output spline curve.

• k (int, optional) – Degree of the smoothing spline. Must be 1 <= k <= 5. Default to 3 which is a cubic spline.

• s (float or None, optional) – Smoothing factor is used for determining the number of knots. This arg controls the tradeoff between closeness and smoothness of fit. Larger s means more smoothing while smaller values of s indicates less smoothing. If None (default), smoothing is done with all data points.

Returns:

Spline – A Spline object with x, y, phi, radius, distance, and line attributes. The line attribute returns the Spline as a shapely.LineString.

Return type:

Spline

pygeoutils.geotools.multi2poly(gdf)

Convert multipolygons to polygon and fill holes, if any.

Notes

This function tries to convert multipolygons to polygons by first checking if multiploygons can be directly converted using their exterior boundaries. If not, will try to remove very small sub-polygons that their area is less than 1% of the total area of the multipolygon. If this fails, the original multipolygon will be returned.

Parameters:

gdf (geopandas.GeoDataFrame or geopandas.GeoSeries) – A GeoDataFrame or GeoSeries with (multi)polygons. This will be more accurate if the CRS is projected.

Returns:

geopandas.GeoDataFrame or geopandas.GeoSeries – A GeoDataFrame or GeoSeries with polygons (and multipolygons).

Return type:

GDFTYPE

pygeoutils.geotools.nested_polygons(gdf)

Get nested polygons in a GeoDataFrame.

Parameters:

gdf (geopandas.GeoDataFrame or geopandas.GeoSeries) – A GeoDataFrame or GeoSeries with (multi)polygons.

Returns:

dict – A dictionary where keys are indices of larger polygons and values are a list of indices of smaller polygons that are contained within the larger polygons.

Return type:

dict[int | str, list[int | str]]

pygeoutils.geotools.query_indices(tree_gdf, input_gdf, predicate='intersects')

Find the indices of the input_geo that intersect with the tree_geo.

Parameters:

• tree_gdf (geopandas.GeoDataFrame or geopandas.GeoSeries) – The tree geodataframe.

• input_gdf (geopandas.GeoDataFrame or geopandas.GeoSeries) – The input geodataframe.

• predicate (str, optional) – The predicate to use for the query operation, defaults to intesects.

Returns:

dict – A dictionary of the indices of the input_gdf that intersect with the tree_gdf. Keys are the index of input_gdf and values are a list of indices of the intersecting tree_gdf.

Return type:

dict[Any, list[Any]]

pygeoutils.geotools.smooth_linestring(line, smoothing=None, npts=None)

Smooth a LineString using UnivariateSpline from scipy.

Parameters:

• line (shapely.LineString) – Centerline to be smoothed.

• smoothing (float or None, optional) – Smoothing factor is used for determining the number of knots. This arg controls the tradeoff between closeness and smoothness of fit. Larger smoothing means more smoothing while smaller values of smoothing indicates less smoothing. If None (default), smoothing is done with all points.

• npts (int, optional) – Number of points in the output smoothed line. Defaults to 5 times the number of points in the input line.

Returns:

shapely.LineString – Smoothed line with uniform spacing.

Return type:

shapely.LineString

Examples

>>> import geopandas as gpd
>>> import shapely
>>> line = shapely.LineString(
...     [
...         (-97.06138, 32.837),
...         (-97.06133, 32.836),
...         (-97.06124, 32.834),
...         (-97.06127, 32.832),
...     ]
... )
>>> line_smooth = smooth_linestring(line, 4326, 5)
>>> list(zip(*line_smooth.xy))
[(-97.06138, 32.837),
(-97.06132, 32.83575),
(-97.06126, 32.83450),
(-97.06123, 32.83325),
(-97.06127, 32.83200)]

pygeoutils.geotools.snap2nearest(lines, points, tol)

Find the nearest points on a line to a set of points.

Parameters:

• lines (geopandas.GeoDataFrame or geopandas.GeoSeries) – Lines.

• points (geopandas.GeoDataFrame or geopandas.GeoSeries) – Points to snap to lines.

• tol (float, optional) – Tolerance for snapping points to the nearest lines in meters. It must be greater than 0.0.

Returns:

geopandas.GeoDataFrame or geopandas.GeoSeries – Points snapped to lines.

Return type:

GDFTYPE

pygeoutils.geotools.spline_curvature(spline_x, spline_y, konts)

Compute the curvature of a Spline curve.

Notes

The formula for the curvature of a Spline curve is:

\[\kappa = \frac{\dot{x}\ddot{y} - \ddot{x}\dot{y}}{(\dot{x}^2 + \dot{y}^2)^{3/2}}\]

where \(\dot{x}\) and \(\dot{y}\) are the first derivatives of the Spline curve and \(\ddot{x}\) and \(\ddot{y}\) are the second derivatives of the Spline curve. Also, the radius of curvature is:

\[\rho = \frac{1}{|\kappa|}\]

Parameters:

• spline_x (scipy.interpolate.UnivariateSpline) – Spline curve for the x-coordinates of the points.

• spline_y (scipy.interpolate.UnivariateSpline) – Spline curve for the y-coordinates of the points.

• konts (numpy.ndarray) – Knots along the Spline curve to compute the curvature at. The knots must be strictly increasing.

Returns:

• phi (numpy.ndarray) – Angle of the tangent of the Spline curve.

• curvature (numpy.ndarray) – Curvature of the Spline curve.

• radius (numpy.ndarray) – Radius of curvature of the Spline curve.

Return type:

tuple[FloatArray, FloatArray, FloatArray]

pygeoutils.geotools.spline_linestring(line, crs, n_pts, degree=3, smoothing=None)

Generate a parametric spline from a LineString.

Parameters:

• line (shapely.LineString, shapely.MultiLineString) – Line to smooth. Note that if line is MultiLineString it will be merged into a single LineString. If the merge fails, an exception will be raised.

• crs (int, str, or pyproj.CRS) – CRS of the input line. It must be a projected CRS.

• n_pts (int) – Number of points in the output spline curve.

• degree (int, optional) – Degree of the smoothing spline. Must be 1 <= degree <= 5. Default to 3 which is a cubic spline.

• smoothing (float or None, optional) – Smoothing factor is used for determining the number of knots. This arg controls the tradeoff between closeness and smoothness of fit. Larger smoothing means more smoothing while smaller values of smoothing indicates less smoothing. If None (default), smoothing is done with all points.

Returns:

Spline – A Spline object with x, y, phi, radius, distance, and line attributes. The line attribute returns the Spline as a shapely.LineString.

Return type:

Spline

Examples

>>> import geopandas as gpd
>>> import shapely
>>> line = shapely.LineString(
...     [
...         (-97.06138, 32.837),
...         (-97.06133, 32.836),
...         (-97.06124, 32.834),
...         (-97.06127, 32.832),
...     ]
... )
>>> sp = spline_linestring(line, 4326, 5)
>>> list(zip(*sp.line.xy))
[(-97.06138, 32.837),
(-97.06132, 32.83575),
(-97.06126, 32.83450),
(-97.06123, 32.83325),
(-97.06127, 32.83200)]

pygeoutils.pygeoutils

Some utilities for manipulating GeoSpatial data.

Module Contents

pygeoutils.pygeoutils.arcgis2geojson(arcgis, id_attr=None)

Convert ESRIGeoJSON format to GeoJSON.

Notes

Based on arcgis2geojson.

Parameters:

• arcgis (str or binary) – The ESRIGeoJSON format str (or binary)

• id_attr (str, optional) – ID of the attribute of interest, defaults to None.

Returns:

dict – A GeoJSON file readable by GeoPandas.

Return type:

dict[str, Any]

pygeoutils.pygeoutils.geodf2xarray(geodf, resolution, attr_col=None, fill=0, projected_crs=5070)

Rasterize a geopandas.GeoDataFrame to xarray.DataArray.

Parameters:

• geodf (geopandas.GeoDataFrame or geopandas.GeoSeries) – GeoDataFrame or GeoSeries to rasterize.

• resolution (float) – Target resolution of the output raster in the projected_crs unit. Since the default projected_crs is EPSG:5070, the default unit for the resolution is meters.

• attr_col (str, optional) – Column name of the attribute to use as variable., defaults to None, i.e., the variable will be a boolean mask where 1 indicates the presence of a geometry. Also, note that the attribute must be numeric and have one of the following numpy types: int16, int32, uint8, uint16, uint32, float32, and float64.

• fill (int or float, optional) – Value to use for filling the missing values (mask) of the output raster, defaults to 0.

• projected_crs (int, str, or pyproj.CRS, optional) – A projected CRS to use for the output raster, defaults to EPSG:5070.

Returns:

xarray.Dataset – The xarray Dataset with a single variable.

Return type:

xarray.Dataset

pygeoutils.pygeoutils.gtiff2vrt(file_list, vrt_path)

Create a VRT file from a list of (Geo)Tiff files.

Note

This function requires gdal to be installed.

Parameters:

• file_list (list) – List of paths to the GeoTiff files.

• vrt_path (str or Path) – Path to the output VRT file.

pygeoutils.pygeoutils.gtiff2xarray(r_dict, geometry=None, geo_crs=None, ds_dims=None, driver=None, all_touched=False, nodata=None, drop=True)

Convert (Geo)Tiff byte responses to xarray.Dataset.

Parameters:

• r_dict (dict) – Dictionary of (Geo)Tiff byte responses where keys are some names that are used for naming each responses, and values are bytes.

• geometry (Polygon, MultiPolygon, or tuple, optional) – The geometry to mask the data that should be in the same CRS as the r_dict. Defaults to None.

• geo_crs (int, str, or pyproj.CRS, optional) – The spatial reference of the input geometry, defaults to None. This argument should be given when geometry is given.

• ds_dims (tuple of str, optional) – The names of the vertical and horizontal dimensions (in that order) of the target dataset, default to None. If None, dimension names are determined from a list of common names.

• driver (str, optional) – A GDAL driver for reading the content, defaults to automatic detection. A list of the drivers can be found here.

• all_touched (bool, optional) – Include a pixel in the mask if it touches any of the shapes. If False (default), include a pixel only if its center is within one of the shapes, or if it is selected by Bresenham’s line algorithm.

• nodata (float or int, optional) – The nodata value of the raster, defaults to None, i.e., it is determined from the raster.

• drop (bool, optional) – If True, drop the data outside of the extent of the mask geometries. Otherwise, it will return the same raster with the data masked. Default is True.

Returns:

xarray.Dataset or xarray.DataAraay – Requested dataset or dataarray.

Return type:

xarray.DataArray | xarray.Dataset

pygeoutils.pygeoutils.json2geodf(content, in_crs=4326, crs=4326)

Create GeoDataFrame from (Geo)JSON.

Parameters:

• content (dict or list of dict) – A (Geo)JSON dictionary e.g., response.json() or a list of them.

• in_crs (int, str, or pyproj.CRS, optional) – CRS of the content, defaults to epsg:4326.

• crs (int, str, or pyproj.CRS, optional) – The target CRS of the output GeoDataFrame, defaults to epsg:4326.

Returns:

geopandas.GeoDataFrame – Generated geo-data frame from a GeoJSON

Return type:

geopandas.GeoDataFrame

pygeoutils.pygeoutils.xarray2geodf(da, dtype, mask_da=None, connectivity=8)

Vectorize a xarray.DataArray to a geopandas.GeoDataFrame.

Parameters:

• da (xarray.DataArray) – The dataarray to vectorize.

• dtype (type) – The data type of the dataarray. Valid types are int16, int32, uint8, uint16, and float32.

• mask_da (xarray.DataArray, optional) – The dataarray to use as a mask, defaults to None.

• connectivity (int, optional) – Use 4 or 8 pixel connectivity for grouping pixels into features, defaults to 8.

Returns:

geopandas.GeoDataFrame – The vectorized dataarray.

Return type:

geopandas.GeoDataFrame

pygeoutils.pygeoutils.xarray_geomask(ds, geometry, crs, all_touched=False, drop=True, from_disk=False)

Mask a xarray.Dataset based on a geometry.

Parameters:

• ds (xarray.Dataset or xarray.DataArray) – The dataset(array) to be masked

• geometry (Polygon, MultiPolygon, or tuple of length 4) – The geometry to mask the data

• crs (int, str, or pyproj.CRS) – The spatial reference of the input geometry

• all_touched (bool, optional) – Include a pixel in the mask if it touches any of the shapes. If False (default), include a pixel only if its center is within one of the shapes, or if it is selected by Bresenham’s line algorithm.

• drop (bool, optional) – If True, drop the data outside of the extent of the mask geometries. Otherwise, it will return the same raster with the data masked. Default is True.

• from_disk (bool, optional) – If True, it will clip from disk using rasterio.mask.mask if possible. This is beneficial when the size of the data is larger than memory. Default is False.

Returns:

xarray.Dataset or xarray.DataArray – The input dataset with a mask applied (np.nan)

Return type:

XD

Package Contents

Release Notes

PyNHD

PyGeoHydro

Py3DEP

PyDaymet

PyGridMET

PyNLDAS2

HydroSignatures

AsyncRetriever

PyGeoOGC

PyGeoUtils

History

0.16.2 (2024-02-12)

Bug Fixes

• In NLDI.get_basins, the indices used to be station IDs but in the previous release they were reset by mistake. This version retains the correct indices.

New Features

• In nhdplus_l48 function, when the layer is NHDFlowline_Network or NHDFlowline_NonNetwork, merge all MultiLineString geometries to LineString.

0.16.1 (2024-01-03)

Bug Fixes

• Fix an issue in network_xsection and flowline_xsection related to the changes in shapely 2 API. Now, these functions should return the correct cross-sections.

0.16.0 (2024-01-03)

New Features

• Add access to USGS 3D Hydrography Program (3DHP) service. The new class is called HP3D. It can be queried by IDs, geometry, or SQL where clause.

• Add support for the new PyGeoAPI endpoints called xsatpathpts. This new endpoint is useful for getting elevation profile along A shapely.LineString. You can use pygeoapi function with service="elevation_profile" (or PyGeoAPI class) to access this new endpoint. Previously, the elevation_profile endpoint was used for getting elevation profile along a path from two endpoints and the input GeoDataFrame must have been a MultiPoint with two coordinates. Now, you must the input must contain LineString geometries.

• Switch to using the new smoothing algorithm from pygeoutils for resampling the flowlines and getting their cross-sections. This new algorithm is more robust, accurate, and faster. It has a new argument called smoothing for controlling the number knots of the spline. Higher values result in smoother curves. The default value is None which uses all the points from the input flowline.

0.15.2 (2023-09-22)

Bug Fixes

• Update GeoConnex based on the latest changes in the web service.

0.15.1 (2023-09-02)

Bug Fixes

• Fix HyRiver libraries requirements by specifying a range instead of exact version so conda-forge can resolve the dependencies.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

New Features

• Add a new function, called nhdplus_h12pp, for retrieving HUC12 pour points across CONUS.

• Add use_arrow=True to pynhd.nhdplus_l48 when reading the NHDPlus dataset. This speeds up the process since pyarrow is installed.

• In nhdplus_l48 make layer option so sql parameter of pyogrio.read_dataframe can also be used. This is necessary since pyogrio.read_dataframe does not support passing both layer and sql parameters.

• Update the mainstems dataset link to version 2.0 in mainstem_huc12_nx.

• Expose NHDTools class to the public API.

• For now, retain compatibility with shapely<2 while supporting shapley>=2.

Bug Fixes

• Remove unnecessary conversion of id_col and toid_col to Int64 in nhdflw2nx and vector_accumulation. This ensures that the input data types are preserved.

• Fix an issue in nhdplus_l48, where if the input data_dir is not absolute py7zr fails to extract the file.

0.14.0 (2023-03-05)

New Features

• Rewrite the GeoConnex class to provide access to new capabilities of the web service. Support for spatial queries have been added via CQL queries. For more information, check out the updated GeoConnex example notebook.

• Add a new property to StreamCat, called metrics_df that gets a dataframe of metric names and their description.

• Create a new private StreamCatValidator class to avoid polluting the public StreamCat class with private attributes and methods. Moreover, add a new alternative metric names attribute to StreamCat called alt_names for handling those metric names that do not follow METRIC+YYYY convention. This attribute is a dictionary that maps the alternative names to the actual metric names, so users can use METRIC_NAME column of metrics_df and add a year suffix from valid_years attribute of StreamCat to get the actual metric name.

• In navigate_by* functions of NLDI add stop_comid, which is another criterion for stopping the navigation in addition to distance.

• Improve UserWarning messages of NLDI and WaterData.

Breaking Changes

• Remove pynhd.geoconnex function since more functionality has been added to the GeoConnex service that existence of this function does not make sense anymore. All queries should be done via pynhd.GeoConnex class.

• Rewrite NLDI to improve code readability and significantly improving performance. Now, its methods do now return tuples if there are failed requests, instead they will be shown as a UserWarning.

• Bump the minimum required version of shapely to 2.0, and use its new API.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

0.13.12 (2023-02-10)

New Features

• Update the link to version 2.0 of the ENHD dataset in enhd_attrs.

Internal Changes

• Improve columns data types in enhd_attrs and nhdplus_vaa by using int32 instead of Int64, where applicable.

• Sync all patch versions of HyRiver packages to x.x.12.

0.13.11 (2023-01-24)

New Features

• The prepare_nhdplus now supports NHDPlus HR in addition to NHDPlus MR. It automatically detects the NHDPlus version based on the ID column name: nhdplusid for HR and comid for MR.

Internal Changes

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Improve performance of prepare_nhdplus by using pandas.merge instead of applying a function to each row of the dataframe.

0.13.10 (2023-01-08)

New Features

• Add support for the new EPA’s StreamCat Restful API with around 600 NHDPlus catchment level metrics. One class is added for getting the service properties such as valid metrics, called StreamCat. You can use streamcat function to get the metrics as a pandas.DataFrame.

• Refactor the show_versions function to improve performance and print the output in a nicer table-like format.

Internal Changes

• Skip 0.13.9 version so the minor version of all HyRiver packages become the same.

• Modify the codebase based on the latest changes in geopandas related to empty dataframes.

0.13.8 (2022-12-09)

New Features

• Add a new function, called nhdplus_attrs_s3, for accessing the recently released NHDPlus derived attributes on a USGS’s S3 bucket. The attributes are provided in parquet files, so getting them is faster than nhdplus_attrs. Also, you can request for multiple attributes at once whereas in nhdplus_attrs you had to request for each attribute one at a time. This function will replace nhdplus_attrs in a future release, as soon as all data that are available on the ScienceBase version are also accessible from the S3 bucket.

• Add two new functions called mainstem_huc12_nx and enhd_flowlines_nx. These functions generate a networkx directed graph object of NHD HUC12 water boundaries and flowlines, respectively. They also return a dictionary mapping of COMID and HUC12 to the corresponding networkx node. Additionally, a topologically sorted list of COMIDs/HUC12s are returned. The generated data are useful for doing US-scale network analysis and flow accumulation on the NHD network. The NHD graph has about 2.7 million edges and the mainstem HUC12 graph has about 80K edges.

• Add a new function for getting the entire NHDPlus dataset for CONUS (Lower 48), called nhdplus_l48. The entire NHDPlus dataset is downloaded from here. This 7.3 GB file will take a while to download, depending on your internet connection. The first time you run this function, the file will be downloaded and stored in the ./cache directory. Subsequent calls will use the cached file. Moreover, there are two additional dependencies for using this function: pyogrio and py7zr. These dependencies can be installed using pip install pyogrio py7zr or conda install -c conda-forge pyogrio py7zr.

Internal Changes

• Refactor vector_accumulation for significant performance improvements.

• Modify the codebase based on Refurb suggestions.

0.13.7 (2022-11-04)

New Features

• Add a new function called epa_nhd_catchments to access one of the EPA’s HMS endpoints called WSCatchment. You can use this function to access 414 catchment-scale characteristics for all the NHDPlus catchments including 16-day average curve number. More information on the curve number dataset can be found at its project page here.

Bug Fixes

• Fix a bug in NHDTools where due to the recent changes in pandas exception handling, the NHDTools fails in converting columns with NaN values to integer type. Now, pandas throws IntCastingNaNError instead of TypeError when using astype method on a column.

Internal Changes

• Use pyupgrade package to update the type hinting annotations to Python 3.10 style.

0.13.6 (2022-08-30)

Internal Changes

• Add the missing PyPi classifiers for the supported Python versions.

0.13.5 (2022-08-29)

Breaking Changes

• Append “Error” to all exception classes for conforming to PEP-8 naming conventions.

Internal Changes

• Bump the minimum versions of pygeoogc and pygeoutils to 0.13.5 and that of async-retriever to 0.3.5.

Bug Fixes

• Fix an issue in nhdplus_vaa and enhd_attrs functions where if cache folder does not exist, it would not have been created, thus resulting to an error.

0.13.3 (2022-07-31)

Internal Changes

• Use the new async_retriever.stream_write function to download files in nhdplus_vaa and enhd_attrs functions. This is more memory efficient.

• Convert the type of list of not found items in NLDI.comid_byloc and NLDI.feature_byloc to list of tuples of coordinates from list of strings. This matches the type of returned not found coordinates to that of the inputs.

• Fix an issue with NLDI that was caused by the recent changes in the NLDI web service’s error handling. The NLDI web service now returns more descriptive error messages in a json format instead of returning the usual status errors.

• Slice the ENHD dataframe in NHDTools.clean_flowlines before updating the flowline dataframe to reduce the required memory for the update operation.

0.13.2 (2022-06-14)

Breaking Changes

• Set the minimum supported version of Python to 3.8 since many of the dependencies such as xarray, pandas, rioxarray have dropped support for Python 3.7.

Internal Changes

• Use micromamba for running tests and use nox for linting in CI.

0.13.1 (2022-06-11)

New Features

• Add support for all the GeoConnex web service endpoints. There are two ways to use it. For a single query, you can use the geoconnex function and for multiple queries, it’s more efficient to use the GeoConnex class.

• Add support for passing any of the supported NLDI feature sources to the get_basins method of the NLDI class. The default is nwissite to retain backward compatibility.

Bug Fixes

• Set the type of “ReachCode” column to str instead of int in pygeoapi and nhdplus_vaa functions.

0.13.0 (2022-04-03)

New Features

• Add two new functions called flowline_resample and network_resample for resampling a flowline or network of flowlines based on a given spacing. This is useful for smoothing jagged flowlines similar to those in the NHDPlus database.

• Add support for the new NLDI endpoint called “hydrolocation”. The NLDI class now has two methods for getting features by coordinates: feature_byloc and comid_byloc. The feature_byloc method returns the flowline that is associated with the closest NHDPlus feature to the given coordinates. The comid_byloc method returns a point on the closest downstream flowline to the given coordinates.

• Add a new function called pygeoapi for calling the API in batch mode. This function accepts the input coordinates as a geopandas.GeoDataFrame. It is more performant than calling its counteract PyGeoAPI multiple times. It’s recommended to switch to using this new batch function instead of the PyGeoAPI class. Users just need to prepare an input data frame that has all the required service parameters as columns.

• Add a new step to prepare_nhdplus to convert MultiLineString to LineString.

• Add support for the simplified flag of NLDI’s get_basins function. The default value is True to retain the old behavior.

Breaking Changes

• Remove caching-related arguments from all functions since now they can be set globally via three environmental variables:

  • HYRIVER_CACHE_NAME: Path to the caching SQLite database.

  • HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds.

  • HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache file.

  You can do this like so:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/file.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"

0.12.2 (2022-02-04)

New Features

• Add a new class called NHD for accessing the latest National Hydrography Dataset. More info regarding this data can be found here.

• Add two new functions for getting cross-sections along a single flowline via flowline_xsection or throughout a network of flowlines via network_xsection. You can specify spacing and width parameters to control their location. For more information and examples please consult the documentation.

• Add a new property to AGRBase called service_info to include some useful info about the service including feature_types which can be handy for converting numeric values of types to their string equivalent.

Internal Changes

• Use the new PyGeoAPI API.

• Refactor prepare_nhdplus for improving the performance and robustness of determining tocomid within a network of NHD flowlines.

• Add empty geometries that NLDI.getbasins returns to the list of not found IDs. This is because the NLDI service does not include non-network flowlines and instead returns an empty geometry for these flowlines. (GH #48)

0.12.1 (2021-12-31)

Internal Changes

• Use the three new ar.retrieve_* functions instead of the old ar.retrieve function to improve type hinting and to make the API more consistent.

• Revert to the original PyGeoAPI base URL.

0.12.0 (2021-12-27)

Breaking Changes

• Rewrite ScienceBase to make it applicable for working with other ScienceBase items. A new function has been added for staging the Additional NHDPlus attributes items called stage_nhdplus_attrs.

• Refactor AGRBase to remove unnecessary functions and make them more general.

• Update PyGeoAPI class to conform to the new pygeoapi API. This web service is undergoing some changes at the time of this release and the API is not stable, might not work as expected. As soon as the web service is stable, a new version will be released.

New Features

• In WaterData.byid show a warning if there are any missing feature IDs that are requested but are not available in the dataset.

• For all by* methods of WaterData throw a ZeroMatched exception if no features are found.

• Add expire_after and disable_caching arguments to all functions that use async_retriever. Set the default request caching expiration time to never expire. You can use disable_caching if you don’t want to use the cached responses. Please refer to documentation of the functions for more details.

Internal Changes

• Refactor prepare_nhdplus to reduce code complexity by grouping all the NHDPlus tools as a private class.

• Modify AGRBase to reflect the latest API changes in pygeoogc.ArcGISRESTfull class.

• Refactor prepare_nhdplus by creating a private class that includes all the previously used private functions. This will make the code more readable and easier to maintain.

• Add all the missing types so mypy --strict passes.

0.11.4 (2021-11-12)

New Features

• Add a new argument to NLDI.get_basins called split_catchment that if is set to True will split the basin geometry at the watershed outlet.

Internal Changes

• Catch service errors in PyGeoAPI and show useful error messages.

• Use importlib-metadata for getting the version instead of pkg_resources to decrease import time as discussed in this issue.

0.11.3 (2021-09-10)

Internal Changes

• More robust handling of inputs and outputs of NLDI’s methods.

• Use an alternative download link for NHDPlus VAA file on Hydroshare.

• Restructure the codebase to reduce the complexity of pynhd.py file by dividing it into three files: pynhd all classes that provide access to the supported web services, core that includes base classes, and nhdplus_derived that has functions for getting databases that provided additional attributes for the NHDPlus database.

0.11.2 (2021-08-26)

New Features

• Add support for PyGeoAPI. It offers four functionalities: flow_trace, split_catchment, elevation_profile, and cross_section.

0.11.1 (2021-07-31)

New Features

• Add a function for getting all NHD FCodes as a data frame, called nhd_fcode.

• Improve prepare_nhdplus function by removing all coastlines and better detection of the terminal point in a network.

Internal Changes

• Migrate to using AsyncRetriever for handling communications with web services.

• Catch the ConnectionError separately in NLDI and raise a ServiceError instead. So user knows that data cannot be returned due to the out of service status of the server not ZeroMatched.

0.11.0 (2021-06-19)

New Features

• Add nhdplus_vaa to access NHDPlus Value Added Attributes for all its flowlines.

• To see a list of available layers in NHDPlus HR, you can instantiate its class without passing any argument like so NHDPlusHR().

Breaking Changes

• Drop support for Python 3.6 since many of the dependencies such as xarray and pandas have done so.

Internal Changes

• Use persistent caching for all requests which can help speed up network responses significantly.

• Improve documentation and testing.

0.10.1 (2021-03-27)

• Add an announcement regarding the new name for the software stack, HyRiver.

• Improve pip installation and release workflow.

0.10.0 (2021-03-06)

• The first release after renaming hydrodata to PyGeoHydro.

• Make mypy checks more strict and fix all the errors and prevent possible bugs.

• Speed up CI testing by using mamba and caching.

0.9.0 (2021-02-14)

• Bump version to the same version as PyGeoHydro.

Breaking Changes

• Add a new function for getting basins geometries for a list of USGS station IDs. The function is a method of NLDI class called get_basins. So, now NLDI.getfeature_byid function does not have a basin flag. This change makes getting geometries easier and faster.

• Remove characteristics_dataframe method from NLDI and make a standalone function called nhdplus_attrs for accessing NHDPlus attributes directly from ScienceBase.

• Add support for using hydro or edits webs services for getting NHDPlus High-Resolution using NHDPlusHR function. The new arguments are service which accepts hydro or edits, and autos_switch flag for automatically switching to the other service if the ones passed by service fails.

New Features

• Add a new argument to topoogical_sort called edge_attr that allows adding attribute(s) to the returned Networkx Graph. By default, it is None.

• A new base class, AGRBase for connecting to ArcGISRESTful-based services such as National Map and EPA’s WaterGEOS.

• Add support for setting the buffer distance for the input geometries to AGRBase.bygeom.

• Add comid_byloc to NLDI class for getting ComIDs of the closest flowlines from a list of lon/lat coordinates.

• Add bydistance to WaterData for getting features within a given radius of a point.

0.2.0 (2020-12-06)

Breaking Changes

• Re-wrote the NLDI function to use API v3 of the NLDI service.

• The crs argument of WaterData now is the target CRS of the output dataframe. The service CRS is now EPSG:4269 for all the layers.

• Remove the url_only argument of NLDI since it’s not applicable anymore.

New Features

• Added support for NHDPlus High Resolution for getting features by geometry, IDs, or SQL where clause.

• The following functions are added to NLDI:

• getcharacteristic_byid: Getting characteristics of NHDPlus catchments.

• navigate_byloc: Getting the nearest ComID to a coordinate and performing navigation.

• characteristics_dataframe: Getting all the available catchment-scale characteristics as a data frame.

• get_validchars: Getting a list of available characteristic IDs for a specified characteristic type.

• The following function is added to WaterData:

• byfilter: Getting data based on any valid CQL filter.

• bygeom: Getting data within a geometry (polygon and multipolygon).

• Add support for Python 3.9 and tests for Windows.

Bug Fixes

• Refactored WaterData to fix the CRS inconsistencies (#1).

0.1.3 (2020-08-18)

• Replaced simplejson with orjson to speed-up JSON operations.

0.1.2 (2020-08-11)

• Add show_versions function for showing versions of the installed deps.

• Improve documentation

0.1.1 (2020-08-03)

• Improved documentation

• Refactored WaterData to improve readability.

0.1.0 (2020-07-23)

• First release on PyPI.

History

0.16.2 (2024-XX-XX)

Bug Fixes

• In nlcd_helper function the roughness value for class 82 was set to 0.16 instead of 0.037.

New Features

• Converted all methods of NWIS class to classmethod so the class can be used without instantiating it. This change makes the class more flexible and easier to use.

• In NID class, the stage_nid_inventory method now checks if the remote NID database has been modified since the last download and only downloads the new data if it has been modified. This change makes the method more efficient and reduces the network traffic while ensuring that the local database is always up-to-date.

0.16.0 (2024-01-03)

Breaking Changes

• Bump the minimum supported version of shapely to 2.

Internal Changes

• Update the link to NWIS error codes tables in the nwis_errors function.

• Update NWIS class based on the latest changes to the NWIS web service.

• Use the default tiles for the interactive_map function.

0.15.2 (2023-09-22)

New Features

• Add a new attribute to EHydro class called survey_grid. It’s a geopandas.GeoDataFrame that includes the survey grid of the eHydro dataset which is a 35-km hexagonal grid.

• Add support for getting point cloud and survey outline data from eHydro. You can set data_type in EHydro to bathymetry, points, outlines, or contours to get the corresponding data. The default is points since this is the recommended data type by USACE.

• Add NFHL class within nfhl module to access FEMA’s National Flood Hazard Layer (NFHL) using six different ArcGISRESTFul services. Contributed by Fernando Aristizabal. (PR 108)

Internal Changes

• Remove dependency on dask.

• Move all NLCD related functions to a separate module called nlcd. This doesn’t affect the API since the functions are still available under pygeohydro namespace.

0.15.1 (2023-08-02)

This release provides access to three new datasets:

• USACE Hydrographic Surveys (eHydro) and

• USGS Short-Term Network (STN) Flood Event Data, contributed by Fernando Aristizabal. (PR 108)

• NLCD 2021

New Features

• Add support for getting topobathymetry data from USACE Hydrographic Surveys (eHydro). The new class is called EHydro and gives users the ability to subset the eHydro dataset by geometry, ID, or SQL queries.

• Add new stnfloodevents module with STNFloodEventData class for retrieving flood event data from the USGS Short-Term Network (STN) RESTful Service. This Python API abstracts away RESTful principles and produces analysis ready data in geo-referenced GeoDataFrames, DataFrames, lists, or dictionaries as desired. The core class methods available are data_dictionary, get_all_data, and get_filtered_data. These class methods retrieve the data dictionaries by type, get all the available data by type, and make filtered requests for data by type as well, respectively. The four types of data include instruments, peaks, hwms, and sites. Contributed by Fernando Aristizabal.

• Add a wrapper function for the STNFloodEventData class called stn_flood_event.

• Add support for the new NLCD data (2021) for the three supported layers.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

New Features

• Add a new option to NWIS.get_info, called nhd_info, for retrieving NHDPlus related info on the sites. This will two new service calls that might slow down the function, so it’s disabled by default.

• Update links in NID to the latest CSV and GPKG versions of the NID dataset.

• Add two new properties to NID to access the entire NID dataset. You can use NID.df to access the CSV version as a pandas.DataFrame and NID.gdf to access the GPKG version as a geopandas.GeoDataFrame. Installing pyogrio is highly recommended for much faster reading of the GPKG version.

• Refactor NID.bygeom to use the new NID.gdf property for spatial querying of the dataset. This change should make the query much faster.

• For now, retain compatibility with shapely<2 while supporting shapley>=2.

0.14.0 (2023-03-05)

New Features

• Add a new function, called nlcd_area_percent, for computing the percentages or natural, developed, and impervious areas within geometries of a given GeoDataFrame. This function uses imperviousness and land use/land cover data from NLCD to compute the area percentages of the natural, developed, and impervious areas. For more information please refer to the function’s documentation.

• Add a new column to the dataframe returned by NWIS.get_info, called nhd_comid, and rename drain_sqkm to nhd_areasqkm. The new drainage area is the best available estimates of stations’ drainage area that have been extracted from the NHDPlus. The new nhd_comid column makes it easier to link stations to NHDPlus.

• In get_camels, return qobs with negatives values set to NaN. Also, Add a new variable called Newman_2017 to both datasets for identifying the 531 stations that were used in Newman et al. (2017).

• Add a new function, called streamflow_fillna, for filling missing streamflow values (NAN) with day-of-year average values.

Breaking Changes

• Bump the minimum required version of shapely to 2.0, and use its new API.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

• Improve performance of all NLCD functions by merging two methods of the NLCD and also reducing the memory footprint of the functions.

0.13.12 (2023-02-10)

New Features

• Add initial support for SensorThings API Currently, the SensorThings class only supports Things endpoint. Users need to provide a valid Odata filter. The class has a odata_helper function that can be used to generate and validate Odata filters. Additionally, using sensor_info and sensor_property functions users can request for information about sensors themselves or their properties.

Internal Changes

• Simplify geometry validation by using pygeoutils.geo2polygon function in ssebopeta_bygeom.

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Sync all patch versions of HyRiver packages to x.x.12.

0.13.10 (2023-01-09)

Breaking Changes

• The NID service has changed some of its endpoints to use Federal ID instead of Dam ID. This change affects the NID.inventory_byid function. This function now accepts Federal IDs instead of dam IDs.

New Features

• Refactor the show_versions function to improve performance and print the output in a nicer table-like format.

Internal Changes

• Use the new pygeoogc.streaming_download function in huc_wb_full to improve performance and reduce code complexity.

• Skip 0.13.9 version so the minor version of all HyRiver packages become the same.

• Modify the codebase based on the latest changes in geopandas related to empty dataframes.

• Use pyright for static type checking instead of mypy and address all typing issues that it raised.

0.13.8 (2022-12-09)

New Features

• Add a function called huc_wb_full that returns the full watershed boundary GeoDataFrame of a given HUC level. If only a subset of HUCs is needed the pygeohydro.WBD class should be used. The full dataset is downloaded from the National Maps’ WBD staged products.

• Add a new function called irrigation_withdrawals for retrieving estimated monthly water use for irrigation by 12-digit hydrologic unit in the CONUS for 2015 from ScienceBase.

• Add a new property to NID, called data_units for indicating the units of NID dataset variables.

• The get_us_states now accepts conus as a subset_key which is equivalent to contiguous.

Internal Changes

• Add get_us_states to __init__ file, so it can be loaded directly, e.g., gh.get_us_states("TX").

• Modify the codebase based on Refurb suggestions.

• Significant performance improvements in NWIS.get_streamflow especially for large requests by refactoring the timezone handling.

Bug Fixes

• Fix the dam types and purposes mapping dictionaries in NID class.

0.13.7 (2022-11-04)

New Features

• Add a two new function for retrieving soil properties across the US:

  • soil_properties: Porosity, available water capacity, and field capacity,

  • soil_gnatsgo: Soil properties from the gNATSGO database.

• Add a new help function called state_lookup_table for getting a lookup table of US states and their counties. This can be particularly useful for mapping the digit state_cd and county_cd that NWIS returns to state names/codes.

• Add support for getting individual state geometries using get_us_states function by passing their two letter state code. Also, use TIGER 2022 data for the US states and counties instead of TIGER 2021.

Internal Changes

• Remove proplot as a dependency and use matplotlib instead.

0.13.6 (2022-08-30)

Internal Changes

• Add the missing PyPi classifiers for the supported Python versions.

0.13.5 (2022-08-29)

Breaking Changes

• Append “Error” to all exception classes for conforming to PEP-8 naming conventions.

• Deprecate ssebopeta_byloc since it’s been replaced with ssebopeta_bycoords since version 0.13.0.

Internal Changes

• Bump the minimum versions of pygeoogc and pygeoutils to 0.13.5 and that of async-retriever to 0.3.5.

0.13.3 (2022-07-31)

New Features

• Add a new argument to NID.inventory_byid class for staging the entire NID dataset prior to inventory queries. There a new public method called NID.stage_nid_inventory that can be used to download the entire NID dataset and save it as a feather file. This is useful inventory queries with large number of IDs and is much more efficient than querying the NID web service.

Bug Fixes

• The background value in cover_statistics function should have been 127 not 0. Also, dropped the background value from the return statistics.

0.13.2 (2022-06-14)

Breaking Changes

• Set the minimum supported version of Python to 3.8 since many of the dependencies such as xarray, pandas, rioxarray have dropped support for Python 3.7.

Internal Changes

• Remove USGS prefixes from the input station IDs in NWIS.get_streamflow function. Also, check if the remaining parts of the IDs are all digits and throw an exception if otherwise. Additionally, make sure that IDs have at least 8 chars by adding leading zeros (GH 99).

• Use micromamba for running tests and use nox for linting in CI.

0.13.1 (2022-06-11)

New Features

• Add a new function called get_us_states to the helpers module for obtaining a GeoDataFrame of the US states. It has an optional argument for returning the contiguous states, continental states, commonwealths states, or US territories. The data are retrieved from the Census’ Tiger 2021 database.

• In the NID class keep the valid_fields property as a pandas.Series instead of a list, so it can be searched easier via its str accessor.

Internal Changes

• Refactor the plot.signatures function to use proplot instead of matplotlib.

• Improve performance of NWIS.get_streamflow by not validating the layer name when instantiating the WaterData class. Also, make the function more robust by checking if streamflow data is available for each station and throw a warning if not.

Bug Fixes

• Fix an issue in NWIS.get_streamflow where -9999 values were not being filtered out. According to NWIS, these values are reserved for ice-affected data. This fix sets these values to numpy.nan.

0.13.0 (2022-04-03)

New Features

• Add a new flag to nlcd_* functions called ssl for disabling SSL verification.

• Add a new function called get_camels for getting the CAMELS dataset. The function returns a geopandas.GeoDataFrame that includes basin-level attributes for all 671 stations in the dataset and a xarray.Dataset that contains streamflow data for all 671 stations and their basin-level attributes.

• Add a new function named overland_roughness for getting the overland roughness values from land cover data.

• Add a new class called WBD for getting watershed boundary (HUC) data.

from pygeohydro import WBD

wbd = WBD("huc4")
hudson = wbd.byids("huc4", ["0202", "0203"])

Breaking Changes

• Remove caching-related arguments from all functions since now they can be set globally via three environmental variables:

  • HYRIVER_CACHE_NAME: Path to the caching SQLite database.

  • HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds.

  • HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache file.

  You can do this like so:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/file.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"

Internal Changes

• Write nodata attribute using rioxarray in nlcd_bygeom since the clipping operation of rioxarray uses this value as the fill value.

0.12.4 (2022-02-04)

Internal Changes

• Return a named tuple instead of a dict of percentages in the cover_statistics function. It makes accessing the values easier.

• Add pycln as a new pre-commit hooks for removing unused imports.

• Remove time zone info from the inputs to plot.signatures to avoid issues with the matplotlib backend.

Bug Fixes

• Fix an issue in plot.signatures where the new matplotlib version requires a numpy array instead of a pandas.DataFrame.

0.12.3 (2022-01-15)

Bug Fixes

• Replace no data values of data in ssebopeta_bygeom with np.nan before converting it to mm/day.

• Fix an inconsistency issue with CRS projection when using UTM in nlcd_*. Use EPSG:3857 for all reprojections and get the data from NLCD in the same projection. (GH 85)

• Improve performance of nlcd_* functions by reducing number of service calls.

Internal Changes

• Add type checking with typeguard and fix type hinting issues raised by typeguard.

• Refactor show_versions to ensure getting correct versions of all dependencies.

0.12.2 (2021-12-31)

New Features

• The NWIS.get_info now returns a geopandas.GeoDataFrame instead of a pandas.DataFrame.

Bug Fixes

• Fix a bug in NWIS.get_streamflow where the drainage area might not be computed correctly if target stations are not located at the outlet of their watersheds.

0.12.1 (2021-12-31)

Internal Changes

• Use the three new ar.retrieve_* functions instead of the old ar.retrieve function to improve type hinting and to make the API more consistent.

Bug Fixes

• Fix an in issue with NWIS.get_streamflow where time zone of the data was not being correctly determined when it was US specific abbreviations such as CST.

0.12.0 (2021-12-27)

New Features

• Add support for getting instantaneous streamflow from NWIS in addition to the daily streamflow by adding freq argument to NWIS.get_streamflow that can be either iv or dv. The default is dv to retain the previous behavior of the function.

• Convert the time zone of the streamflow data to UTC.

• Add attributes of the requested stations as attrs parameter to the returned pandas.DataFrame. (GH 75)

• Add a new flag to NWIS.get_streamflow for returning the streamflow as xarray.Dataset. This dataset has two dimensions; time and station_id. It has ten variables which includes discharge and nine other station attributes. (GH 75)

• Add drain_sqkm from GagesII to NWIS.get_info.

• Show drain_sqkm in the interactive map generated by interactive_map.

• Add two new functions for getting NLCD data; nlcd_bygeom and nlcd_bycoords. The new nlcd_bycoords function returns a geopandas.GeoDataFrame with the NLCD layers as columns and input coordinates, which should be a list of (lon, lat) tuples, as the geometry column. Moreover, The new nlcd_bygeom function now accepts a geopandas.GeoDataFrame as the input. In this case, it returns a dict with keys as indices of the input geopandas.GeoDataFrame. (GH 80)

• The previous nlcd function is being deprecated. For now, it calls nlcd_bygeom internally and retains the old behavior. This function will be removed in future versions.

Breaking Changes

• The ssebop_byloc is being deprecated and replaced by ssebop_bycoords. The new function accepts a pandas.DataFrame as input that should include three columns: id, x, and y. It returns a xarray.Dataset with two dimensions: time and location_id. The id columns from the input is used as the location_id dimension. The ssebop_byloc function still retains the old behavior and will be removed in future versions.

• Set the request caching’s expiration time to never expire. Add two flags to all functions to control the caching: expire_after and disable_caching.

• Replace NID class with the new RESTful-based web service of National Inventory of Dams. The new NID service is very different from the old one, so this is considered a breaking change.

Internal Changes

• Improve exception handling in NWIS.get_info when NWIS returns an error message rather than 500s web service error.

• The NWIS.get_streamflow function now checks if the site info dataset contains any duplicates. Therefore, all the remaining station numbers will be unique. This prevents an issue with setting attrs where duplicate indexes cause an exception when being converted to a dict. (GH 75)

• Add all the missing types so mypy --strict passes.

0.11.4 (2021-11-24)

New Features

• Add support for the Water Quality Portal Web Services. (GH 72)

• Add support for two versions of NID web service. The original NID web service is considered version 2 and the new NID is considered version 3. You can pass the version number to the NID like so NID(2). The default version is 2.

Bug Fixes

• Fix an issue with background percentage calculation in cover_statistics.

0.11.3 (2021-11-12)

New Features

• Add a new map service for National Inventory of Dams (NID).

Internal Changes

• Use importlib-metadata for getting the version instead of pkg_resources to decrease import time as discussed in this issue.

0.11.2 (2021-07-31)

Bug Fixes

• Refactor cover_statistics to address an issue with wrong category names and also improve performance for large datasets by using numpy’s functions.

• Fix an issue with detecting wrong number of stations in NWIS.get_streamflow. Also, improve filtering stations that their start/end date don’t match the user requested interval.

0.11.1 (2021-07-31)

The highlight of this release is adding support for NLCD 2019 and significant improvements in NWIS support.

New Features

• Add support for the recently released version of NLCD (2019), including the impervious descriptor layer. Highlights of the new database are:

  NLCD 2019 now offers land cover for years 2001, 2004, 2006, 2008, 2011, 2013, 2016, 2019, and impervious surface and impervious descriptor products now updated to match each date of land cover. These products update all previously released versions of land cover and impervious products for CONUS (NLCD 2001, NLCD 2006, NLCD 2011, NLCD 2016) and are not directly comparable to previous products. NLCD 2019 land cover and impervious surface product versions of previous dates must be downloaded for proper comparison. NLCD 2019 also offers an impervious surface descriptor product that identifies the type of each impervious surface pixel. This product identifies types of roads, wind tower sites, building locations, and energy production sites to allow deeper analysis of developed features.

  —MRLC

• Add support for all the supported regions of NLCD database (CONUS, AK, HI, and PR).

• Add support for passing multiple years to the NLCD function, like so {"cover": [2016, 2019]}.

• Add plot.descriptor_legends function to plot the legend for the impervious descriptor layer.

• New features in NWIS class are:

  • Remove query_* methods since it’s not convenient to pass them directly as a dictionary.

  • Add a new function called get_parameter_codes to query parameters and get information about them.

  • To decrease complexity of get_streamflow method add a new private function to handle some tasks.

  • For handling more of NWIS’s services make retrieve_rdb more general.

• Add a new argument called nwis_kwds to interactive_map so any NWIS specific keywords can be passed for filtering stations.

• Improve exception handling in get_info method and simplify and improve its performance for getting HCDN.

Internal Changes

• Migrate to using AsyncRetriever for handling communications with web services.

0.11.0 (2021-06-19)

Breaking Changes

• Drop support for Python 3.6 since many of the dependencies such as xarray and pandas have done so.

• Remove get_nid and get_nid_codes functions since NID now has a ArcGISRESTFul service.

New Features

• Add a new class called NID for accessing the recently released National Inventory of Dams web service. This service is based on ArcGIS’s RESTful service. So now the user just need to instantiate the class like so NID() and with three methods of AGRBase class, the user can retrieve the data. These methods are: bygeom, byids, and bysql. Moreover, it has a attrs property that includes descriptions of the database fields with their units.

• Refactor NWIS.get_info to be more generic by accepting any valid queries that are documented at USGS Site Web Service.

• Allow for passing a list of queries to NWIS.get_info and use async_retriever that significantly improves the network response time.

• Add two new flags to interactive_map for limiting the stations to those with daily values (dv=True) and/or instantaneous values (iv=True). This function now includes a link to stations webpage on USGS website.

Internal Changes

• Use persistent caching for all send/receive requests that can significantly improve the network response time.

• Explicitly include all the hard dependencies in setup.cfg.

• Refactor interactive_map and NWIS.get_info to make them more efficient and reduce their code complexity.

0.10.2 (2021-03-27)

Internal Changes

• Add announcement regarding the new name for the software stack, HyRiver.

• Improve pip installation and release workflow.

0.10.1 (2021-03-06)

Internal Changes

• Add lxml to deps.

0.10.0 (2021-03-06)

Internal Changes

• The official first release of PyGeoHydro with a new name and logo.

• Replace cElementTree with ElementTree since it’s been deprecated by defusedxml.

• Make mypy checks more strict and fix all the errors and prevent possible bugs.

• Speed up CI testing by using mamba and caching.

0.9.2 (2021-03-02)

Internal Changes

• Rename hydrodata package to PyGeoHydro for publication on JOSS.

• In NWIS.get_info, drop rows that don’t have mean daily discharge data instead of slicing.

• Speed up Github Actions by using mamba and caching.

• Improve pip installation by adding pyproject.toml.

New Features

• Add support for the National Inventory of Dams (NID) via get_nid function.

0.9.1 (2021-02-22)

Internal Changes

• Fix an issue with NWIS.get_info method where stations with False values as their hcdn_2009 value were returned as None instead.

0.9.0 (2021-02-14)

Internal Changes

• Bump versions of packages across the stack to the same version.

• Use the new PyNHD function for getting basins, NLDI.get_basisn.

• Made mypy checks more strict and added all the missing type annotations.

0.8.0 (2020-12-06)

• Fixed the issue with WaterData due to the recent changes on the server side.

• Updated the examples based on the latest changes across the stack.

• Add support for multipolygon.

• Remove the fill_hole argument.

• Fix a warning in nlcd regarding performing division on nan values.

0.7.2 (2020-8-18)

Enhancements

• Replaced simplejson with orjson to speed-up JSON operations.

• Explicitly sort the time dimension of the ssebopeta_bygeom function.

Bug Fixes

• Fix an issue with the nlcd function where high resolution requests fail.

0.7.1 (2020-8-13)

New Features

• Added a new argument to plot.signatures for controlling the vertical position of the plot title, called title_ypos. This could be useful for multi-line titles.

Bug Fixes

• Fixed an issue with the nlcd function where none layers are not dropped and cause the function to fails.

0.7.0 (2020-8-12)

This version divides PyGeoHydro into six standalone Python libraries. So many of the changes listed below belong to the modules and functions that are now a separate package. This decision was made for reducing the complexity of the code base and allow the users to only install the packages that they need without having to install all the PyGeoHydro dependencies.

Breaking changes

• The services module is now a separate package called PyGeoOGCC and is set as a requirement for PyGeoHydro. PyGeoOGC is a leaner package with much fewer dependencies and is suitable for people who might only need an interface to web services.

• Unified function names for getting feature by ID and by box.

• Combined start and end arguments into a tuple argument called dates across the code base.

• Rewrote NLDI function and moved most of its classmethods to Station so now Station class has more cohesion.

• Removed exploratory functionality of ArcGISREST, since it’s more convenient to do so from a browser. Now, base_url is a required argument.

• Renamed in_crs in datasets and services functions to geo_crs for geometry and box_crs for bounding box inputs.

• Re-wrote the signatures function from scratch using NamedTuple to improve readability and efficiency. Now, the daily argument should be just a pandas.DataFrame or pandas.Series and the column names are used for legends.

• Removed utils.geom_mask function and replaced it with rasterio.mask.mask.

• Removed width as an input in functions with raster output since resolution is almost always the preferred way to request for data. This change made the code more readable.

• Renamed two functions: ArcGISRESTful and wms_bybox. These function now return requests.Response type output.

• onlyipv4 is now a class method in RetrySession.

• The plot.signatures function now assumes that the input time series are in mm/day.

• Added a flag to get_streamflow function in the NWIS class to convert from cms to mm/day which is useful for plotting hydrologic signatures using the signatures functions.

Enhancements

• Remove soft requirements from the env files.

• Refactored requests functions into a single class and a separate file.

• Made all the classes available directly from PyGeoHydro.

• Added CodeFactor to the Github pipeline and addressed some issues that CodeFactor found.

• Added Bandit to check the code for security issue.

• Improved docstrings and documentations.

• Added customized exceptions for better exception handling.

• Added pytest fixtures to improve the tests speed.

• Refactored daymet and nwis_siteinfo functions to reduce code complexity and improve readability.

• Major refactoring of the code base while adding type hinting.

• The input geometry (or bounding box) can be provided in any projection and the necessary re-projections are done under the hood.

• Refactored the method for getting object IDs in ArcGISREST class to improve robustness and efficiency.

• Refactored Daymet class to improve readability.

• Add Deepsource for further code quality checking.

• Automatic handling of large WMS requests (more than 8 million pixels i.e., width x height)

• The json_togeodf function now accepts both a single (Geo)JSON or a list of them

• Refactored plot.signatures using add_gridspec for a much cleaner code.

New Features

• Added access to WaterData’s GeoServer databases.

• Added access to the remaining NLDI database (Water Quality Portal and Water Data Exchange).

• Created a Binder for launching a computing environment on the cloud and testing PyGeoHydro.

• Added a URL repository for the supported services called ServiceURL

• Added support for FEMA web services for flood maps and FWS for wetlands.

• Added a new function called wms_toxarray for converting WMS request responses to xarray.DataArray or xarray.Dataset.

Bug Fixes

• Re-projection issues for function with input geometry.

• Start and end variables not being initialized when coords was used in Station.

• Geometry mask for xarray.DataArray

• WMS output re-projections

0.6.0 (2020-06-23)

• Refactor requests session

• Improve overall code quality based on CodeFactor suggestions

• Migrate to Github Actions from TravisCI

0.5.5 (2020-06-03)

• Add to conda-forge

• Remove pqdm and arcgis2geojson dependencies

0.5.3 (2020-06-07)

• Added threading capability to the flow accumulation function

• Generalized WFS to include both by bbox and by featureID

• Migrate RTD to pip from conda.

• Changed HCDN database source to GagesII database

• Increased robustness of functions that need network connections

• Made the flow accumulation output a pandas Series for better handling of time series input

• Combined DEM, slope, and aspect in a class called NationalMap.

• Installation from pip installs all the dependencies

0.5.0 (2020-04-25)

• An almost complete re-writing of the code base and not backward-compatible

• New website design

• Added vector accumulation

• Added base classes and function accessing any ArcGIS REST, WMS, WMS service

• Standalone functions for creating datasets from responses and masking the data

• Added threading using pqdm to speed up the downloads

• Interactive map for exploring USGS stations

• Replaced OpenTopography with 3DEP

• Added HCDN database for identifying natural watersheds

0.4.4 (2020-03-12)

• Added new databases: NLDI, NHDPLus V2, OpenTopography, gridded Daymet, and SSEBop

• The gridded data are returned as xarray DataArrays

• Removed dependency on StreamStats and replaced it by NLDI

• Improved overall robustness and efficiency of the code

• Not backward comparable

• Added code style enforcement with isort, black, flake8 and pre-commit

• Added a new shiny logo!

• New installation method

• Changed OpenTopography base url to their new server

• Fixed NLCD legend and statistics bug

0.3.0 (2020-02-10)

• Clipped the obtained NLCD data using the watershed geometry

• Added support for specifying the year for getting NLCD

• Removed direct NHDPlus data download dependency by using StreamStats and USGS APIs

• Renamed get_lulc function to get_nlcd

0.2.0 (2020-02-09)

• Simplified import method

• Changed usage from rst format to ipynb

• Auto-formatting with the black python package

• Change docstring format based on Sphinx

• Fixed pytest warnings and changed its working directory

• Added an example notebook with data files

• Added docstring for all the functions

• Added Module section to the documentation

• Fixed py7zr issue

• Changed 7z extractor from pyunpack to py7zr

• Fixed some linting issues.

0.1.0 (2020-01-31)

• First release on PyPI.

History

0.16.2 (2024-02-12)

Bug Fixes

• In add_elvation function, fix a bug where the function fails to add elevation to a xarray.Dataset with x and y dims not being x and y.

Internal Changes

• Refactor fill_depressions function by porting the code from pyflwdir and improve its performance and also now, it directly support xarray.DataArray. Now, pyflwdir is not an optional dependency anymore. You can install numba to improve the performance of the function.

Breaking Changes

• The AirMap service has been deprecated and removed from the package. The elevation_bycoords function now only supports the the National Map and the 3DEP services.

0.16.1 (2024-01-15)

Bug Fixes

• In the check_3dep_availability function when the web service is down the function raises a TypeError instead of setting the value of the failed resolution to Failed. This is fixed now. (GH 66).

Internal Changes

• Simplify the logic of adding elevation to a Dataset in the add_elevation function to avoid modifying CRS of the input Dataset.

0.16.0 (2024-01-03)

New Features

• Add a new function called get_map_vrt for getting DEM within a bounding box and saving it as a VRT file. This function has low memory usage and is useful for cases where the DEM is needed for a large area. Moreover, even for usual use cases it can be much faster than get_dem since it loads the data lazily, at the cost of higher disk usage.

• In the get_map function, check if the input geometry is within the bounds of the 3DEP’s WMS service and if not, raise an exception.

• In the fill_depressions function add a new argument called outlets for specifying outlet detection method: At the edge of all cells (edge) or only the minimum elevation edge cell (min; default).

• Significantly improve the performance of check_3dep_availability function by minimizng the number of requests to the service and sending all requests asynchronously. Also, the returned dict now uses Failed for those resolutions where the service fails to return a valid response. It will remove the failed responses from the cache, so next time the function is called, it will try to get only the failed resolutions.

• Add four new options to add_elevation: mask for passing a mask and resolution for specifying the resolution of the source DEM, and x_dim and y_dim for passing the names of spatial dimensions in the input dataset. The mask option is useful for cases where the input xarray.DataArray or xarray.Dataset has a mask and the user wants to use that mask for the elevation data as well. The resolution option is useful for cases where the user wants to get the elevation data at a higher resolution that will be downsampled by bilinear interpolation to the resolution of the input xarray.DataArray or xarray.Dataset. The default is resolution=None which means the resolution of the input xarray.DataArray or xarray.Dataset will be used. The x_dim and y_dim options are useful for cases where the input xarray.DataArray or xarray.Dataset has different names for spatial dimensions than x and y. The default is x_dim="x" and y_dim="y".

Breaking Changes

• In the elevation_profile function remove the res argument and use 10-m resolution DEM from 3DEP. Also, add two new attributes to the output xarray.Dataset: source for the dataset to state the data source used and units for the distance variable to state the units of the distance, which is meters.

Internal Changes

• Improve initial load time by moving import pyflwdir to the fill_depressions function.

Bug Fixes

• Decrease the number of pixels per request from 10e6 to 8e6 to reduce the request load (GH 65).

0.15.2 (2023-09-22)

Internal Changes

• Remove dependency on dask.

0.15.1 (2023-09-02)

Bug Fixes

• Fix HyRiver libraries requirements by specifying a range instead of exact version so conda-forge can resolve the dependencies.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

New Features

• In static_3dep_dem use rioxarray directly instead of rasterio since it can handle VRT files.

• Improve performance and accuracy of add_elevation by using the dynamic 3DEP service and setting the resolution based on the input xarray.DataArray or xarray.Dataset.

• Improve the performance of elevation_profile by using the static 3DEP service when the input resolution is 10 m (which is the default for this function).

• For now, retain compatibility with shapely<2 while supporting shapley>=2.

Bug Fixes

• In add_elevation, ensure that the resolution is in meters by reprojecting the input dataset to 5070 before extracting resolution and bound attributes.

0.14.0 (2023-03-05)

New Features

• Add a new function called add_elevation for adding elevation data as a new variable to an input xarray.DataArray or xarray.Dataset.

• The elevation_bycoords function now accepts a single coordinate and returns a float in addition to a list of coordinates that returned a list of elevations.

• Modify the elevation_bycoords function to use the new elevation point query service (EPQS) web service. This only affects the source="tnm" option.

Breaking Changes

• Bump the minimum required version of shapely to 2.0, and use its new API.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

0.13.12 (2023-02-01)

New Features

• Use pyflwdir package for depression filling operation instead of richdem since it appears to be unmaintained. Note that pyflwdir is an optional dependency. Also, pyflwdir depends on numba which is not available for Python 3.11 yet. You can follow the progress of numba’s support for Python 3.11 here.

• Add a new function called get_dem for obtaining DEM that is a wrapper of static_3dep_dem and get_map functions. Since static_3dep_dem is faster, if the requested resolution is 10 m, 30 m, or 60 m, static_3dep_dem will be used. Otherwise, get_map will be used.

Internal Changes

• Significantly improve the performance of elevation_bycoords when tep is used as the source by using the static DEM data instead of the dynamic DEM.

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Sync all patch versions of HyRiver packages to x.x.12.

0.13.10 (2023-01-08)

New Features

• Refactor the show_versions function to improve performance and print the output in a nicer table-like format.

Bug Fixes

• Fix a compatibility issue with the new scipy version in elevation_profile where led to failure of interpolation.

0.13.9 (2022-12-15)

Bug Fixes

• Add the missing annotation import to the cache_keys to ensure Python 3.8 and 3.9 work with Python 3.10 style type hinting.

0.13.8 (2022-12-09)

New Features

• Add a new function called static_3dep_dem for getting only DEM data at 10 m, 30, or 60 m resolution. This is useful for cases where only DEM data (i.e., not slope, aspect, or other terrain attributes that the Dynamic 3DEP service provides) is needed. This function is faster than get_map but is less flexible.

Internal Changes

• Modify the codebase based on Refurb suggestions.

0.13.7 (2022-11-04)

Internal Changes

• Use pyupgrade package to update the type hinting annotations to Python 3.10 style.

• Bump the minimum required version of HyRiver dependencies to the latest versions.

0.13.6 (2022-08-30)

Internal Changes

• Add the missing PyPi classifiers for the supported Python versions.

0.13.5 (2022-08-29)

Breaking Changes

• Append “Error” to all exception classes for conforming to PEP-8 naming conventions.

Internal Changes

• Increase the pixel limit for 3DEP’s WMS from 8M to 10M to reduce number of service calls and improve performance.

• Bump the minimum versions of pygeoogc and pygeoutils to 0.13.5 and that of async-retriever to 0.3.5.

0.13.3 (2022-06-25)

Bug Fixes

• Fix a bug in check_3dep_availability where due to changes in pygeoogc ZeroMatched exception is raised instead of TypeError and as a result check_3dep_availability was not working as expected.

0.13.2 (2022-06-14)

Breaking Changes

• Set the minimum supported version of Python to 3.8 since many of the dependencies such as xarray, pandas, rioxarray have dropped support for Python 3.7.

Internal Changes

• Use micromamba for running tests and use nox for linting in CI.

0.13.1 (2022-06-11)

New Features

• In deg2mpm function look for _FillValue and nodatavals in the attributes and if not found, fall back to numpy.nan.

Internal Changes

• Ensure that the deg2mpm function uses dask if the input is dask-enabled.

• In the elevation_profile function use a bounding box to get DEM and a linear interpolation to get the elevation along the profile.

0.13.0 (2022-04-03)

New Features

• Add a new function called query_3dep_sources for querying bounds of 3DEP’s data sources within a bounding box. It returns a geo-dataframe that contains the bounding box of each data source and a column dem_res identifying the resolution of the raw topographic data within each geometry.

• Add a new function called elevation_profile for getting elevation profile along a line at a given spacing. This function converts the line to a B-spline and then calculates the elevation along the spline at a given uniform spacing.

Breaking Changes

• Remove caching-related arguments from all functions since now they can be set globally via three environmental variables:

  • HYRIVER_CACHE_NAME: Path to the caching SQLite database.

  • HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds.

  • HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache file.

  You can do this like so:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/file.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"

0.12.2 (2022-01-15)

New Features

• Add a new DEM source to elevation_bycoords to get elevation from the National Map’s 3DEP WMS service. This can replace the tnm source since tnm is not stable.

• Add a new function called check_3dep_availability to check the availability of 3DEP’s native resolutions within an area of interest. It returns a dict with keys corresponding to the available resolutions and its values are boolean values indicating whether the resolution is available or not.

• Replace no data values of slope in deg2mm with np.nan, so they do not get converted to another value. The output of this function has np.float64 type.

Internal Changes

• Refactor ElevationByCoords by using __post_init__ for validating the input parameters rather than pydantic’s validators.

• Refactor elevation_bygrid by using get_map to get DEM and rioxarray for re-projection.

• Add type checking with typeguard and fixed typing issues raised by typeguard.

• Refactor show_versions to ensure getting correct versions of all dependencies.

0.12.1 (2021-12-31)

Internal Changes

• Use the three new ar.retrieve_* functions instead of the old ar.retrieve function to improve type hinting and to make the API more consistent.

0.12.0 (2021-12-27)

Breaking Changes

• Set the request caching’s expiration time to never expire. Add two flags to all functions to control the caching: expire_after and disable_caching.

Internal Changes

• Add all the missing types so mypy --strict passes.

• Improve performance of elevation_bygrid by ignoring unnecessary validation.

0.11.4 (2021-11-12)

Internal Changes

• Use rioxarray for dealing with GeoTIFF binaries since xarray deprecated the xarray.open_rasterio function, as it’s discussed in this PR.

• Use importlib-metadata for getting the version instead of pkg_resources to decrease import time as discussed in this issue.

0.11.3 (2021-10-03)

Breaking Changes

• Rewrite the command-line interface using click.group to improve UX. The command is now py3dep [command] [args] [options]. The two supported commands are coords for getting elevations of a dataframe of coordinates in EPSG:4326 CRS and geometry for getting the elevation of a geo-dataframe of geometries. Each sub-command now has a separate help message. The format of the input file for the coords command is now csv and for the geometry command is .shp or .gpkg and must have a crs attribute. Also, the geometry command now accepts multiple layers via the --layers (-l) option. More information and examples can be in the README.rst file.

New Features

• Make fill_depressions function public. This function conditions an input DEM by applying depression filling and flat area resolution operations.

Internal Changes

• The get_map function now checks for validation of the input layers argument before sending the actual request with a more helpful message.

• Improve docstrings.

• Move deg2mpm, fill_depressions, and reproject_gtiff functions to a new file called utils. Both deg2mpm and fill_depressions functions are still accessible from py3dep directly.

• Increase the test coverage.

• Use one of the click’s internal functions, click..testing.CliRunner, to run the CLI tests.

0.11.2 (2021-09-17)

Bug Fixes

• Fix a bug related to elevation_bycoords where CRS validation fails if its type is pyrpoj.CRS by converting inputs with CRS types to string.

Internal Changes

• Fix a couple of typing issues and update the get_transform API based on the recent changes in pygeoutils v0.11.5.

0.11.1 (2021-07-31)

The first highlight of this release is a major refactor of elevation_bycoords by adding support for the Bulk Point Query Service and improving the overall performance of the function. Another highlight is support for performing depression filling in elevation_bygrid before sampling the underlying DEM.

New Features

• Refactor elevation_bycoords function to add support for getting elevations of a list of coordinates via The National Map’s Point Query Service. This service is more accurate than Airmap, but it’s limited to the US only. You can select the source via a new argument called source. You can set it to source=tnm to use the TNM service. The default is tnm.

• Refactor elevation_bygrid function to add a new capability via fill_depressions argument for filling depressions in the obtained DEM before extracting elevation data for the input grid points. This is achieved via RichDEM that needs to be installed if this functionality is desired. You can install it via pip or conda (mamba).

Internal Changes

• Migrate to using AsyncRetriever for handling communications with web services.

• Handle the interpolation step in elevation_bygrid function more efficiently using xarray.

0.11.0 (2021-06-19)

New Features

• Added command-line interface (GH 10).

• All feature query functions use persistent caching that can significantly improve the performance.

Breaking Changes

• Drop support for Python 3.6 since many of the dependencies such as xarray and pandas have done so.

• The returned xarray objects are in parallel mode, i.e., in some cases compute method should be used to get the results.

• Save the output as a netcdf instead of raster since conversion from nc to tiff can be easily done with rioxarray.

0.10.1 (2021-03-27)

• Add announcement regarding the new name for the software stack, HyRiver.

• Improve pip installation and release workflow.

0.10.0 (2021-03-06)

• The first release after renaming hydrodata to PyGeoHydro.

• Make mypy checks more strict and fix all the errors and prevent possible bugs.

• Speed up CI testing by using mamba and caching.

0.9.0 (2021-02-14)

• Bump version to the same version as PyGeoHydro.

• Add support for saving maps as geotiff file(s).

• Replace Elevation Point Query Service service with AirMap for getting elevations for a list of coordinates in bulk since AirMap is much faster. The resolution of AirMap is 30 m.

• Use cytoolz for some operations for improving performance.

0.2.0 (2020-12-06)

• Add support for multipolygon.

• Remove the fill_hole argument.

• Add a new function to get elevations for a list of coordinates called elevation_bycoords.

• Refactor elevation_bygrid function for increasing readability and performance.

0.1.7 (2020-08-18)

• Added a rename operation to get_map to automatically rename the variables to a more sensible one.

• Replaced simplejson with orjson to speed-up JSON operations.

0.1.6 (2020-08-11)

• Add a new function, show_versions, for getting versions of the installed dependencies which is useful for debugging and reporting.

• Fix typos in the docs and improved the README.

• Improve testing and coverage.

0.1.5 (2020-08-03)

• Fixed the geometry CRS issue

• Improved the documentation

0.1.4 (2020-07-23)

• Refactor get_map to use pygeoutils package.

• Change the versioning method to setuptools_scm.

• Polish README and add installation from conda-forge.

0.1.0 (2020-07-19)

• First release on PyPI.

History

0.16.1 (2024-01-15)

New Features

• Add a new function for getting Daymet data from Microsoft’s Planetary Computer called get_bystac. Although this function can be much faster than get_bygeom, currently, it gives access to Daymet v4.2 from 1980 to 2020. As discussed here, the Daymet v4.5 will be added to the Planetary Computer in the future. Until then, for accessing the latest version of Daymet (v4.5) you need to use get_bygeom. Additionally, this function requires fsspec, dask, zarr, and pystac-client packages.

• Make separate_snow a standalone, pure, and public function. Now, it can be used like so: pydaymet.separate_snow.

• Change the length unit from km to m for get_bygeom.

Internal Changes

• The potential_et function uses py3dep.add_elevation function but the CRS info gets lost in the process for the new elevation variable. This version fixes this issue by adding the CRS info to the elevation variable.

• Change PetParams class from NamedTuple to dataclass for better performance and consistency. Now, it has a new classmethod called fields that returns a list of the four fields of the class.

0.16.0 (2024-01-03)

Breaking Changes

• Bump min version of shapely to 2.

Internal Changes

• Use the new py3dep.add_elevation API.

0.15.2 (2023-09-22)

Internal Changes

• Remove dependency on dask.

0.15.1 (2023-09-02)

Bug Fixes

• Fix HyRiver libraries requirements by specifying a range instead of exact version so conda-forge can resolve the dependencies.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

New Features

• For now, retain compatibility with shapely<2 while supporting shapley>=2.

0.14.0 (2023-03-05)

New Features

• Change missing value of both single-pixel and gridded versions to numpy.nan from -9999.

• Add a new model parameter for computing PET using priestlet_taylor and penman_monteith models called arid_correction. For arid regions, FAO 56 suggests subtracting the min temperature by 2 degrees. This parameter can be passed via pet_params in daymet_by* functions, or params in potential_pet function.

• Refactor get_bycoords to reduce memory usage by using a combination of itertools and Generator objects.

• Refactor the pet module to improve performance and readability, and reduce code duplication.

Documentation

• Add more information about parameters that pet functions accept.

Breaking Changes

• Bump the minimum required version of shapely to 2.0, and use its new API.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

0.13.12 (2023-02-10)

Internal Changes

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Sync all patch versions of HyRiver packages to x.x.12.

0.13.10 (2023-01-08)

New Features

• Refactor the show_versions function to improve performance and print the output in a nicer table-like format.

Bug Fixes

• Fix a bug in get_bygeom where for small requests that lead to a single download URL, the function failed.

Internal Changes

• Skip 0.13.9 version so the minor version of all HyRiver packages become the same.

0.13.8 (2022-12-09)

Internal Changes

• More robust handling of getting large gridded data. Instead of caching the requests/ responses, directly store the responses as NetCDF files to a cache folder using pygeoogc.streaming_download and ultimately read them using xarray.open_mfdataset. This should make the bygeom function even faster than before and also make it possible to make large requests without having to worry about running out of memory (GH 59).

• Modify the codebase based on Refurb suggestions.

0.13.7 (2022-11-04)

Since the release of Daymet v4 R1 on November 2022, the URL of Daymet’s server has been changed. Therefore, only PyDaymet v0.13.7+ is going to work, and previous versions will not work anymore.

New Features

• Add support for passing a list of coordinates to the get_bycoords function. Also, optionally, you can pass a list of IDs for the input coordinates that will be used as keys for the returned pandas.DataFrame or a dimension called id in the returned xarray.Dataset if to_xarray is enabled.

• Add a new argument called to_xarray to the get_bycoords function for returning the results as a xarray.Dataset instead of a pandas.DataFrame. When set to True, the returned xarray.Dataset will have three attributes called units, description, and long_name.

• The date argument of both get_bycoords and by_geom functions now accepts range-type objects for passing years, e.g., range(2000-2005).

import pydaymet as daymet

coords = [(-94.986, 29.973), (-95.478, 30.134)]
idx = ["P1", "P2"]
clm = daymet.get_bycoords(coords, range(2000, 2021), coords_id=idx, to_xarray=True)

Internal Changes

• Use pyupgrade package to update the type hinting annotations to Python 3.10 style.

• Fix the Daymet server URL.

0.13.6 (2022-08-30)

Internal Changes

• Add the missing PyPi classifiers for the supported Python versions.

0.13.5 (2022-08-29)

Breaking Changes

• Append “Error” to all exception classes for conforming to PEP-8 naming conventions.

Internal Changes

• Bump the minimum versions of pygeoogc, pygeoutils, py3dep to 0.13.5 and that of async-retriever to 0.3.5.

0.13.3 (2022-07-31)

Bug Fixes

• Fix a bug in PETGridded where the wrong data type was being set for pet and elevation variables.

• When initializing PETGridded, only chunk the elevation if the input climate data is chunked.

0.13.2 (2022-06-14)

Breaking Changes

• Set the minimum supported version of Python to 3.8 since many of the dependencies such as xarray, pandas, rioxarray have dropped support for Python 3.7.

Internal Changes

• Use micromamba for running tests and use nox for linting in CI.

0.13.1 (2022-06-11)

New Features

• Adopt the default snow parameters’ values from a new source https://doi.org/10.5194/gmd-11-1077-2018 and add the citation.

Bug Fixes

• Set the end year based on the current year since Daymet data get updated every year (PR 55) by Tim Cera.

• Set the months for the annual timescale to correct values (PR 55) by Tim Cera.

0.13.0 (2022-03-03)

Breaking Changes

• Remove caching-related arguments from all functions since now they can be set globally via three environmental variables:

  • HYRIVER_CACHE_NAME: Path to the caching SQLite database.

  • HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds.

  • HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache file.

  You can do this like so:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/file.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"

0.12.3 (2022-02-04)

New Features

• Add a new flag to both get_bycoords and get_bygeom functions called snow which separates snow from the precipitation using the Martinez and Gupta (2010) method.

Internal Changes

• Add elevation data when computing PET regardless of the pet method.

• Match the chunk size of elevation with that of the climate data.

• Drop time dimension from elevation, lon, and lat variables.

Bug Fixes

• Fix a bug in setting dates for monthly timescales. For monthly timescale Daymet calendar is at 15th or 16th of the month, so input dates need to be adjusted accordingly.

0.12.2 (2022-01-15)

Internal Changes

• Clean up the PET computation functions’ output by removing temporary variables that are created during the computation.

• Add more attributes for elevation and pet variables.

• Add type checking with typeguard and fixed typing issues raised by typeguard.

• Refactor show_versions to ensure getting correct versions of all dependencies.

0.12.1 (2021-12-31)

Internal Changes

• Use the three new ar.retrieve_* functions instead of the old ar.retrieve function to improve type hinting and to make the API more consistent.

0.12.0 (2021-12-27)

New Features

• Expose the ssl argument for disabling the SSL certification verification (GH 41). Now, you can pass ssl=False to disable the SSL verification in both get_bygeom and get_bycoord functions. Moreover, you can pass --disable_ssl to PyDaymet’s command line interface to disable the SSL verification.

Breaking Changes

• Set the request caching’s expiration time to never expire. Add two flags to all functions to control the caching: expire_after and disable_caching.

Internal Changes

• Add all the missing types so mypy --strict passes.

0.11.4 (2021-11-12)

Internal Changes

• Use importlib-metadata for getting the version instead of pkg_resources to decrease import time as discussed in this issue.

0.11.3 (2021-10-07)

Bug Fixes

• There was an issue in the PET computation due to dayofyear being added as a new dimension. This version fixes it and even further simplifies the code by using xarray’s dt accessor to gain access to the dayofyear method.

0.11.2 (2021-10-07)

New Features

• Add hargreaves_samani and priestley_taylor methods for computing PET.

Breaking Changes

• Rewrite the command-line interface using click.group to improve UX. The command is now pydaymet [command] [args] [options]. The two supported commands are coords for getting climate data for a dataframe of coordinates and geometry for getting gridded climate data for a geo-dataframe. Moreover, Each sub-command now has a separate help message and example.

• Deprecate get_byloc in favor of get_bycoords.

• The pet argument in both get_bycoords and get_bygeom functions now accepts hargreaves_samani, penman_monteith, priestley_taylor, and None.

Internal Changes

• Refactor the pet module for reducing duplicate code and improving readability and maintainability. The code is smaller now and the functions for computing physical properties include references to equations from the respective original paper.

0.11.1 (2021-07-31)

The highlight of this release is a major refactor of Daymet to allow for extending PET computation function for using methods other than FAO-56.

New Features

• Refactor Daymet class by removing pet_bycoords and pet_bygrid methods and creating a new public function called potential_et. This function computes potential evapotranspiration (PET) and supports both gridded (xarray.Dataset) and single pixel (pandas.DataFrame) climate data. The long-term plan is to add support for methods other than FAO 56 for computing PET.

0.11.0 (2021-06-19)

New Features

• Add command-line interface (GH 7).

• Use AsyncRetriever for sending requests asynchronously with persistent caching. A cache folder in the current directory is created.

• Check for validity of start/end dates based on Daymet V4 since Puerto Rico data starts from 1950 while North America and Hawaii start from 1980.

• Check for validity of input coordinate/geometry based on the Daymet V4 bounding boxes.

• Improve accuracy of computing Psychometric constant in PET calculations by using an equation in Allen et al. 1998.

Breaking Changes

• Drop support for Python 3.6 since many of the dependencies such as xarray and pandas have done so.

• Change loc_crs and geo_crs arguments to crs in get_bycoords and get_bygeom.

Documentation

• Add examples to docstrings and improve writing.

• Add more notes regarding the underlying assumptions for pet_bycoords and pet_bygrid.

Internal Changes

• Refactor Daymet class to use pydantic for validating the inputs.

• Increase test coverage.

0.10.2 (2021-03-27)

• Add announcement regarding the new name for the software stack, HyRiver.

• Improve pip installation and release workflow.

0.10.0 (2021-03-06)

• The first release after renaming hydrodata to PyGeoHydro.

• Make mypy checks more strict and fix all the errors and prevent possible bugs.

• Speed up CI testing by using mamba and caching.

0.9.0 (2021-02-14)

• Bump version to the same version as PyGeoHydro.

• Update to version 4 of Daymet database. You can check the release information here

• Add a new function called get_bycoords that provides an alternative to get_byloc for getting climate data at a single pixel. This new function uses THREDDS data server with NetCDF Subset Service (NCSS), and supports getting monthly and annual averages directly from the server. Note that this function will replace get_byloc in the future. So consider migrating your code by replacing get_byloc with get_bycoords. The input arguments of get_bycoords is very similar to get_bygeom. Another difference between get_byloc and get_bycoords is column names where get_bycoords uses the units that are return by NCSS server.

• Add support for downloading monthly and annual summaries in addition to the daily timescale. You can pass time_scale as daily, monthly, or annual to get_bygeom or get_bycoords functions to download the respective summaries.

• Add support for getting climate data for Hawaii and Puerto Rico by passing region to get_bygeom and get_bycoords functions. The acceptable values are na for CONUS, hi for Hawaii, and pr for Puerto Rico.

0.2.0 (2020-12-06)

• Add support for multipolygon.

• Remove the fill_hole argument.

• Improve masking by geometry.

• Use the newly added async_requests function from pygeoogc for getting Daymet data to increase the performance (almost 2x faster)

0.1.3 (2020-08-18)

• Replaced simplejson with orjson to speed-up JSON operations.

0.1.2 (2020-08-11)

• Add show_versions for showing versions of the installed deps.

0.1.1 (2020-08-03)

• Retained the compatibility with xarray 0.15 by removing the attrs flag.

• Replaced open_dataset with load_dataset for automatic handling of closing the input after reading the content.

• Removed years argument from both byloc and bygeom functions. The dates argument now accepts both a tuple of start and end dates and a list of years.

0.1.0 (2020-07-27)

• Initial release on PyPI.

History

0.16.0 (2024-01-03)

• Initial release on PyPI.

History

0.16.0 (2024-01-03)

Internal Changes

• Drop support for shapely<2.

0.15.2 (2023-09-22)

Internal Changes

• Remove dependency on dask.

• Reduce complexity of the code by breaking down the _check_inputs function into _get_variables and _get_dates functions.

0.15.1 (2023-07-10)

Bug Fixes

• Fix a bug in computing snow where the t_snow argument was not being converted to Kelvin.

New Features

• If snow=True is passed to both get_bygeom and get_bycoords functions, the variables argument will be checked to see if it contains prcp and temp, if not, they will be added to the list of variables to be retrieved. This is to ensure that the snow argument works as expected.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

New Features

• Add source argument to both get_bygeom and get_bycoords functions. Valid values for source are grib (default) and netcdf. Both return the same values, the latter also offers additional variable psurf for surface pressure. Valid variable names for netcdf are: prcp, pet, wind_u, wind_v, humidity, temp, rsds, rlds, psurf Valid variable names for grib source are unchanged as to not introduce breaking changes. By Luc Rébillout.

• For now, retain compatibility with shapely<2 while supporting shapley>=2.

0.14.0 (2023-03-05)

New Features

• Add snow and snow_params arguments to both get_bygeom and get_bycoords functions for computing snow from prcp and temp.

• Rewrite by_coords functions to improve performance and reduce memory usage. Also, its to_xarray argument now returns a much better structured xarray.Dataset. Moreover, the function has a new argument called coords_id which allows the user to specify IDs for the input coordinates. This is useful for cases where the coordinates belong to some specific features, such as station location, that have their own IDs. These IDs will be used for both cases where the data is returned as pandas.DataFrame or xarray.Dataset.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

0.1.12 (2023-02-10)

Internal Changes

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Sync all patch versions of HyRiver packages to x.x.12.

0.1.2 (2023-01-08)

New Features

• Refactor the show_versions function to improve performance and print the output in a nicer table-like format.

0.1.1 (2022-12-16)

Bug Fixes

• Fix an issue where for single variable, i.e., not a list, could not be detected correctly.

• Fix an issue in converting the response from the service to a dataframe or dataset when service fails and throws an error.

0.1.0 (2022-12-15)

• Initial release.

History

0.16.0 (2024-01-03)

New Features

• Add a new function called flashiness_index for computing the flashiness index of a daily streamflow time series following Baker et al. (2004).

Breaking Changes

• Improve function naming convention by removing the compute_ prefix from all functions and spelling out the full name of the function. For example, compute_fdc_slope and compute_ai are now flow_duration_curve_slope and aridity_index, respectively.

0.15.2 (2023-09-22)

New Features

• Add an option to compute_mean_monthly for specifying whether the input data unit is in mm/day or m3/s. If m3/s, then the monthly values are computed by taking the mean of the daily values for each month. If mm/day, then the monthly values are computed by taking the sum of the daily values for each month.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

Internal Changes

• Explicitly use nopython mode in numba-decorated functions to avoid deprecation warnings.

0.14.0 (2023-03-05)

Bug Fixes

• Address an issue in compute_fdc_slope where if the input includes NANs, it returns NAN. Now, the function correctly handles NAN values. Also, this function now works with any array-like input, i.e., pandas.Series, pandas.DataFrame, numpy.ndarray, and xarray.DataArray. Also, the denominator should have been divided by 100 since the input bins are percentiles.

• Fix a bug in compute_ai where instead of using mean annual average values, daily values was being used. Also, this function now accepts xarray.DataArray too.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

0.1.12 (2023-02-10)

Internal Changes

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Sync all patch versions of HyRiver packages to x.x.12.

0.1.2 (2023-01-08)

New Features

• Refactor the show_versions function to improve performance and print the output in a nicer table-like format.

Internal Changes

• Use pyright for type checking and fix all typing issues that it raised.

• Add xarray as a dependency.

0.1.1 (2022-11-04)

New Features

• Add a new function called compute_ai for computing the aridity index.

• Add a new function called compute_flood_moments for computing flood moments: Mean annual flood, coefficient of variation, and coefficient of skewness.

• Add a stand-alone function for computing the FDC slope, called compute_fdc_slope.

Breaking Changes

• Remove the runoff_ratio_annual function.

0.1.0 (2022-10-03)

• First release on PyPI.

History

0.16.0 (2024-01-03)

New Features

• Add a new environmental variable called "HYRIVER_SSL_CERT" for setting the path to a SSL certificate file other than the default one. You can do this like so:

import os

os.environ["HYRIVER_SSL_CERT"] = "path/to/file.pem"

0.15.2 (2023-09-22)

Bug Fixes

• Fix an issue with getting all valid keywords that aiohttp accepts by using aiohttp.ClientSession()._request directly.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

Bug Fixes

• When raise_status is False, responses for failed requests used to return as None but their requests ID was not returned, so sorting would have failed. Now request IDs are returned for all requests regardless of whether they were successful or not.

• Give precedence to non-default arguments for caching related arguments instead of directly getting them from env variables. This is to avoid the case where the user sets the env variables but then passes different arguments to the function. In this case, the function should use the passed arguments instead of the env variables.

0.14.0 (2023-03-05)

New Features

• Add a new option to all functions called raise_status. If False no exception will be raised and instead None is returned for those requests that led to exceptions. This will allow for returning all responses that were successful and ignoring the ones that failed. This option defaults to True for retaining backward compatibility.

• Set the cache expiration time to one week from never expire. To ensure all users have a smooth transition, cache files that were created before the release of this version will be deleted, and a new cache will be created.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

0.3.12 (2023-02-10)

Internal Changes

• Rewrite the private async_session function as two separate functions called async_session_without_cache and async_session_with_cache. This makes the code more readable and easier to maintain.

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Make utils module private.

• Sync all patch versions of HyRiver packages to x.x.12.

0.3.10 (2023-01-08)

New Features

• Refactor the show_versions function to improve performance and print the output in a nicer table-like format.

Bug Fixes

• Fix a bug in reading the HYRIVER_CACHE_EXPIRE environmental variable.

• Bump the minimum version of aiohttp-client-cache to 0.8.1 to fix a bug in reading cache files that were created with previous versions. (GH 41)

Internal Changes

• Enable fast_save in aiohttp-client-cache to speed up saving responses to the cache file.

• Use pyright for type checking instead of mypy and fix all type errors.

• Skip 0.13.8/9 versions so the minor version of all HyRiver packages become the same.

0.3.7 (2022-12-09)

New Features

• Add support for specifying the chunk size in stream_write. Defaults to None which was the default behavior before, and means iterating over and writing the responses as they are received from the server.

Internal Changes

• Use pyupgrade package to update the type hinting annotations to Python 3.10 style.

• Modify the codebase based on Refurb suggestions.

0.3.6 (2022-08-30)

Internal Changes

• Add the missing PyPi classifiers for the supported Python versions.

• Release the package as both async_retriever and async-retriever on PyPi and Conda-forge.

0.3.5 (2022-08-29)

Breaking Changes

• Append “Error” to all exception classes for conforming to PEP-8 naming conventions.

Internal Changes

• Bump minimum version of aiohttp-client-cache to 0.7.3 since the attrs version issue has been addressed.

0.3.4 (2022-07-31)

New Features

• Add a new function, stream_write, for writing a response to a file as it’s being retrieved. This could be very useful for downloading large files. This function does not use persistent caching.

0.3.3 (2022-06-14)

Breaking Changes

• Set the minimum supported version of Python to 3.8 since many of the dependencies such as xarray, pandas, rioxarray have dropped support for Python 3.7.

Internal Changes

• Use micromamba for running tests and use nox for linting in CI.

0.3.2 (2022-04-03)

New Features

• Add support for setting caching-related arguments using three environmental variables:

  • HYRIVER_CACHE_NAME: Path to the caching SQLite database.

  • HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds.

  • HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache file.

  You can do this like so:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/file.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"

Internal Changes

• Include the URL of a failed request in its exception error message.

0.3.1 (2021-12-31)

New Features

• Add three new functions called retrieve_text, retrieve_json, and retrieve_binary. These functions are derived from the retrieve function and are used to retrieve the text, JSON, or binary content of a response. They are meant to help with type hinting since they have only one return type instead of the three different return types that the retrieve function has.

Internal Changes

• Move all private functions to a new module called utils. This makes the code-base more readable and easier to maintain.

0.3.0 (2021-12-27)

Breaking Changes

• Set the expiration time to never expire by default.

New Features

• Add two new arguments to retrieve for controlling caching. First, delete_url_cache for deleting caches for specific requests. Second, expire_after for setting a custom expiration time.

• Expose the ssl argument for disabling the SSL certification verification (GH 41).

• Add a new option called disable that temporarily disables caching requests/responses if set to True. It defaults to False.

0.2.5 (2021-11-09)

New Features

• Add two new arguments, timeout and expire_after, to retrieve. These two arguments give the user more control in dealing with issues related to caching.

Internal Changes

• Revert to pytest as the testing framework.

• Use importlib-metadata for getting the version instead of pkg_resources to decrease import time as discussed in this issue.

0.2.4 (2021-09-10)

Internal Changes

• Use ujon for converting responses to JSON.

Bug Fixes

• Fix an issue with catching service error messages.

0.2.3 (2021-08-26)

Internal Changes

• Use ujson for JSON parsing instead of orjson since orjson only serializes to bytes which is not compatible with aiohttp.

0.2.2 (2021-08-19)

New Features

• Add a new function, clean_cache, for manually removing the expired responses from the cache database.

Internal Changes

• Handle all cache file-related operations in the create_cachefile function.

0.2.1 (2021-07-31)

New Features

• The responses now are returned to the same order as the input URLs.

• Add support for passing connection type, i.e., IPv4 only, IPv6 only, or both via the family argument. Defaults to both.

• Set trust_env=True, so the session can read the system’s netrc files. This can be useful for working with services such as EarthData service that read the user authentication info from a netrc file.

Internal Changes

• Replace the AsyncRequest class with the _retrieve function to increase readability and reduce overhead.

• More robust handling of validating user inputs via a new class called ValidateInputs.

• Move all if-blocks in async_session to other functions to improve performance.

0.2.0 (2021-06-17)

Breaking Changes

• Make persistent caching dependencies required.

• Rename request argument to request_method in retrieve which now accepts both lower and upper cases of get and post.

Bug Fixes

• Pass a new loop explicitly to nest_asyncio (GH 1).

Internal Changes

• Refactor the entire code-base for more efficient handling of different request methods.

• Check the validity of inputs before sending requests.

• Improve documentation.

• Improve cache handling by removing the expired responses before returning the results.

• Increase testing coverage to 100%.

0.1.0 (2021-05-01)

• Initial release.

History

0.16.2 (2024-XX-XX)

Internal Changes

• Remove the deprecated AirMap URL.

0.16.1 (2024-01-15)

Bug Fixes

• pyproj uses its own env variables for SSL certification. This release fixes the issue with pyproj not being able to download the grid database when using DOI SSL certification file. This release uses pyproj.network.set_ca_bundle_path for setting the SSL certification file given by the user via HYRIVER_SSL_CERT env variable.

• Fix an issue in WFS.getfeature_byid where the max_nrecords argument was not being used correctly, thus resulting in large requests to fail.

Internal Changes

• For ServiceURL class, use dataclass instead for better performance and consistency.

0.16.0 (2024-01-03)

New Features

• Add a new arg to WMS.getmap_bybox called tiff_dir for storing the responses from a WMS request as a GeoTIFF file on disk instead of keeping all responses in memory. When this arg is given the function return a list of paths to these files. This is useful for large requests where the response is too large to be kept in memory. You can create a VRT file from these files using pygeoutils.gtiff2vrt function.

0.15.2 (2023-09-22)

New Features

• Added RESTfulURLs for FEMA’s National Flood Hazard Layer (NFHL) service. Contributed by Fernando Aristizabal. (PR 62)

• Now, RetrySession can be used as a context manager. This is useful for closing the session after using it. For example:

from pygeoogc import RetrySession

with RetrySession() as session:
    r = session.get("https://httpbin.org/get").json()

Internal Changes

• Improve the example in the docstring of traverse_json function.

• Improve exception handling in the ArcGISRESTful class and return a more informative error message.

0.15.1 (2023-08-02)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have pygeoogc==0.14.x installed, you cannot install pygeoogc==0.15.x series. This is to ensure that the API is consistent across all minor versions.

New Features

• Add the STN Flood Event Data URL to the list of RESTfuls. Contributed by Fernando Aristizabal. (PR 59)

• Add the link for the eHydro’s web service.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have pygeoogc==0.14.x installed, you cannot install pygeoogc==0.15.x series. This is to ensure that the API is consistent across all minor versions.

New Features

• For now, retain compatibility with shapely<2 while supporting shapley>=2.

Bug Fixes

• Fix an issue in WFS where the getfeature_bygeom method fails if the requested web service does not have geometry_column attribute in its schema. This release addresses this issue by trying to find the name from other attributes in the schema. If it fails to find, it raises a ValueError.

• Catch an edge case in match_crs function where the input is a list of coordinates of length 4.

• Give precedence to non-default arguments for caching related arguments instead of directly getting them from env variables. This is to avoid the case where the user sets the env variables but then passes different arguments to the function. In this case, the function should use the passed arguments instead of the env variables.

Internal Changes

• Remove pyyaml as a dependency since it is not used anymore.

0.14.0 (2023-03-05)

Breaking Changes

• Bump the minimum required version of shapely to 2.0, and use its new API.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

0.13.12 (2023-02-10)

New Features

• Make match_crs less strict in terms of the input geometry type being tuple or list by relying on shapely and contextlib.suppress. So, now users can pass any combination of list or tuple as coordinates or bounding box.

• More robust handling of inputs and outputs in streaming_download. Now, only if input is str the function returns a single Path object. Previously if there was only one URL, whether list of length one or str, the output was a single Path, which could have had unintended consequences.

Bug Fixes

• In WFS when some layers have missing schema info, the class failed to initialize. This release fixes this issue by ignoring layers with missing schema info and asks the user to pass a sort parameter instead of trying to automatically find a sort parameter. This fix also improves the performance of this function by making fewer web requests.

Internal Changes

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Sync all patch versions of HyRiver packages to x.x.12.

0.13.10 (2023-01-08)

Bug Fixes

• Remove all Python 3.9 type-annotation-style in the codebase except for function signatures to ensure compatibility with Python 3.8. (GH 57, PR 58). Thanks to Tim Cera for reporting and fixing the issue.

Internal Changes

• Use pyright for type checking instead of mypy since it is faster and more accurate. Also, fix all the type errors reported by pyright.

• Improve code quality by addressing issues raised by DeepSource.

0.13.9 (2022-12-15)

Bug Fixes

• Add the missing annotation import to the cache_keys to ensure Python 3.8 and 3.9 work with Python 3.10 style type hinting.

0.13.8 (2022-12-09)

New Features

• Add a new property to WFS class called schema that contains information about column names and their types for all layers. It also the geometry type and its name for each layer.

• Automatically determine the geometry keyword that should be passed to WFS.getfeature_bygeom using the new schema property of WFS.

• Add support for disabling SSL verification to RetrySession via ssl parameter.

• Add support for streaming responses to RetrySession via stream parameter to get and post methods.

• Add support for closing the session to RetrySession via close method.

• Add support for passing params, data, and json to RetrySession via get and post methods. Previously, keyword payload was used for params in get and data in post. Now, params and data can also be passed as keyword arguments to these methods.

• Add a new function called streaming_download for downloading large files in parallel and in chunks.

Bug Fixes

• Fix an issue in WFS class where number of requested features exceeds the maximum number of features allowed by the server, but only a portion of the features are returned. This release addresses this issue by first getting only the number of features and then requesting the features in chunks of features IDs based on the maximum number of features allowed by the server.

Internal Changes

• Drop support for WFS version 1.0.0 since it does not support paging.

• Modify the codebase based on Refurb suggestions.

Bug Fixes

• Fix the warning message in ArcGISRESTFul where wrong number of missing feature IDs were being reported.

0.13.7 (2022-11-04)

New Features

• Add a new method to RetrySession for getting the request head called RetrySession.head. This is useful for getting the headers of a request without having to make a full request which is useful for getting the Content-Length header for example, i.e., download size.

Bug Fixes

• Fix an issue in the decompose function, utils.bbox_decompose, where the generated bounding boxes might overlap in some cases. A new approach has been implemented based on finding the number of required bounding boxes from max allowable no. of pixels and total requested pixels without changing the input bounding box projection. This ensures that the decomposed bounding boxes are not overlapping so xarray.open_mfdataset can be used without any issues.

Internal Changes

• In the utils.match_crs function, don’t perform any projection if the source target CRS are the same.

• Improve type hints for CRS-related arguments of all functions by including string, integer, and pyproj.CRS types.

• Add a new class method to WMSBase and WFSBase classes called get_service_options for retrieving the available layers, output formats, and CRSs for a given service. Here’s an example:

• Use pyupgrade package to update the type hinting annotations to Python 3.10 style.

from pygeoogc.core import WMSBase

url = "https://elevation.nationalmap.gov/arcgis/services/3DEPElevation/ImageServer/WMSServer"
wms = WMSBase(url, validation=False)
wms.get_service_options()
print(wms.available_layer)

0.13.6 (2022-08-30)

Internal Changes

• Add the missing PyPi classifiers for the supported Python versions.

0.13.5 (2022-08-29)

Breaking Changes

• Append “Error” to all exception classes for conforming to PEP-8 naming conventions.

Internal Changes

• Bump minimum version of owslib to 0.27.2 since the pyproj incompatibility issue has been addressed in this issue.

• Bump minimum version of requests-cache to 0.9.6 since the attrs version issue has been addressed.

0.13.3 (2022-07-31)

New Features

• Add support for disabling persistent caching in RetrySession via an argument and also HYRIVER_CACHE_DISABLE environmental variable.

0.13.2 (2022-06-14)

Breaking Changes

• Set the minimum supported version of Python to 3.8 since many of the dependencies such as xarray, pandas, rioxarray have dropped support for Python 3.7.

• Pin owslib to version <0.26 since version 0.26 has pinned pyproj to version <3.3 which is not compatible with rasterio on macOS.

Internal Changes

• Use micromamba for running tests and use nox for linting in CI.

0.13.1 (2022-06-11)

New Features

• More robust handling of errors in ArcGISRESTful by catching None responses. Also, use the POST method for ArcGISRESTful.bysql since the SQL Clause could be a long string.

0.13.0 (2022-04-03)

Breaking Changes

• Remove caching-related arguments from all functions since now they can be set globally via three environmental variables:

  • HYRIVER_CACHE_NAME: Path to the caching SQLite database.

  • HYRIVER_CACHE_EXPIRE: Expiration time for cached requests in seconds.

  • HYRIVER_CACHE_DISABLE: Disable reading/writing from/to the cache file.

  You can do this like so:

import os

os.environ["HYRIVER_CACHE_NAME"] = "path/to/file.sqlite"
os.environ["HYRIVER_CACHE_EXPIRE"] = "3600"
os.environ["HYRIVER_CACHE_DISABLE"] = "true"

Bug Fixes

• In ArcGISRESTful.oids_byfield convert the input ids to a list if a user passes a single id.

Internal Changes

• Refactor ServicURL to hard code the supported links instead of reading them from a file. Also, the class now is based on NamedTuple that has a nicer __repr__.

0.12.2 (2022-01-15)

New Features

• Make validate_crs public that can be accessed from the utils module. This is useful for checking validity of user input CRS values and getting its string representation.

• Add pygeoogc.utils.valid_wms_crs function for getting a list of valid CRS values from a WMS service.

• Add 3DEP’s index WFS service for querying availability of 3DEP data within a bounding box.

Internal Changes

• Add type checking with typeguard and fixed typing issues raised by typeguard.

• Refactor show_versions to ensure getting correct versions of all dependencies.

0.12.1 (2021-12-31)

Internal Changes

• Use the three new ar.retrieve_* functions instead of the old ar.retrieve function to improve type hinting and to make the API more consistent.

0.12.0 (2021-12-27)

New Features

• Add a new argument to ArcGISRESTful called verbose to turn on/off all info level logs.

• Add an option to ArcGISRESTful.get_features called get_geometry to turn on/off requesting the data with or without geometry.

• Now, ArcGISRESTful saves the object IDs of the features that user requested but are not available in the database to ./cache/failed_request_ids.txt.

• Add a new parameter to ArcGISRESTful called disable_retry that If True in case there are any failed queries, no retrying attempts is done and object IDs of the failed requests are saved to a text file which its path can be accessed via ArcGISRESTful.client.failed_path.

• Set response caching expiration time to never expire, for all base classes. A new argument has been added to all three base classes called expire_after that can be used to set the expiration time.

• Add a new method to all three base classes called clear_cache that clears all cached responses for that specific client.

Breaking Changes

• All oids_by* methods of ArcGISRESTful class now return a list of object IDs rather than setting self.featureids. This makes it possible to pass the outputs of the oids_by* functions directly to the get_features method.

Internal Changes

• Make ArcGISRESTful less cluttered by instantiating ArcGISRESTfulBase in the init method of ArcGISRESTful rather than inheriting from its base class.

• Explicitly set a minimum value of 1 for the maximum number of feature IDs per request in ArcGISRESTful, i.e., self.max_nrecords.

• Add all the missing types so mypy --strict passes.

0.11.7 (2021-11-09)

Breaking Changes

• Remove the onlyipv4 method from RetrySession since it can be easily be achieved using with unittest.mock.patch("socket.has_ipv6", False):.

Internal Changes

• Use the geoms method for iterating over geometries to address the deprecation warning of shapely.

• Use importlib-metadata for getting the version instead of pkg_resources to decrease import time as discussed in this issue.

• Remove unnecessary dependency on simplejson and use ujson instead.

0.11.5 (2021-09-09)

Bug Fixes

• Update the code to use the latest requsts-cache API.

0.11.4 (2021-08-26)

New Features

• Add URL for PyGeoAPI service.

0.11.3 (2021-08-21)

Internal Changes

• Fix a bug in WFS.getfeature_byid when the number of IDs exceeds the service’s limit by splitting large requests into multiple smaller requests.

• Add two new arguments, max_nrecords and read_method, to WFS to control the maximum number of records per request (defaults to 1000) and specify the response read method (defaults to json), respectively.

0.11.2 (2021-08-19)

Internal Changes

• Simplify the retry logic ArcGISRESTFul by making it run four times and making sure that the last retry is one object ID per request.

0.11.1 (2021-07-31)

The highlight of this release is migrating to use AsyncRetriever that can improve the network response time significantly. Another highlight is a major refactoring of ArcGISRESTFul that improves performance and reduce code complexity.

New Features

• Add a new method to ArcGISRESTFul class for automatically retrying the failed requests. This private method plucks out individual features that were in a failed request with several features. This happens when there are some object IDs that are not available on the server, and they are included in the request. In these situations the request will fail, although there are valid object IDs in the request. This method will pluck out the valid object IDs.

• Add support for passing additional parameters to WMS requests such as styles.

• Add support for WFS version 1.0.0.

Internal Changes

• Migrate to AsyncRetriever from requests-cache for all the web services.

• Rename ServiceError to ServiceUnavailable and ServerError to ServiceError Since it’s more representative of the intended exception.

• Raise for response status in RetrySession before the try-except block so RequestsException can raise, and its error messaged be parsed.

• Deprecate utils.threading since all threading operations are now handled by AsyncRetriever.

• Increase test coverage.

0.11.0 (2021-06-18)

New Features

• Add support for requesting LineString polygon for ArcGISRESTful.

• Add a new argument called distance to ArcGISRESTful.oids_bygeom for specifying the buffer distance from the input geometry for getting features.

Breaking Changes

• Drop support for Python 3.6 since many of the dependencies such as xarray and pandas have done so.

• Remove async_requests function, since it has been packaged as a new Python library called AsyncRetriever.

• Refactor MatchCRS. Now, it should be instantiated by providing the in and out CRSs like so: MatchCRS(in_crs, out_crs). Then its methods, namely, geometry, bounds and coords, can be called. These methods now have only one input, geometry.

• Change input and output types of MatchCRS.coords from tuple of lists of coordinates to list of (x, y) coordinates.

• ArcGISRESTful now has a new argument, layer, for specifying the layer number (int). Now, the target layer should either be a part of base_url or be passed with layer argument.

• Move the spatial_relation argument from ArcGISRESTful class to oids_bygeom method, since that’s where it’s applicable.

Internal Changes

• Refactor ArcGISRESTfulBase class to reduce its code complexity and make the service initialization logic much simpler. The class is faster since it makes fewer requests during the initialization process.

• Add pydantic as a new dependency that takes care of ArcGISRESTfulBase validation.

• Use persistent caching for all send/receive requests that can significantly improve the network response time.

• Explicitly include all the hard dependencies in setup.cfg.

• Set a default value of 1000 for max_nrecords in ArcGISRESTfulBase.

• Use dataclass for WMSBase and WFSBase since support for Python 3.6 is dropped.

0.10.1 (2021-03-27)

• Add announcement regarding the new name for the software stack, HyRiver.

• Improve pip installation and release workflow.

0.10.0 (2021-03-06)

• The first release after renaming hydrodata to PyGeoHydro.

• Fix extent property of ArcGISRESTful being set to None incorrectly.

• Add feature types property to ArcGISRESTFul for getting names and IDs of types of features in the database.

• Replace cElementTree with ElementTree since it’s been deprecated by defusedxml.

• Remove dependency on dataclasses since its benefits and usage in the code was minimal.

• Speed up CI testing by using mamba and caching.

• ArcGISRESTFull now prints number of found features before attempting to retrieve them.

• Use logging module for printing information.

0.9.0 (2021-02-14)

• Bump version to the same version as PyGeoHydro.

• Add support for query by point and multi-points to ArcGISRESTful.bygeom.

• Add support for buffer distance to ArcGISRESTful.bygeom.

• Add support for generating ESRI-based queries for points and multi-points to ESRIGeomQuery.

• Add all the missing type annotations.

• Update the Daymet URL to version 4. You can check the release information here

• Use cytoolz library for improving performance of some operations.

• Add extent property to ArcGISRESTful class that get the spatial extent of the service.

• Add URL to airmap service for getting elevation data at 30 m resolution.

0.2.3 (2020-12-19)

• Fix urlib3 deprecation warning about using method_whitelist.

0.2.2 (2020-12-05)

• Remove unused variables in async_requests and use max_workers.

• Fix the async_requests issue on Windows systems.

0.2.0 (2020-12-06)

• Added/Renamed three class methods in ArcGISRESTful: oids_bygeom, oids_byfield, and oids_bysql. So you can query feature within a geometry, using specific field ID(s), or more generally using any valid SQL 92 WHERE clause.

• Added support for query with SQL WHERE clause to ArcGISRESTful.

• Changed the NLDI’s URL for migrating to its new API v3.

• Added support for CQL filter to WFS, credits to Emilio.

• Moved all the web services URLs to a YAML file that ServiceURL class reads. It makes managing the new URLs easier. The file is located at pygeoogc/static/urls.yml.

• Turned off threading by default for all the services since not all web services supports it.

• Added support for setting the request method, GET or POST, for WFS.byfilter, which could be useful when the filter string is long.

• Added support for asynchronous download via the function async_requests.

0.1.10 (2020-08-18)

• Improved bbox_decompose to fix the WMS issue with high resolution requests.

• Replaces simplejson with orjson to speed up JSON operations.

0.1.8 (2020-08-12)

• Removed threading for WMS due to inconsistent behavior.

• Addressed an issue with domain decomposition for WMS where width/height becomes 0.

0.1.7 (2020-08-11)

• Renamed vsplit_bbox to bbox_decompose. The function now decomposes the domain in both directions and return squares and rectangular.

0.1.5 (2020-07-23)

• Re-wrote wms_bybox function as a class called WMS with a similar interface to the WFS class.

• Added support for WMS 1.3.0 and WFS 2.0.0.

• Added a custom Exception for the threading function called ThreadingException.

• Add always_xy flag to WMS and WFS which is False by default. It is useful for cases where a web service doesn’t change the axis order from the transitional xy to yx for versions higher than 1.3.0.

0.1.3 (2020-07-21)

• Remove unnecessary transformation of the input bbox in WFS.

• Use setuptools_scm for versioning.

0.1.2 (2020-07-16)

• Add the missing max_pixel argument to the wms_bybox function.

• Change the onlyIPv4 method of RetrySession class to onlyipv4 to conform to the snake_case convention.

• Improve docstrings.

0.1.1 (2020-07-15)

• Initial release.

History

0.16.1 (2024-01-15)

Bug Fixes

• pyproj uses its own env variables for SSL certification. This release fixes the issue with pyproj not being able to download the grid database when using DOI SSL certification file. This release uses pyproj.network.set_ca_bundle_path for setting the SSL certification file given by the user via HYRIVER_SSL_CERT env variable.

• Ignore FutureWarning of pandas 2.1.0 for all-NaN columns in json2geodf.

Internal Changes

• For Attrs class, use dataclass instead for better performance and consistency.

0.16.0 (2024-01-03)

Breaking Changes

• Refactor the spline generation functions to make them more efficient, more accurate, and more robust. Switched to using UnivariateSpline from scipy instead of BSpline. This allows for more control over smoothness of the spline via the smooth parameter. References to BSpline has been removed from the functions and a new functionality has been added. The new spline generation functions are GeoSpline, make_spline, spline_linestring, smooth_linestring, spline_curvature, and line_curvature. The smooth_linestring function now returns a LineString instead of a Spline object. This function is intended for smoothing a LineString when curvature, radius of curvature, and tangent angles are not needed. The spline_linestring function now returns a Spline object that contains the smoothed LineString and curvature, radius of curvature, and tangent angles. Also, line_curvature function can be used to compute curvature, radius of curvature, and tangent angles of a LineString at all point along the LineString.

New Features

• Add a new function called gtiff2vrt for creating a VRT file from a list of GeoTiff files. Note that this new function requires gdal to be installed.

• The xd_write_crs function now keeps spatial_ref attribute of the input xarray.DataArray or xarray.Dataset to retain CF compliance.

0.15.2 (2023-09-22)

New Features

• Add geometry_reproject function for reprojecting a geometry (bounding box, list of coordinates, or any shapely.geometry) to a new CRS.

• Add smooth_linestring function for smoothing a LineString using B-splines.

• Make make_bspline and bspline_curvature functions public. The make_bspline function uses scipy to generate a BSplines object and the bspline_curvature function calculates the tangent angles, curvature, and radius of curvature of a B-spline at any point along the B-spline.

• Improve the accuracy and performance of B-spline generation functions.

Internal Changes

• Remove dependency on dask.

0.15.0 (2023-05-07)

From release 0.15 onward, all minor versions of HyRiver packages will be pinned. This ensures that previous minor versions of HyRiver packages cannot be installed with later minor releases. For example, if you have py3dep==0.14.x installed, you cannot install pydaymet==0.15.x. This is to ensure that the API is consistent across all minor versions.

New Features

• For now, retain compatibility with shapely<2 while supporting shapley>=2.

0.14.0 (2023-03-05)

New Features

• Ignore index when concatenating multiple responses in json2geodf to ensure indices are unique

• Add a new function, called coords_list, for converting/validating input coordinates of any type to a list of tuple, i.e., [(x1, y1), (x2, y2), ...].

• Make xd_write_crs function public.

• In xarray_geomask if the input geometry is very small return at least one pixel.

• Add a new function, called multi2poly, for converting a MultiPolygon to a Polygon in a GeoDataFrame. This function tries to convert MultiPolygon to Polygon by first checking if MultiPolygon can be directly converted using their exterior boundaries. If not, will try to remove those small sub-Polygon that their area is less than 1% of the total area of the MultiPolygon. If this fails, the original MultiPolygon will be returned.

Breaking Changes

• Bump the minimum required version of shapely to 2.0, and use its new API.

Internal Changes

• Sync all minor versions of HyRiver packages to 0.14.0.

0.13.12 (2023-02-10)

Breaking Changes

• The input GeoDataFrame to break_lines now should be in a projected CRS.

New Features

• Significant improvements in the accuracy and performance of nested_``Polygon`` by changing the logic. Now, the function first determines the nested Polygon by comparing the centroids of the geometries with their geometry and then picks the largest geometry from each group of nested geometries.

• Add a new function called query_indicies which is a wrapper around geopandas.sindex.query_bulk. However, instead of returning an array of positional indices, it returns a dictionary of indices where keys are the indices of the input geometry and values are a list of indices of the tree geometries that intersect with the input geometry.

Internal Changes

• Simplify geo2polygon by making the two CRS arguments optional and only reproject if CRS values are given and different.

• Apply the geometry mask in gtiff2xarray even if the input geometry is a bounding box since the mask might not be the same geometry as the one that was used during data query.

• Fully migrate setup.cfg and setup.py to pyproject.toml.

• Convert relative imports to absolute with absolufy-imports.

• Sync all patch versions of HyRiver packages to x.x.12.

0.13.11 (2023-01-08)

Bug Fixes

• Fix an in issue xarray_geomask where for geometries that are smaller than a single pixel, the bbox clipping operation fails. This is fixed by using the auto_expand option of rioxarray.clip_box.

0.13.10 (2022-12-09)

New Features

• Add a new function called nested_``Polygon`` for determining nested (multi)``Polygon`` in a gepandas.GeoDataFrame or geopandas.GeoSeries.

• Add a new function called geodf2xarray for rasterizing a geopandas.GeoDataFrame to a xarray.DataArray.

Internal Changes

• Modify the codebase based on Refurb suggestions.

• In xarray_geomask, if drop=True recalculate its transform to ensure the correct geo references are set if the shape of the dataset changes.

0.13.8 (2022-11-04)

Internal Changes

• Improve the performance of xarray_geomask significantly by first clipping the data to the geometry’s bounding box, then if the geometry is a polygon, masking the data with the polygon. This is much faster than directly masking the data with the polygon. Also, support passing a bounding box to xarray_geomask in addition to polygon and MultiPolygon.

• Fix deprecation warning of pandas when changing the geometry column of a GeoDataFrame in then break_lines function.

0.13.7 (2022-11-04)

Internal Changes

• When combining the responses, now dask handles data chunking more efficiently. This is especially important for handling large responses from WMS services.

• Improve type hints for CRS-related arguments of all functions by including string, integer, and pyproj.CRS types.

• In gtiff2xarray use rasterio engine to make sure all rioxarray attrs are read.

0.13.6 (2022-08-30)

Internal Changes

• Add the missing PyPi classifiers for the supported Python versions.

0.13.5 (2022-08-29)

Breaking Changes

• Append “Error” to all exception classes for conforming to PEP-8 naming conventions.

0.13.2 (2022-06-14)

Breaking Changes

• Set the minimum supported version of Python to 3.8 since many of the dependencies such as xarray, pandas, rioxarray have dropped support for Python 3.7.

• Bump min versions of rioxarray to 0.10 since it adds reading/writing GCPs.

Internal Changes

• Use micromamba for running tests and use nox for linting in CI.

0.13.1 (2022-06-11)

New Features

• Add support for passing a custom bounding box in the Coordinates class. The default is the bounds of EPSG:4326 to retain backward compatibility. This new class parameter allows a user to check if a list of coordinates is within a custom bounding box. The bounds should be the EPSG:4326 coordinate system.

• Add a new function called geometry_list for converting a list of multi-geometries to a list of geometries.

0.13.0 (2022-03-03)

Internal Changes

• Write nodata attribute using rioxarray in geotiff2xarray since the clipping operation of rioxarray uses this value as fill value.

Bug Fixes

• In the break_lines function, convert MultiLineString into LineString since shapely.ops.substring only accepts LineString.

0.12.3 (2022-02-04)

New Features

• Add a function called break_lines for breaking lines at given points.

• Add a function called snap2nearest for snapping points to the nearest point on a line with a given tolerance. It accepts a geopandas.GeoSeries of points and a geopandas.GeoSeries or geopandas.GeoDataFrame of lines. It automatically snaps to the closest lines in the input data.

0.12.2 (2022-01-15)

New Features

• Add a new class called GeoBSpline that generates B-splines from a set of coordinates. The spline attribute of this class has five attributes: x and y coordinates, phi and radius which are curvature and radius of curvature, respectively, and distance which is the total distance of each point along the B-spline from the starting points.

• Add a new class called Coordinates that validates a set of lon/lat coordinates. It normalizes longitudes to the range [-180, 180) and has a points property that is geopandas.GeoSeries with validated coordinates. It uses spatial indexing to speed up the validation and should be able to handle large datasets efficiently.

• Make transform2tuple a public function.

Internal Changes

• The geometry and geo_crs arguments of gtiff2xarray are now optional. This is useful for cases when the input GeoTiff response is the results of a bounding box query and there is no need for a geometry mask.

• Replace the missing values after adding geometry mask via xarray_geomask by the nodatavals attribute of the input xarray.DataArray or xarray.Dataset. Therefore, the data type of the input xarray.DataArray or xarray.Dataset is conserved.

• Expose connectivity argument of rasterio.features.shapes function in xarray2geodf function.

• Move all private functions to a new module to make the main module less cluttered.

0.12.1 (2021-12-31)

Internal Changes

• Refactor arcgis2geojson for better readability and maintainability.

• In arcgis2geojson set the geometry to null if its type is not supported, such as curved polylines.

0.12.0 (2021-12-27)

Internal Changes

• Add all the missing types so mypy --strict passes.

• Bump version to 0.12.0 to match the release of pygeoogc.

0.11.7 (2021-11-09)

Internal Changes

• Use rioxarray for dealing with GeoTIFF binaries since xarray deprecated the xarray.open_rasterio function, as it’s discussed in this PR.

• Use importlib-metadata for getting the version instead of pkg_resources to decrease import time as discussed in this issue.

0.11.6 (2021-10-06)

New Features

• Add a new function, xarray2geodf, to convert a xarray.DataArray to a geopandas.GeoDataFrame.

0.11.5 (2021-06-16)

Bug Fixes

• Fix an issue with gtiff2xarray where the scales and offsets attributes of the output DataArray were floats rather than tuples (GH 30).

Internal Changes

• Add a new function, transform2tuple, for converting Affine transforms to a tuple. Previously, the Affine transform was converted to a tuple using to_gdal() method of rasterio.Affine which was not compatible with rioxarray.

0.11.4 (2021-08-26)

Internal Changes

• Use ujson for JSON parsing instead of orjson since orjson only serializes to bytes which is not compatible with aiohttp.

• Convert the transform attribute data type from Affine to tuple since saving a data array to netcdf cannot handle the Affine type.

0.11.3 (2021-08-19)

• Fix an issue in geotiff2xarray related to saving a xarray object to NetCDF when its transform attribute has Affine type rather than a tuple.

0.11.2 (2021-07-31)

The highlight of this release is performance improvement in gtiff2xarray for handling large responses.

New Features

• Automatic detection of the driver by default in gtiff2xarray as opposed to it being GTiff.

Internal Changes

• Make geo2polygon, get_transform, and get_nodata_crs public functions since other packages use it.

• Make xarray_mask a public function and simplify gtiff2xarray.

• Remove MatchCRS since it’s already available in pygeoogc.

• Validate input geometry in geo2polygon.

• Refactor gtiff2xarray to check for the ds_dims outside the main loops to improve the performance. Also, the function tries to detect the dimension names automatically if ds_dims is not provided by the user, explicitly.

• Improve performance of json2geodf by using list comprehension and performing checks outside the main loop.

Bug Fixes

• Add the missing arguments for masking the data in gtiff2xarray.

0.11.1 (2021-06-19)

Bug Fixes

• In some edge cases the y-coordinates of a response might not be monotonically sorted so dask fails. This release sorts them to address this issue.

0.11.0 (2021-06-19)

New Features

• Function gtiff2xarray returns a parallelized xarray.Dataset or xarray.DataAraay that can handle large responses much more efficiently. This is achieved using dask.

Breaking Changes

• Drop support for Python 3.6 since many of the dependencies such as xarray and pandas have done so.

• Refactor MatchCRS. Now, it should be instantiated by providing the in and out CRSs like so: MatchCRS(in_crs, out_crs). Then its methods, namely, geometry, bounds and coords, can be called. These methods now have only one input, geometry.

• Change input and output types of MatchCRS.coords from tuple of lists of coordinates to list of (x, y) coordinates.

• Remove xarray_mask and gtiff2file since rioxarray is more general and suitable.

Internal Changes

• Remove unnecessary type checks for private functions.

• Refactor json2geodf to improve robustness. Use get method of dict for checking key availability.

0.10.1 (2021-03-27)

• Setting transform of the merged dataset explicitly (GH 3).

• Add announcement regarding the new name for the software stack, HyRiver.

• Improve pip installation and release workflow.

0.10.0 (2021-03-06)

• The first release after renaming hydrodata to PyGeoHydro.

• Address GH 1 by sorting y coordinate after merge.

• Make mypy checks more strict and fix all the errors and prevent possible bugs.

• Speed up CI testing by using mamba and caching.

0.9.0 (2021-02-14)

• Bump version to the same version as PyGeoHydro.

• Add gtiff2file for saving raster responses as geotiff file(s).

• Fix an error in _get_nodata_crs for handling no data value when its value in the source is None.

• Fix the warning during the GeoDataFrame generation in json2geodf when there is no geometry column in the input JSON.

0.2.0 (2020-12-06)

• Added checking the validity of input arguments in gtiff2xarray function and provide useful messages for debugging.

• Add support for MultiPolygon.

• Remove the fill_hole argument.

• Fixed a bug in xarray_geomask for getting the transform.

0.1.10 (2020-08-18)

• Fixed the gtiff2xarray issue with high resolution requests and improved robustness of the function.

• Replaced simplejson with orjson to speed up JSON operations.

0.1.9 (2020-08-11)

• Modified griff2xarray to reflect the latest changes in pygeoogc 0.1.7.

0.1.8 (2020-08-03)

• Retained the compatibility with xarray 0.15 by removing the attrs flag.

• Added xarray_geomask function and made it a public function.

• More efficient handling of large GeoTiff responses by cropping the response before converting it into a dataset.

• Added a new function called geo2polygon for converting and transforming a polygon or bounding box into a Shapely’s Polygon in the target CRS.

0.1.6 (2020-07-23)

• Fixed the issue with flipped mask in WMS.

• Removed drop_duplicates since it may cause issues in some instances.

0.1.4 (2020-07-22)

• Refactor griff2xarray and added support for WMS 1.3.0 and WFS 2.0.0.

• Add MatchCRS class.

• Remove dependency on PyGeoOGC.

• Increase test coverage.

0.1.3 (2020-07-21)

• Remove duplicate rows before returning the dataframe in the json2geodf function.

• Add the missing dependency

0.1.0 (2020-07-21)

• First release on PyPI.

Contributing

Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given.

You can contribute in many ways to any of the packages that are included in HyRiver project. The workflow is the same for all packages. In this page, a contribution workflow for PyGridMET is explained. You can easily adapt it to other packages by replacing pygridmet with the name of the package that you want to contribute to.

Types of Contributions

Report Bugs

Report bugs at hyriver/pygridmet#issues.

Fix Bugs

Look through the GitHub issues for bugs. Anything tagged with “bug” and “help wanted” is open to whoever wants to implement it.

Implement Features

Other than new features that you might have in mind, you can look through the GitHub issues for features. Anything tagged with “enhancement” and “help wanted” is open to whoever wants to implement it.

Write Documentation

PyGridMET could always use more documentation, whether as part of the official PyGridMET docs, in docstrings, or even on the web in blog posts, articles, and such.

Submit Feedback

The best way to send feedback is to file an issue at hyriver/pygridmet#issues.

If you are proposing a feature:

• Explain in detail how it would work.

• Keep the scope as narrow as possible, to make it easier to implement.

• Remember that this is a volunteer-driven project, and that contributions are welcome :)

Get Started!

Ready to contribute? Here’s how to set up pygridmet for local development.

1. Fork the PyGridMET repo through the GitHub website.

2. Clone your fork locally and add the main pygridmet as the upstream remote:

$ git clone git@github.com:your_name_here/pygridmet.git
$ git remote add upstream git@github.com:hyriver/pygridmet.git

3. Install your local copy into a virtualenv. Assuming you have mamba installed, this is how you can set up your fork for local development:

$ cd pygridmet/
$ mamba env create -f ci/requirements/environment-dev.yml
$ mamba activate pygridmet-dev
$ python -m pip install . --no-deps

4. Create a branch for local development:

$ git checkout -b bugfix-or-feature/name-of-your-bugfix-or-feature
$ git push

5. Now you can make your changes locally, make sure to add a description of the changes to HISTORY.rst file and add extra tests, if applicable, to tests folder. Also, make sure to give yourself credit by adding your name at the end of the item(s) that you add in the history like this By `Taher Chegini <https://github.com/hyriver>`_. Then, fetch the latest updates from the remote and resolve any merge conflicts:

$ git fetch upstream
$ git merge upstream/name-of-your-branch

6. Then create a new environment for linting and another for testing:

 $ mamba create -n py11 python=3.11 nox tomli pre-commit codespell gdal
 $ mamba activate py11
 $ nox -s pre-commit
 $ nox -s type-check

 $ mamba create -n py38 python=3.8 nox tomli pre-commit codespell gdal
 $ mamba activate py38
 $ nox -s tests

Note that if Python 3.11 is already installed on your system, you can
skip creating the ``py11`` environment and just use your system's Python 3.11
to run the linting and type-checking tests, like this:

$ mamba create -n py38 python=3.8 nox tomli pre-commit codespell gdal
$ mamba activate py38
$ nox

7. If you are making breaking changes make sure to reflect them in the documentation, README.rst, and tests if necessary.

8. Commit your changes and push your branch to GitHub. Start the commit message with ENH:, BUG:, DOC: to indicate whether the commit is a new feature, documentation related, or a bug fix. For example:

$ git add .
$ git commit -m "ENH: A detailed description of your changes."
$ git push origin name-of-your-branch

9. Submit a pull request through the GitHub website.

Tips

To run a subset of tests:

$ nox -s tests -- -n=1 -k "test_name1 or test_name2"

Deploying

A reminder for the maintainers on how to deploy. Make sure all your changes are committed (including an entry in HISTORY.rst). Then run:

$ git tag -a vX.X.X -m "vX.X.X"
$ git push --follow-tags

where X.X.X is the version number following the semantic versioning spec i.e., MAJOR.MINOR.PATCH. Then release the tag from Github and Github Actions will deploy it to PyPi.

Development Team

Lead

• Taher Cheginil

Contributors

• Fernando Aristizabal

• Luc Rébillout

• Tim Cera

License

MIT License

Copyright (c) 2020, Taher Chegini

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.