In 2014 HarvestChoice released a new version of its data API1 under the title HarvestChoice Data Services v3.0. We currently maintain two APIs, HarvestChoice Data Services (v3.0 as of writing) and HarvestChoice Data API. The latter provides backend methods required to run two of our flagship reporting and visualization applications, MAPPR and TABLR. We envision merging these two APIs in the near future, but currently focus our efforts on building new and faster data access methods into HC Data Services.
Through transparent data services our aim is to facilitate the re-use and integration of HarvestChoice’s spatially-consistent biophysical and socio-economic indicators into 3rd-party tools and models. As of 2016-07-30 over 750 indicators are available for sub-Saharan Africa.
This R package cannot be installed locally. Instead we provide access to all methods through a RESTful API (credits to (http://github.com/jeroenooms/opencpu)[OpenCPU]). Method names are endpoints and function arguments must be converted from R syntax to simple JSON objects, foo(arg1=c("val1", "val2")) must be passed to endpoint http://harvestchoice.org/tools/hcapi/R/foo using {"arg1" : ["val1", "val2"]}.
The examples below show sample request mechanics using cURL at the command line, but the same approach will work in R using jsonlite::curl() and in STATA using procedure ??.
Forthcomming
This API implements a RESTful2 interface allowing users to query, summarize, plot, and download any subset of HarvestChoice’s 700 spatially-explicit indicators for sub-Saharan Africa. We describe here some of the methods available to developers and researchers to programmatically interact with the API.
Because the API uses standard REST protocol and HTTP verbs (GET, POST), requests may be sent through any URL transfer utility (e.g. cURL on both Windows and Linux), or using any programming language that implements HTTP requests (including statistical languages such as STATA, R, and SAS). The logic consists in calling a specific method and passing any number of required or optional arguments or parameters. In the examples below we use the command line tool cURL in Windows (available in Windows 7 and 8).
The API’s main endpoint is at http://hcapi.harvestchoice.org/ocpu/library/hcapi3. There are a couple of useful methods exposing the API documentation. These are explained here.
The API exposes a list of available methods at http://hcapi.harvestchoice.org/ocpu/library/hcapi3/info/. In turn each method is further documented at /man/{method}/html e.g. http://hcapi.harvestchoice.org/ocpu/library/hcapi3/man/getLayer/html ([#fig2]).
# List of API methods
curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 66 0 66 0 0 976 0 --:--:-- --:--:-- --:--:-- 1245
## Redirect to http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/The main data query method is hcapi() with a REST endpoint at (http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/hcapi)[/R/hcapi]. category() and describe() may be used to return indicator metadata. Other methods, like classify() (summarizing indicators across custom geographic domains) and similar() (ranking areas by similarity) are useful to conduct advanced analyses.
# Sample method documentation
curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/man/hcapi/text## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0hcapi package:hcapi3 R Documentation
##
## Query, subset, summarize, and download HarvestChoice indicators
##
## Description:
##
## Wrapper method to query, subset and/or aggregate HarvestChoice
## layers. This method may also be used to summarize classified
## variables along continuous variables, e.g. hcapi(var="AEZ16_CLAS",
## by="bmi"). Here AEZ16_CLAS is a classified (categorical) raster,
## and bmi is a continuous raster, but the request is valid. The
## dominant class of AEZ16_CLAS is returned along intervals of bmi.
## Default interval breaks are used but custom intervals may also be
## defined, e.g. hcapi(var="AEZ16_CLAS",
## by=list(bmi=c(0,5,10,15,20,25))). The dominant class of a variable
## var is defined by dominant(var). Layers may also be summarized
## over spatial points are areas (passed as WKT representations using
## argument wkt). Use the format argument to control the output
## format (see examples below).
##
## Usage:
##
## hcapi(var, iso3 = "SSA", by = NULL, wkt = NULL, format = NULL, ...)
##
## Arguments:
##
## var: character array of variable codes, passed to getLayer
##
## iso3: character array of ISO3 country or region codes, passed to
## getLayer
##
## by: character array of variable codes to summarize by, passed to
## getLayer
##
## wkt: WKT representation of a spatial object (points, multipoints,
## or polygons, multipolygons)
##
## format: output format, one of "data.table", "csv", "tif", "dta",
## "asc", "grd", "rds", else "png" to plot the rasters, or
## "hist" to plot histogram and univariate statistics
##
## ...: other optional arguments passed to getLayer, genFile, or to
## genPlot, e.g. collapse, as.class, dir, pal.
##
## Details:
##
## API call: mean body mass index and cassava yield across provinces
## and districts of Tanzania
##
## $ curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/hcapi
## \ -d '{"var":"cass_y", "iso3":"CIV", "format":"tif"}' \ -X POST -H
## 'Content-Type:application/json' -- /ocpu/tmp/x0bc1ac9bdf/R/.val
## /ocpu/tmp/x0bc1ac9bdf/stdout /ocpu/tmp/x0bc1ac9bdf/warnings
## /ocpu/tmp/x0bc1ac9bdf/source /ocpu/tmp/x0bc1ac9bdf/console
## /ocpu/tmp/x0bc1ac9bdf/info
## /ocpu/tmp/x0bc1ac9bdf/files/bmi--TZA.tfw
## /ocpu/tmp/x0bc1ac9bdf/files/bmi--TZA.tif
## /ocpu/tmp/x0bc1ac9bdf/files/bmi--TZA.tif.aux.xml
## /ocpu/tmp/x0bc1ac9bdf/files/DESCRIPTION
## /ocpu/tmp/x0bc1ac9bdf/files/README
##
## GET all generated files in a ZIP archive
##
## $ wget http://hcapi.harvestchoice.org/ocpu/tmp/x0bc1ac9bdf/zip
##
## API call: the method may be expanded to summarize classified
## (discrete) variables along continuous variables. For example the
## call below returns the dominant agro-ecological zone and average
## stunting in children under 5 over Ethiopia's provinces and generic
## elevation zones
##
## $ curl
## http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/hcapi/json \
## -d '{"var":["AEZ8_CLAS","stunted_moderate"], "iso3":"ETH",
## "by":["ADM1_NAME_ALT","ELEVATION"]}' \ -X POST -H
## 'Content-Type:application/json'
##
## Value:
##
## a data.table (or other formats) of var indicators summarized by by
## domains
##
## See Also:
##
## getLayer getLayerWKT genFile
##
## Examples:
##
## # Mean body mass index and cassava yield across provinces and districts of Tanzania
## x <- hcapi(c("bmi", "cass_y"), iso3="TZA", by=c("ADM1_NAME_ALT", "ADM2_NAME_ALT"))
## x
##
## # Plot results for Mara province
## require(lattice)
## barchart(ADM2_NAME_ALT~bmi, data=x[ADM1_NAME_ALT=="Mara"], col="grey90")
##
## # Mean cassava yield in Ivory Coast in GeoTIFF raster format
## x <- hcapi("cass_y", iso3="CIV", format="tif")
## x
##
## # Plot the generated TIF raster (one band only)
## require(raster)
## plot(raster(x[2]))
##
## # The method may be expanded to summarize classified (discrete) variables along continuous
## # variables. For example the call below returns the dominant agro-ecological zone and
## # average stunting in children under 5 over E
100 4606 0 4606 0 0 65932 0 --:--:-- --:--:-- --:--:-- 109k
## thiopia's provinces by elevation class
## x <- hcapi(c("AEZ8_CLAS", "stunted_moderate"), iso3="ETH", by=c("ADM1_NAME_ALT", "ELEVATION"))
## x
##
## # Mean harvested maize area summarized across a custom polygon
## x <- hcapi(var="maiz_h", wkt="POLYGON((-16.35819663578485006 15.36599264077935345,
## -15.42501860768386379 15.69472580976947462, -15.11749403024149174 14.83577785208561117,
## -16.13550642453347805 14.68731771125136376, -16.35819663578485006 15.36599264077935345))")
## x
## The documentation for each method provides a short description and a list of parameters and default parameter values, as well as an example. getLayer() is one of the API’s main methods allowing users to query and summarize indicators. The documentation above shows that getLayer() takes at minimum one parameter var indicating which indicator to query (e.g. var='PD05_TOT' to return 2005 population density). Other optional parameters include iso3 limiting the results to one (or multiple) countri(es) (e.g. iso3=['GHA', 'NGA'] for Ghana and Nigeria), and by summarizing the indicators over a geographic domain (e.g. 8-class agro-ecological zones by='AEZ8_CLAS').
You can test this query by manually forming the following cURL request. This returns a JSON array.
curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/hcapi/json \
-d "var='PD05_TOT'&iso3='GHA'&by='AEZ8_CLAS'" -X POST## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 40 0 3028 --:--:-- --:--:-- --:--:-- 3028
143 389 0 389 0 40 1209 124 --:--:-- --:--:-- --:--:-- 1133
## [
## {
## "ISO3": "GHA",
## "ADM0_NAME": "Ghana",
## "PD05_TOT": 26.3
## },
## {
## "ISO3": "GHA",
## "ADM0_NAME": "Ghana",
## "AEZ8_CLAS": "Humid",
## "PD05_TOT": 104.8
## },
## {
## "ISO3": "GHA",
## "ADM0_NAME": "Ghana",
## "AEZ8_CLAS": "Semi-Arid",
## "PD05_TOT": 280
## },
## {
## "ISO3": "GHA",
## "ADM0_NAME": "Ghana",
## "AEZ8_CLAS": "Sub-Humid",
## "PD05_TOT": 80.9
## }
## ]For users of STATA or R the same dataset may be returned in a native format by using the same method appending a format parameter at the end of the request, and appending /zip to the URL endpoint. This will return all generated files in a single ZIP file.
curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/hcapi \
-d "var='PD05_TOT'&iso3='GHA'&by='AEZ8_CLAS'&format='dta'" -X POST## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 53 0 1444 --:--:-- --:--:-- --:--:-- 1444
114 404 0 404 0 53 1124 147 --:--:-- --:--:-- --:--:-- 1086
114 404 0 404 0 53 1124 147 --:--:-- --:--:-- --:--:-- 1086
## /ocpu/tmp/x0151e5b896/R/.val
## /ocpu/tmp/x0151e5b896/stdout
## /ocpu/tmp/x0151e5b896/source
## /ocpu/tmp/x0151e5b896/console
## /ocpu/tmp/x0151e5b896/info
## /ocpu/tmp/x0151e5b896/files/datapackage.json
## /ocpu/tmp/x0151e5b896/files/DESCRIPTION
## /ocpu/tmp/x0151e5b896/files/hcapi-PD05_TOT-AEZ8_CLAS-gha.dta
## /ocpu/tmp/x0151e5b896/files/meta.csv
## /ocpu/tmp/x0151e5b896/files/README.md
## /ocpu/tmp/x0151e5b896/files/terms.htmlcurl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/hcapi \
-d "var='PD05_TOT'&iso3='GHA'&by='AEZ8_CLAS'&format='rda'" -X POST## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 53 0 3973 --:--:-- --:--:-- --:--:-- 3973
114 404 0 404 0 53 1189 156 --:--:-- --:--:-- --:--:-- 1076
## /ocpu/tmp/x02b6bdef83/R/.val
## /ocpu/tmp/x02b6bdef83/stdout
## /ocpu/tmp/x02b6bdef83/source
## /ocpu/tmp/x02b6bdef83/console
## /ocpu/tmp/x02b6bdef83/info
## /ocpu/tmp/x02b6bdef83/files/datapackage.json
## /ocpu/tmp/x02b6bdef83/files/DESCRIPTION
## /ocpu/tmp/x02b6bdef83/files/hcapi-PD05_TOT-AEZ8_CLAS-gha.rds
## /ocpu/tmp/x02b6bdef83/files/meta.csv
## /ocpu/tmp/x02b6bdef83/files/README.md
## /ocpu/tmp/x02b6bdef83/files/terms.htmlOther possible formats include csv, json for GeoJSON, and raster formats grd, tif and asc for R raster, GeoTiff, and ESRI ASCII raster respectively. Note that as-is these requests do not return any file, only a JSON response with (if the response is succesful) pointers to the generated files. To download all generated files, add /zip to the session endpoint.
# Generate ZIP file
curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/hcapi \
-d "var='whea_p'&iso3='GHA'&by=['ADM2_CODE_ALT','ADM2_NAME_ALT']&format='csv'" -X POST## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 73 0 5607 --:--:-- --:--:-- --:--:-- 5607
0 55 0 55 0 73 1090 1447 --:--:-- --:--:-- --:--:-- 0
## Unparsable argument: ['ADM2_CODE_ALT','ADM2_NAME_ALT']--2015-02-03 04:10:38--
Resolving hcapi.harvestchoice.org (hcapi.harvestchoice.org)... 206.190.150.74
Connecting to hcapi.harvestchoice.org (hcapi.harvestchoice.org)|206.190.150.74|:80...
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: 'PD05_TOT-AEZ8_CLAS-GHA.csv.zip'
PD05_TOT-AEZ8_CLAS-GH [ <=> ] 2.38K --.-KB/s in 0s
2015-02-03 04:10:38 (24.0 MB/s) - 'PD05_TOT-AEZ8_CLAS-GHA.csv.zip' saved [2432]All of the API methods require passing indicator codes as parameters. These codes uniquely identify an indicator and may be accessed using two of the metadata query methods category() and describe(). category() returns a list of indicators in any category matching a cat keyword. describe() returns complete metadata records for selected indicators. Sample requests are shown here. The returned Code element is the variable code to pass into other API methods.
curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/category/json \
-d "cat='climate'" -X POST## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 13 0 969 --:--:-- --:--:-- --:--:-- 969
0 70 0 70 0 13 769 142 --:--:-- --:--:-- --:--:-- 740
## unused argument (cat = "climate")
##
## In call:
## category(cat = "climate")curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/man/describe/text## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 96 0 96 0 0 2223 0 --:--:-- --:--:-- --:--:-- 3310
## No documentation for 'describe' in specified packages and libraries:
## you could try '??describe'curl http://hcapi.harvestchoice.org/ocpu/library/hcapi3/R/describe/json \
-d "cat='bmi'" -X POST## % Total % Received % Xferd Average Speed Time Time Time Current
## Dload Upload Total Spent Left Speed
##
0 0 0 0 0 9 0 364 --:--:-- --:--:-- --:--:-- 364
121 112 0 112 0 9 1802 144 --:--:-- --:--:-- --:--:-- 2710
121 112 0 112 0 9 1797 144 --:--:-- --:--:-- --:--:-- 2710
## object 'describe' not found
##
## In call:
## get(reqobject, paste("package", reqpackage, sep = ":"), inherits = FALSE)Indicator metadata is only available in JSON format. However it is possible to download all metadata records in CSV format using http://hcapi.harvestchoice.org/ocpu/library/hcapi3/data/vi/csv (vi in the URL stands for Variable Inventory).
Application Programming Interface, see http://en.wikipedia.org/wiki/Application_programming_interface↩
Representational State Transfer, see http://en.wikipedia.org/wiki/Representational_state_transfer↩