DataWeb API User Guide

The USITC DataWeb API provides secure programming access to U.S. international trade data. The API enables users to efficiently retrieve and analyze import and export trade statistics for research and analysis.

The DataWeb API is most effective when used as an extension of the USITC DataWeb platform.

DataWeb API allows users with API credentials to obtain and send queries directly to the API. A registered USITC DataWeb account and a valid API key are required to access the API. API keys are valid for a duration of six months, do not automatically refresh, and must be regenerated after the previous key's expiration date.

Prerequisites

• Python 3.x with pandas and requests libraries installed. The DataWeb API is tool-agnostic, so any HTTP-capable tools will be compatible. However, the examples below will use Python.
• An active DataWeb account with API key. Keys must be regenerated after six months.
  • API key can be generated after registering an account and signing in.
• Optional - Jupyter notebook

User Guide Outline

This user guide demonstrates two methods for pulling data through the API. We highly recommend using the first method whenever possible. The second method is more complex than the first because users need to know how to navigate a web browser's DevTools. There is a table of contents on the left side of this page for ease of navigation.

1. Retrieving a Saved Query
2. Copying a Query from DataWeb

Set Up

1. Libraries

These are libraries that are used in the examples below.

import pandas as pd      
import requests          
import json

2. API token and request settings

To obtain a DataWeb API key, users must create a free account at https://dataweb.usitc.gov/sign-in. The instructions to link a user's Login.gov account can be found at https://www.usitc.gov/dataweb_login_process.

The screenshot below shows where users can copy their DataWeb API token. After signing in, select the “API” tab in the top left corner next to “Sign Out”, then select “Generate Token” to obtain the API token. The code block after the screenshot contains the API request settings.

token = 'Copy and paste token from the API tab in DataWeb here' 
baseUrl = 'https://datawebws.usitc.gov/dataweb'
headers = {
    "Content-Type": "application/json; charset=utf-8",
    "Authorization": "Bearer " + token
}

requests.packages.urllib3.disable_warnings()

Retrieving a Saved Query

The first method for pulling data from the DataWeb API is to retrieve a saved query. This method ensures that a user's retrieved data exactly matches what they see on DataWeb. The example below demonstrates how to create a saved query and retrieve the data from the saved query.

Creating a Saved Query

Let's say we are interested in the imports of frozen dungeness crab (HTS code: 0306.14.4030). We want the annual consumption value and the first unit of quantity from 2020 to 2024.

• Go to https://dataweb.usitc.gov/trade
• Sign in on the top right
• Select "New Request"
• Follow the screenshot and instructions below to create a saved query.

Step 1: Trade Flow and Classification System

• Imports: For Consumption
• Classification System: HTS Items

Step 2: Data and Years

• Summable Measure: Customs Value, First Unit of Quantity
  • First selected summable measure will be the first measure that a user can call in the function printQueryResults().
• Value Multiple: Actual
• Timeframe Option: Full Years
• Timeframe Aggregation: Annual
• Full Years: 2020, 2021, 2023, 2024

Step 3: Countries

• Choose Countries: Use All Countries
• Aggregate All Countries Together

Step 4: Commodities

• Select Individual Commodities
• Enter Specific Commodity Code
• Enter in frozen dungeness crab HTS code: 0306.14.4030
• Click "Add" to add the specific commodity code
• Commodity Aggregation: Display Commodities Separately
• Aggregation Level: HTS-10
• Description Display Format: Display Commodity Description
• Check Show Detail

Step 5 through 10

• No changes to options for the simplicity of this example.

Step 11: Query Options

• Name the query under "Save Query Name," in this example it is named "CrabExample"
• Give the query a description
• Select "Save a Copy"

Getting All Saved Queries

The DataWeb API provides a GET request to retrieve all saved queries associated with a user's DataWeb account. More information about the DataWeb API requests can be found on the USITC DataWeb API Swagger.

1. Fetch all queries

This first step calls all the saved queries associated with the user's account.

savedQuery_res = requests.get(baseUrl+"/api/v2/savedQuery/getAllSavedQueries",
                        headers=headers, verify=False)
all_queries = savedQuery_res.json().get('list', [])

2. Find the specific saved query by name

In the code below, the saved query name to search for is "CrabExample."

query_name_to_find = "CrabExample"
CrabExample_res = next((q for q in all_queries if q['savedQueryName'] == query_name_to_find), None)

3. Load the helper functions

These three helper functions getColumns(), getData(), and printQueryResults() are used to parse the JSON response from the API query and return the data as a pandas dataframe.

• getColumns() obtains the column headers from the JSON response.
  • This function will be used inside printQueryResults().
  • Argument: columnGroups is the part of the JSON response that contains the column headers as a collection of objects. The code inside printQueryResults() specifies the collection to call from the JSON response that contains the column headers. The location of the column headers in the JSON response is consistent throughout all DataWeb API calls.

def getColumns(columnGroups, prevCols = None):
    if prevCols is None:
        columns = []
    else:
        columns = prevCols
    for group in columnGroups:
        if isinstance(group, dict) and 'columns' in group.keys():
            getColumns(group['columns'], columns)
        elif isinstance(group, dict) and 'label' in group.keys():
            columns.append(group['label'])
        elif isinstance(group, list):
            getColumns(group, columns)
    return columns

• getData() obtains the data from the JSON response.
  • This function will be used inside printQueryResults().
  • Argument: dataGroups is the part of the JSON response that contains the data as a collection of objects. The code inside printQueryResults() specifies the collection to call from the JSON response that contains the data. The location of the data are consistent throughout all DataWeb API calls.

def getData(dataGroups):
    data = []
    for row in dataGroups:
        rowData = []
        for field in row['rowEntries']:
            rowData.append(field['value'])
        data.append(rowData)
    return data

• printQueryResults() makes the request directly to the API and uses the previous two functions to return a dataframe.
  • Argument 1: headers is defined above in the set up section. It contains the content type, authorization, and DataWeb user API token.
  • Argument 2: requestData contains the JSON request for the data.
  • Argument 3: measure_num calls a specific summable measure as an indexed number, such as "First Unit of Quantity" or "Landed Duty-Paid Value." Default is the first summable measure from the request. The order of the summable measures follows the order that a user selected in the query.

def printQueryResults(headers, requestData, measure_num=0): 
    response = requests.post(baseUrl+"/api/v2/report2/runReport",
                            headers=headers, json=requestData, verify=False)

    columns = getColumns(response.json()['dto']['tables'][measure_num]['column_groups'])

    data = getData(response.json()['dto']['tables'][measure_num]['row_groups'][0]['rowsNew'])

    df = pd.DataFrame(data, columns = columns)

    return df

4. Return the data

Let's print the data for the first selected summable measure which was "Customs Value."

printQueryResults(headers, CrabExample_res, measure_num=0).head(10)

	HTS Number	Description	Quantity Description	2020	2021	2022	2023	2024
0	0306144030	DUNGENESS CRABS, FROZEN, EXCEPT CRABMEAT	Value for: kilograms	1,269,699	1,407,083	945,458	925,827	444,120

Let's print the data for the second selected summable measure which was "First Unit of Quantity."

printQueryResults(headers, CrabExample_res, measure_num=1).head(10)

	HTS Number	Description	Quantity Description	2020	2021	2022	2023	2024
0	0306144030	DUNGENESS CRABS, FROZEN, EXCEPT CRABMEAT	kilograms	76,759	74,642	104,070	66,301	26,557

The printed data matches the data on DataWeb.

Copying a Query from DataWeb

This alternative, more complex, method retrieves the same data as the first method above. We recommend using the first method whenever possible.

The steps below outline the process to get the query from Google Chrome. This process also works on other browsers but may have slightly different steps because other browsers may have different configurations.

The example below uses the same crab scenario as above.

Obtaining a Query from a Web Browser's DevTools

1. Build the query on DataWeb.
2. Right click anywhere on the screen to access the web browser's DevTools.
3. Select "Inspect." In Chrome, the "Inspect" window will open a tab on the right side of the screen (as seen in the screenshots). The location of the window may differ when using other web browsers.
4. Select the "Network" tab, which is at the top in the Developer tools window.

5. Run the DataWeb Query using the "View Results" button at the bottom of the query page, which is on the DataWeb side of the browser (left side).
6. After clicking "View Results," a list of items will appear in the Network tab. Click the item named "runReport" that has "xhr" in the "Type" column.
7. Click on "Payload" in Chrome ("Request" in Firefox or "Payload" in Edge). This will display the request payload for the API request. See screenshot below on where to find "Payload."

Loading the Copied Query

1. Copy the JSON/JavaScript in the request payload by right clicking anywhere on it and selecting "Copy."
2. Paste the API request in Python inside the function json.dumps() and then json.loads().
3. Change "true" to "True," "false" to "False," and "null" to "None" in the API request. These items need to be changed from JSON/JavaScript booleans to Python booleans.

CrabExample_InspectQuery = json.dumps({
    "savedQueryType": "",
    "savedQueryID": "cb9b0588-76d6-4c60-82a1-2cce457af050",
    "savedQueryName": "CrabExample",
    "savedQueryDesc": "",
    "isOwner": True,
    "runMonthly": False,
    "unitConversion": "0",
    "manualConversions": [],
    "reportOptions": {
        "tradeType": "Import",
        "classificationSystem": "HTS"
    },
    "searchOptions": {
        "MiscGroup": {
            "districts": {
                "aggregation": "Aggregate District",
                "districtGroups": {
                    "userGroups": []
                },
                "districts": [],
                "districtsExpanded": [],
                "districtsSelectType": "all"
            },
            "importPrograms": {
                "aggregation": None,
                "importPrograms": [],
                "programsSelectType": "all"
            },
            "extImportPrograms": {
                "aggregation": "Aggregate CSC",
                "extImportPrograms": [],
                "extImportProgramsExpanded": [],
                "programsSelectType": "all"
            },
            "provisionCodes": {
                "aggregation": "Aggregate RPCODE",
                "provisionCodesSelectType": "all",
                "rateProvisionCodes": [],
                "rateProvisionCodesExpanded": [],
                "rateProvisionGroups": {
                    "systemGroups": []
                }
            }
        },
        "commodities": {
            "aggregation": "Break Out Commodities",
            "codeDisplayFormat": "YES",
            "commodities": [
                "0306144030"
            ],
            "commoditiesExpanded": [
                {
                    "name": "DUNGENESS CRABS, FROZEN, EXCEPT CRABMEAT",
                    "value": "0306144030"
                }
            ],
            "commoditiesManual": "0306144030",
            "commodityGroups": {
                "systemGroups": [],
                "userGroups": []
            },
            "commoditySelectType": "list",
            "granularity": "10",
            "groupGranularity": None,
            "searchGranularity": None,
            "showHTSValidDetails": True
        },
        "componentSettings": {
            "dataToReport": [
                "CONS_CUSTOMS_VALUE",
                "CONS_FIR_UNIT_QUANT"
            ],
            "scale": "1",
            "timeframeSelectType": "fullYears",
            "years": [
                "2024",
                "2023",
                "2022",
                "2021",
                "2020"
            ],
            "startDate": None,
            "endDate": None,
            "startMonth": None,
            "endMonth": None,
            "yearsTimeline": "Annual"
        },
        "countries": {
            "aggregation": "Aggregate Countries",
            "countries": [],
            "countriesExpanded": [],
            "countriesSelectType": "all",
            "countryGroups": {
                "systemGroups": [],
                "userGroups": []
            }
        }
    },
    "sortingAndDataFormat": {
        "DataSort": {
            "columnOrder": [
                "HTS10 & DESCRIPTION"
            ],
            "fullColumnOrder": [
                {
                    "checked": False,
                    "disabled": False,
                    "hasChildren": False,
                    "name": "HTS10 & DESCRIPTION",
                    "value": "HTS10 & DESCRIPTION",
                    "classificationSystem": "",
                    "groupUUID": "",
                    "items": [],
                    "tradeType": ""
                }
            ],
            "sortOrder": [
                {
                    "sortData": "HTS10 & DESCRIPTION",
                    "orderBy": "asc",
                    "year": "0"
                }
            ]
        },
        "reportCustomizations": {
            "exportCombineTables": False,
            "totalRecords": "20000",
            "exportRawData": False
        }
    },
    "deletedCountryUserGroups": [],
    "deletedCommodityUserGroups": [],
    "deletedDistrictUserGroups": []
})

CrabExample_InspectQuery = json.loads(CrabExample_InspectQuery)

4. Send the request, and check the response. The printed data should match the data on DataWeb.

printQueryResults(headers, CrabExample_InspectQuery)

	HTS Number	Description	Quantity Description	2020	2021	2022	2023	2024
0	0306144030	DUNGENESS CRABS, FROZEN, EXCEPT CRABMEAT	Value for: kilograms	1,269,699	1,407,083	945,458	925,827	444,120

About the DataWeb Swagger Documentation

The DataWeb Swagger documentation provides a complete list of all DataWeb API requests. Swagger outlines available endpoints, parameters, and sample responses, allowing users to explore, test, and integrate DataWeb queries directly into automated workflows and analytical tools. Users can test their requests on Swagger by clicking on the "Authorize" icon and pasting their token.

Link to Swagger: https://datawebws.usitc.gov/dataweb/swagger-ui/index.html

Common DataWeb API Errors and Troubleshooting

When working with the DataWeb API, users may encounter the errors listed below:

1. Authentication Error

• 503 Error: This usually means that the API token is missing, expired, or incorrect. Double-check that the token is valid.

2. Invalid Request Error

• 404 Not Found: The API request does not match the expected schema on the DataWeb server. It may be possible that the saved query does not exist or is mistyped. It is also possible that the API request was edited without changing all the necessary schema to obtain the data of interest. To avoid these pitfalls, we recommend following the methods above to retrieve a saved query or copy a query from DataWeb.

Note: 200 response means the request is good.

DataWeb Help Form

For any questions about the DataWeb API, please contact us through the DataWeb help form:

https://www.usitc.gov/applications/dataweb/dataweb_5_help