An API key is a required parameter that may be passed in the header or body of your request. You can read more about this parameter in the Authentication section of the documentation.
pageUrl should be a valid URL and is a required field. It should be passed through the body of your request as a string. Be sure you're able to load the URL in a browser to confirm it's valid and loads correctly.
The URL passed must return a HTTP Status 200 header when loaded by our service.
The content-type loaded from the URL you pass to the API must be HTML, an image, or a PDF file. If you are loading an image or PDF file, you should also pass the prop unprocessed with a value of true.
Use the unprocessed parameter to receive the raw response from a URL. The parameter is useful when loading images or PDF files. You can also receive unprocessed HTML from a URL to handle data extraction on your own. In some cases, you may also need the full HTML response for use cases, such as change monitoring.
mapping parameter specifies what elements to grab values from after fetching the HTML and also controls the formatting of the JSON response.
The keys of the request object are used as the keys in the response JSON. Response data is returned in the
data key of the response. This gives you great flexability to specify the format of your JSON response.
There are two ways to grab data from a page.
A single element on a page can easily be grabbed from a page and return by using a
"keyName": ".selectorFormat" format. The title key below is a good example of this. The title value of the page is grabbed and returned by passing
"title": "head>title" in the
Multiple Repeating Elements
There will be cases when you want to grab data that repeats. A simple example of this would be weather data. A 10-day weather forecast will contain 10 sets of high and low tempetures.
To grab this repeating data we will specify a
selector key to grab the parent container. We will then use the
foreach key to specify specific elements that repeat on the page.
Take a close look at the complete example below. You can see how the title of the page is grabbed by using the single selector format. You can also see how the forcast key returns multiple values from the page by using
selector to specify the parent container along with
foreach to loop over repeating values found within the parent container such as the high and low tempetures of a 10-day forcast.
mapping Request Object
Example Response JSON
We recommend creating a basic
mapping by first specifiying your structure and key names. You'll need to identify what data will have a single value and what data repeats. Based on this you'll use one of the two selector formats found above. Keep in mind
foreach keys are reserved within the
mapping parameter. This means you should never try to name a key in your response using either of those proteced keywords.
Once you've created your basic object, you'll want to work on your selectors.
Anything API uses a powerful selector system that closely parallels jQuery selectors. This allows you to grab data from a page in many different ways. You may be surprised by how creative you can get with selectors to grab the data you need.
countryCode can load localized data from one of twelve countries. This can be helpful when trying to pull localized shipping data, prices, regional media, translations, etc. Keep in mind a request to a server in another country will add network latency due to the extra travel time between servers. This should not be substantial, but responses will take longer to return.
The default country code is
us if the parameter is not provided. To pull data from another country, use one of these twelve country code values.
- United States
- United Kingdom
Easily reuse the same session for multiple requests with the
sessionID param. The value can be any integer between 1 and 9,999. Sending the same integer within 60 seconds will recycle your session. Sending a new number will create a new session.