Contextractor Apify Actor
Contextractor is distributed as an Apify Actor for extracting content from multiple URLs, crawling entire websites, and running on a schedule. Results are stored in Apify datasets and key-value stores, accessible via API.
Quick start
Open the Actor page on Apify, add your start URLs, and click Start.
From the command line:
apify call glueo/contextractor \
--input='{"startUrls": [{"url": "https://example.com"}]}'
Or via the Apify API:
curl -X POST "https://api.apify.com/v2/acts/glueo~contextractor/runs" \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"startUrls": [{"url": "https://example.com"}]}'
Input reference
In the tables below, a blank Default means the parameter has no default — when omitted, the Actor derives the value at run time.
Crawl settings
| Parameter | Type | Default | Description |
|---|---|---|---|
startUrls | array | required | URLs to extract content from |
crawlerType | enum | "playwright-adaptive" | playwright-adaptive, playwright-firefox, playwright-chromium, or cheerio. Adaptive switches between browser and HTTP per page; cheerio is HTTP-only (fastest, no JS) |
renderingTypeDetectionRatio | number | 0.1 | (Adaptive only) Ratio (0–1) of pages probed for rendering-type detection |
maxRequestsPerCrawl | integer | 0 | Maximum requests to handle (0 = unlimited) |
maxCrawlDepth | integer | 0 | Maximum link depth from start URLs (0 = unlimited) |
initialConcurrency | integer | 0 | Initial parallel pages/clients (0 = Crawlee default) |
maxConcurrency | integer | 3 | Maximum parallel pages/requests (kept low by default — concurrency is the hard cap on peak memory) |
maxRequestRetries | integer | 3 | Retries for failed requests |
maxResultsPerCrawl | integer | 0 | Maximum results saved to dataset (0 = unlimited) |
Crawl filtering
| Parameter | Type | Default | Description |
|---|---|---|---|
globs | array | [] | Glob patterns matching URLs to include; each item is an object {"glob": "..."} |
exclude | array | [] | Glob patterns matching URLs to exclude; each item is an object {"glob": "..."} |
selector | string | "" | CSS selector for links to follow |
keepUrlFragment | boolean | false | Treat URLs with different fragments as different pages |
useSitemaps | boolean | false | Discover and enqueue URLs from sitemap.xml at each start URL domain root |
deduplication | enum | "standard" | minimal (Crawlee URL dedup only), standard (+ canonical URL, default), or aggressive (+ content hash) |
respectRobotsTxtFile | boolean | false | Honor robots.txt rules |
Content extraction
| Parameter | Type | Default | Description |
|---|---|---|---|
mode | enum | "balanced" | precision (less noise), balanced (default), or recall (more content) |
includeComments | boolean | true | Include reader/discussion comment sections at the bottom of articles |
includeTables | boolean | true | Include table content |
includeImages | boolean | false | Include image alt text and captions |
includeLinks | boolean | true | Include hyperlinks |
languageCode | string | "" | Filter by language code (e.g. "en"). Empty = any language |
Output destinations
| Parameter | Type | Default | Description |
|---|---|---|---|
save | array | ["markdown-kvs"] | What to save and where, as format-destination tokens. Format: txt, markdown, json, html, original (raw page HTML before extraction); destination: dataset (inline in the record) or kvs (a blob in the key-value store). List a format twice (e.g. markdown-dataset markdown-kvs) to save it to both. Saving original/html to the dataset risks out-of-memory on large pages — prefer kvs |
datasetName | string | Named dataset for results (empty = default run dataset) | |
keyValueStoreName | string | Named key-value store for content files (empty = default) | |
requestQueueName | string | Named request queue for pending URLs (empty = default) | |
storeSkippedUrls | boolean | false | Push a dataset record for each URL skipped during crawling (auditing) |
Browser settings
| Parameter | Type | Default | Description |
|---|---|---|---|
headless | boolean | true | Run browser in headless mode |
waitUntil | enum | "load" | Navigation event: load (default), domcontentloaded, networkidle, or commit |
navigationTimeoutSecs | integer | 60 | Navigation timeout in seconds |
waitForDynamicContentSecs | integer | 0 | Maximum seconds to wait for dynamic content after navigation — the crawler continues as soon as the network is idle or this timeout elapses, whichever comes first (0 = disabled); also the timeout for waitForSelector and softWaitForSelector |
waitForSelector | string | "" | CSS selector to wait for before extracting (request fails on timeout) |
softWaitForSelector | string | "" | CSS selector to wait for before extracting (continues on timeout) |
blockMedia | boolean | true | Block images, stylesheets, fonts, PDFs, and ZIPs (Chromium only; on by default to cut memory and bandwidth) |
ignoreCorsAndCsp | boolean | false | Disable CORS/CSP restrictions |
closeCookieModals | boolean | true | Automatic cookie-consent handling: Ghostery-based ad/tracker blocking, accepting page-replacing consent walls via the site's own consent manager, and removing residual consent containers before extraction |
maxScrollHeight | integer | 5000 | Max scroll height in pixels (0 = disable) |
userAgent | string | "" | Custom User-Agent string |
ignoreHttpsErrors | boolean | false | Skip SSL certificate verification |
Proxy
| Parameter | Type | Default | Description |
|---|---|---|---|
proxyConfiguration | object | Apify proxy settings (use the proxy editor in Console) | |
proxyRotation | enum | "recommended" | Rotation: recommended, per-request, until-failure |
sessionPoolName | string | Persistent shared session pool name. Sessions (IP + cookies) are saved under this key and reused across Actor runs | |
maxSessionRotations | integer | 10 | Maximum number of session rotations per request on block detection. Independent of maxRequestRetries. Set to 0 to disable |
Cookies & headers
| Parameter | Type | Default | Description |
|---|---|---|---|
initialCookies | array | Cookies pre-set on all pages (JSON array of cookie objects); stored encrypted as a secret input | |
customHttpHeaders | object | Custom HTTP headers added to all requests |
Output
Each crawled page becomes one dataset record with status: 'success'. Failed requests are stored as status: 'failed' records (with errors, retryCount, and crawledTime) after retries are exhausted; skipped URLs can be recorded as status: 'skipped' records (with a skipReason) when storeSkippedUrls is enabled.
A *-kvs token (default) writes the content as a blob to the key-value store and the dataset record references it by key and url; a *-dataset token inlines the content in the record under content. Every content node always carries an MD5 hash and a bytes length.
{
"url": "https://example.com/article",
"status": "success",
"metadata": {
"title": "Article Title",
"author": "Author Name",
"publishedAt": "2026-01-15",
"description": "Article description",
"siteName": "Example",
"languageCode": "en"
},
"crawl": {
"loadedUrl": "https://example.com/article",
"loadedTime": "2026-04-11T12:00:00.000Z",
"httpStatusCode": 200,
"depth": 1,
"referrerUrl": "https://example.com/"
},
"original": {
"hash": "f8e6bd335e04d03e1be6798c2c72349c",
"bytes": 89898
},
"markdown": {
"hash": "c485356090a92c6a45e8c1155c14d8ee",
"bytes": 4523,
"key": "markdown-141fbc787408697a5d22735982be532a.md",
"url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/<key>"
}
}
Each format saved via a save token produces its own field on the record (txt, markdown, json, html). The original node (raw page HTML before extraction) is always present with hash and bytes; its content is included only when an original-* token is in save.
Example
Crawl all blog posts from a site with glob filtering:
{
"startUrls": [{"url": "https://example.com/blog"}],
"globs": [{"glob": "https://example.com/blog/**"}],
"exclude": [{"glob": "https://example.com/blog/archive/**"}],
"maxRequestsPerCrawl": 100,
"maxCrawlDepth": 2,
"save": ["markdown-kvs", "json-kvs"],
"mode": "precision",
"deduplication": "aggressive",
"proxyConfiguration": {
"useApifyProxy": true
}
}
Where to go next
- Help hub — the index of every Contextractor help page.
- Playground — enter a URL and preview extraction results in your browser.
- npm CLI — the command-line tool, flag reference, and JSON config.
- npm library — embed extraction in your own Node.js application.
- PyPI package — extract from Python with the
contextractorPyPI wrapper.
Updated: July 5, 2026