Skip to main content

Contextractor Apify Actor

Contextractor is distributed as an Apify Actor for extracting content from multiple URLs, crawling entire websites, and running on a schedule. Results are stored in Apify datasets and key-value stores, accessible via API.

Quick start

Open the Actor page on Apify, add your start URLs, and click Start.

From the command line:

apify call glueo/contextractor \
  --input='{"startUrls": [{"url": "https://example.com"}]}'

Or via the Apify API:

curl -X POST "https://api.apify.com/v2/acts/glueo~contextractor/runs" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://example.com"}]}'

Input reference

In the tables below, a blank Default means the parameter has no default — when omitted, the Actor derives the value at run time.

Crawl settings

ParameterTypeDefaultDescription
startUrlsarrayrequiredURLs to extract content from
crawlerTypeenum"playwright-adaptive"playwright-adaptive, playwright-firefox, playwright-chromium, or cheerio. Adaptive switches between browser and HTTP per page; cheerio is HTTP-only (fastest, no JS)
renderingTypeDetectionRationumber0.1(Adaptive only) Ratio (0–1) of pages probed for rendering-type detection
maxRequestsPerCrawlinteger0Maximum requests to handle (0 = unlimited)
maxCrawlDepthinteger0Maximum link depth from start URLs (0 = unlimited)
initialConcurrencyinteger0Initial parallel pages/clients (0 = Crawlee default)
maxConcurrencyinteger3Maximum parallel pages/requests (kept low by default — concurrency is the hard cap on peak memory)
maxRequestRetriesinteger3Retries for failed requests
maxResultsPerCrawlinteger0Maximum results saved to dataset (0 = unlimited)

Crawl filtering

ParameterTypeDefaultDescription
globsarray[]Glob patterns matching URLs to include; each item is an object {"glob": "..."}
excludearray[]Glob patterns matching URLs to exclude; each item is an object {"glob": "..."}
selectorstring""CSS selector for links to follow
keepUrlFragmentbooleanfalseTreat URLs with different fragments as different pages
useSitemapsbooleanfalseDiscover and enqueue URLs from sitemap.xml at each start URL domain root
deduplicationenum"standard"minimal (Crawlee URL dedup only), standard (+ canonical URL, default), or aggressive (+ content hash)
respectRobotsTxtFilebooleanfalseHonor robots.txt rules

Content extraction

ParameterTypeDefaultDescription
modeenum"balanced"precision (less noise), balanced (default), or recall (more content)
includeCommentsbooleantrueInclude reader/discussion comment sections at the bottom of articles
includeTablesbooleantrueInclude table content
includeImagesbooleanfalseInclude image alt text and captions
includeLinksbooleantrueInclude hyperlinks
languageCodestring""Filter by language code (e.g. "en"). Empty = any language

Output destinations

ParameterTypeDefaultDescription
savearray["markdown-kvs"]What to save and where, as format-destination tokens. Format: txt, markdown, json, html, original (raw page HTML before extraction); destination: dataset (inline in the record) or kvs (a blob in the key-value store). List a format twice (e.g. markdown-dataset markdown-kvs) to save it to both. Saving original/html to the dataset risks out-of-memory on large pages — prefer kvs
datasetNamestringNamed dataset for results (empty = default run dataset)
keyValueStoreNamestringNamed key-value store for content files (empty = default)
requestQueueNamestringNamed request queue for pending URLs (empty = default)
storeSkippedUrlsbooleanfalsePush a dataset record for each URL skipped during crawling (auditing)

Browser settings

ParameterTypeDefaultDescription
headlessbooleantrueRun browser in headless mode
waitUntilenum"load"Navigation event: load (default), domcontentloaded, networkidle, or commit
navigationTimeoutSecsinteger60Navigation timeout in seconds
waitForDynamicContentSecsinteger0Maximum seconds to wait for dynamic content after navigation — the crawler continues as soon as the network is idle or this timeout elapses, whichever comes first (0 = disabled); also the timeout for waitForSelector and softWaitForSelector
waitForSelectorstring""CSS selector to wait for before extracting (request fails on timeout)
softWaitForSelectorstring""CSS selector to wait for before extracting (continues on timeout)
blockMediabooleantrueBlock images, stylesheets, fonts, PDFs, and ZIPs (Chromium only; on by default to cut memory and bandwidth)
ignoreCorsAndCspbooleanfalseDisable CORS/CSP restrictions
closeCookieModalsbooleantrueAutomatic cookie-consent handling: Ghostery-based ad/tracker blocking, accepting page-replacing consent walls via the site's own consent manager, and removing residual consent containers before extraction
maxScrollHeightinteger5000Max scroll height in pixels (0 = disable)
userAgentstring""Custom User-Agent string
ignoreHttpsErrorsbooleanfalseSkip SSL certificate verification

Proxy

ParameterTypeDefaultDescription
proxyConfigurationobjectApify proxy settings (use the proxy editor in Console)
proxyRotationenum"recommended"Rotation: recommended, per-request, until-failure
sessionPoolNamestringPersistent shared session pool name. Sessions (IP + cookies) are saved under this key and reused across Actor runs
maxSessionRotationsinteger10Maximum number of session rotations per request on block detection. Independent of maxRequestRetries. Set to 0 to disable

Cookies & headers

ParameterTypeDefaultDescription
initialCookiesarrayCookies pre-set on all pages (JSON array of cookie objects); stored encrypted as a secret input
customHttpHeadersobjectCustom HTTP headers added to all requests

Output

Each crawled page becomes one dataset record with status: 'success'. Failed requests are stored as status: 'failed' records (with errors, retryCount, and crawledTime) after retries are exhausted; skipped URLs can be recorded as status: 'skipped' records (with a skipReason) when storeSkippedUrls is enabled.

A *-kvs token (default) writes the content as a blob to the key-value store and the dataset record references it by key and url; a *-dataset token inlines the content in the record under content. Every content node always carries an MD5 hash and a bytes length.

{
  "url": "https://example.com/article",
  "status": "success",
  "metadata": {
    "title": "Article Title",
    "author": "Author Name",
    "publishedAt": "2026-01-15",
    "description": "Article description",
    "siteName": "Example",
    "languageCode": "en"
  },
  "crawl": {
    "loadedUrl": "https://example.com/article",
    "loadedTime": "2026-04-11T12:00:00.000Z",
    "httpStatusCode": 200,
    "depth": 1,
    "referrerUrl": "https://example.com/"
  },
  "original": {
    "hash": "f8e6bd335e04d03e1be6798c2c72349c",
    "bytes": 89898
  },
  "markdown": {
    "hash": "c485356090a92c6a45e8c1155c14d8ee",
    "bytes": 4523,
    "key": "markdown-141fbc787408697a5d22735982be532a.md",
    "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/<key>"
  }
}

Each format saved via a save token produces its own field on the record (txt, markdown, json, html). The original node (raw page HTML before extraction) is always present with hash and bytes; its content is included only when an original-* token is in save.

Example

Crawl all blog posts from a site with glob filtering:

{
  "startUrls": [{"url": "https://example.com/blog"}],
  "globs": [{"glob": "https://example.com/blog/**"}],
  "exclude": [{"glob": "https://example.com/blog/archive/**"}],
  "maxRequestsPerCrawl": 100,
  "maxCrawlDepth": 2,
  "save": ["markdown-kvs", "json-kvs"],
  "mode": "precision",
  "deduplication": "aggressive",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Where to go next

  • Help hub — the index of every Contextractor help page.
  • Playground — enter a URL and preview extraction results in your browser.
  • npm CLI — the command-line tool, flag reference, and JSON config.
  • npm library — embed extraction in your own Node.js application.
  • PyPI package — extract from Python with the contextractor PyPI wrapper.

Updated: July 5, 2026