Universal Content Scraper

📌 How can you extract structured content from multiple public websites with one crawler?

This template collects structured page content and page metadata from user-supplied public websites, subject to the configured crawl depth, page limit, and exclusion rules. It is useful for content researchers, knowledge-base teams, and data operations teams.

Data is collected from User-supplied public websites. This template is not tied to one source website. It starts from the public website URLs supplied by the user and extracts page content within the configured crawl scope.

💰 Pricing

This template is currently free of charge and has no per-line usage fee. Octoparse plan or resource limits may still apply.

📦 Output

Universal Content Scraper returns these fields:

• msg_type
• content
• url
• title
• author
• published_at
• format
• error_message

{
  "msg_type": null,
  "content": "{\"text\": \"Eighty feet below the streets of Manhattan...\"}",
  "url": "https://www.bloomberg.com/opinion/articles/...",
  "title": "Why Is Germany Sitting on $599 Billion of Gold?",
  "author": "Chris Bryant",
  "published_at": "2026-01-29",
  "format": "json",
  "error_message": null
}

🎯 Use Cases

• Create content archives using page URL, title, author, and publication time.
• Prepare website content for downstream analysis using the selected format.
• Crawl bounded site sections using depth, page-limit, and exclusion controls.
• Audit inaccessible pages using URL and error-message fields.

🐙 Why Octoparse

• Built for User-supplied websites: The Python automation starts from each submitted website URL, follows eligible links within the configured depth and page limit, excludes links matching the supplied glob, transforms page content into the selected format, and uploads one record per processed page.
• Inputs match the workflow: The form uses Website URL (up to 10,000 entries), Maximum Crawl Depth (0 to 5), and Maximum Pages per URL (1 to 1000).
• Fields stay connected: Universal Content Scraper returns title, url, content, msg_type, and author in the same structured dataset.
• Ready for repeat use: After Universal Content Scraper runs in the cloud, schedule eligible tasks and export the rows for spreadsheets, dashboards, APIs, or AI workflows.

📝 Input

Complete the following fields:

• Website URL (Required) — Enter the public website URL or URLs where crawling should begin. Up to 10,000 entries per run.
• Maximum Crawl Depth (Optional) — Set how many link levels the crawler may follow from each starting URL. Leave it blank to use a crawl depth of 0. Accepted range: 0 to 5.
• Maximum Pages per URL (Optional) — Set the maximum number of pages to collect for each starting URL. Leave it blank to use 10 pages. Accepted range: 1 to 1000.
• Format (Optional)
• Exclude_Glob (Optional) — Enter a glob pattern for URLs or paths that should be excluded from the crawl.
• Output_Field_Name (Optional)

🚀 How to Use

1. Open Universal Content Scraper and click Try it!.
2. Complete Website URL, Maximum Crawl Depth, and Maximum Pages per URL using the formats and limits shown in the Input section.
3. Run it in the Octoparse cloud and start the task.
4. Review fields such as title, url, and content, then export the rows in the format you need.

💡 Tips

• Test one representative value in Website URL before submitting a large batch to Universal Content Scraper.
• Use title and url when checking duplicates or comparing repeated exports.

❓ FAQ

How many entries can Universal Content Scraper accept in Website URL per run?

Enter up to 10,000 values in Website URL per run.

Which User-supplied websites pages does Universal Content Scraper process?

The workflow is configured for web pages and returns fields such as title, url, content, and msg_type.

What can I use data from Universal Content Scraper for?

Create content archives using page URL, title, author, and publication time.

🔗 Related Templates

• AI Crawl — Use AI-guided scope and filters when a website requires broader multi-page crawling.
• AI Scraper — Extract custom fields when the exact target page URLs are already known.
• Google Search Scraper — Use Google search result URLs to identify public websites or pages for content extraction.