Universal Content Scraper

📌 How can you extract structured content from multiple public websites with one crawler?

This template collects structured page content and page metadata from user-supplied public websites, subject to the configured crawl depth, page limit, and exclusion rules. It is useful for content researchers, knowledge-base teams, and data operations teams.

Data is collected from User-supplied public websites. This template is not tied to one source website. It starts from the public website URLs supplied by the user and extracts page content within the configured crawl scope.

💰 Pricing

This template is currently free of charge and has no per-line usage fee. Octoparse plan or resource limits may still apply.

📦 Output

The current published implementation can return the following fields:

• msg_type
• content
• url
• title
• author
• published_at
• format
• error_message

{
  "msg_type": null,
  "content": "{\"text\": \"Eighty feet below the streets of Manhattan...\"}",
  "url": "https://www.bloomberg.com/opinion/articles/...",
  "title": "Why Is Germany Sitting on $599 Billion of Gold?",
  "author": "Chris Bryant",
  "published_at": "2026-01-29",
  "format": "json",
  "error_message": null
}

🎯 Use Cases

• Create content archives using page URL, title, author, and publication time.
• Prepare website content for downstream analysis using the selected format.
• Crawl bounded site sections using depth, page-limit, and exclusion controls.
• Audit inaccessible pages using URL and error-message fields.

🐙 Why Octoparse

• Ready-to-use workflow: The extraction steps for User-supplied public websites are already configured, so you do not need to build the scraper from scratch.
• Flexible execution: Run the ready-made task in the cloud without keeping a local computer online for the collection.
• Structured, repeatable output: Results are returned as consistent rows that are easier to compare, filter, deduplicate, and process than manually copied pages.
• Verified input guardrails: The form exposes the current inputs, selectable values, and meaningful limits configured for this template.
• Practical data handoff: Review results in Octoparse and export or process them using the options supported by your Octoparse environment.

📝 Input

Complete the following fields:

• Website URL (Required) — Enter the public website URL or URLs where crawling should begin. Up to 10,000 entries per run.
• Maximum Crawl Depth (Optional) — Set how many link levels the crawler may follow from each starting URL. Leave it blank to use a crawl depth of 0. Accepted range: 0 to 5.
• Maximum Pages per URL (Optional) — Set the maximum number of pages to collect for each starting URL. Leave it blank to use 10 pages. Accepted range: 1 to 1000.
• Format (Optional)
• Exclude_Glob (Optional) — Enter a glob pattern for URLs or paths that should be excluded from the crawl.
• Output_Field_Name (Optional)

🚀 How to Use

1. Open the template and select Try it or Start.
2. Complete the input fields listed above.
3. Start the task using the supported cloud run mode.
4. Review the output rows and export or process the structured data.

⚠️ Limitations

Results depend on what the source site exposes at run time. A listed output field can be empty when the source page does not provide that value. Input limits shown above are enforced by the template.

💡 Tips

Use specific, valid inputs and review a representative result before starting a large batch. Remove duplicate inputs when repeated records are not needed.

❓ FAQ

How does Octoparse collect data from User-supplied public websites?

The Python automation starts from each submitted website URL, follows eligible links within the configured depth and page limit, excludes links matching the supplied glob, transforms page content into the selected format, and uploads one record per processed page.

What input does this Universal Content Scraper require?

Use the fields and accepted values shown in the Input section. Only user-relevant limits and selectable options are listed.

What data can I extract from User-supplied public websites?

The current output fields and a representative JSON Data Preview are listed in the Output section. Field availability can vary when the source page does not display a value.

Is this Universal Content Scraper free to use?

It is currently free of charge and has no per-line usage fee. Octoparse plan or resource limits may still apply.

Why can some User-supplied public websites fields be empty?

Source pages do not always expose every value for every record, and layouts can vary by item, market, or current site response. The template returns a field when the current page provides it.

Can I export the collected User-supplied public websites data?

You can review the structured rows in Octoparse and export or process them using the options supported by your Octoparse environment.

🔗 Related Templates

• AI Crawl — Use AI-guided scope and filters when a website requires broader multi-page crawling.
• AI Scraper — Extract custom fields when the exact target page URLs are already known.
• Google Search Scraper — Use Google search result URLs to identify public websites or pages for content extraction.