How to Use a YouTube Scraper API with Python

To use a YouTube scraper API with Python, discover a cloud-capable Octoparse template, send its required inputs, save the returned task and batch IDs, and poll for an export. The example below resumes an existing task and can download a CSV, check required fields, and reject duplicate video URLs before you use the data.

Octoparse operates the extraction workflow; your Python client manages task state and data delivery. This tutorial uses YouTube Channel Scraper for public channel-video metadata. It does not collect comments or transcripts. Start with one channel you are authorized to process.

https://www.octoparse.com/template/youtube-channel-scraper

Code and template schema reviewed: October 15, 2026. The revised client passed local tests with simulated API responses. The August extraction evidence below is historical; this code revision was not tested in a new live extraction run.

How to Use a YouTube Scraper API with Python

The workflow has three stages: discover the exact template, create or resume a task, then retrieve and validate its export. An accepted request is not a completed dataset. The saved state file connects those stages across separate script runs.

Step 1: Prepare Your API Credentials and Channel

Use Python 3.10 or later, an Octoparse API key, a stable external user ID, and access to the cloud template. The official AgentTools workflow explains how x-api-key authenticates requests and how x-external-user-id scopes task creation and task search.

python -m pip install requests

In PowerShell, set credentials for the current terminal session. Keep the same external user ID when reconciling tasks; do not put credentials into the script or shared logs.

$env:OCTOPARSE_API_KEY="your-api-key"
$env:OCTOPARSE_EXTERNAL_USER_ID="your-stable-user-id"

The October 15 schema check confirmed template ID 1813, Cloud and Local execution, and the required youTube_Account_URLs field with value format string[]. Send a full channel /videos URL inside an array, even for one channel. The YouTube Channel Scraper (Free) template exposes ten output fields. The separate YouTube Channel Scraper template has its own contract; do not reuse these parameters without checking it.

https://www.octoparse.com/template/youtube-channel-scraper-free

https://www.octoparse.com/template/youtube-channel-scraper

Step 2: Save the Complete Python Client

Save the following as youtube_api.py. It checks the input schema, creates a local submission marker before the POST, and saves taskId and lotNo after the response. On later invocations, an accepted saved task is resumed without another creation request. Run only one process per state file.

The Execute Task reference requires serialized JSON in parameters and says to pass lotNo to export when returned. The client follows those rules. It requests ten rows as a small-test target; the historical run returned 50, so this is not an exact row or billing cap.

"""Single-process tutorial client. No requests run when this module is imported."""
import argparse
import csv
import io
import json
import math
import os
from pathlib import Path
import time
from urllib.parse import urlparse
import uuid

import requests

BASE_URL = "https://openapi.octoparse.com/api/agentTools"
SLUG = "youtube-channel-scraper-free"
REQUIRED = ("Title", "Video_URL", "Duration", "View_count", "Date")


def api(method, endpoint, key, **kwargs):
    headers = {"x-api-key": key, **kwargs.pop("headers", {})}
    response = requests.request(method, f"{BASE_URL}/{endpoint}",
                                headers=headers, **kwargs)
    response.raise_for_status()
    payload = response.json()
    if payload.get("success") is False or not isinstance(payload.get("data"), dict):
        raise RuntimeError("API rejected the request or changed its response shape")
    return payload["data"]


def save_state(path, state):
    temporary = path.with_suffix(path.suffix + ".tmp")
    temporary.write_text(json.dumps(state, indent=2), encoding="utf-8")
    temporary.replace(path)


def create_or_resume(path, channel, key, external_user):
    if path.exists():
        state = json.loads(path.read_text(encoding="utf-8"))
        if state.get("channel") != channel or state.get("template") != SLUG:
            raise ValueError("State belongs to another input; choose another --state file")
        if not state.get("taskId"):
            raise RuntimeError("Unresolved submission. Reconcile taskName with searchTasks; do not resubmit")
        if state.get("status") not in {"accepted", "ready"}:
            raise RuntimeError("Saved creation was not accepted; inspect the task before recovery")
        return state

    parsed = urlparse(channel)
    if parsed.scheme != "https" or parsed.hostname != "www.youtube.com" or not parsed.path.endswith("/videos"):
        raise ValueError("Use a full HTTPS www.youtube.com channel /videos URL")
    found = api("GET", "searchTemplates", key,
                params={"slug": SLUG, "page": 1, "limit": 1}, timeout=30)
    template = next((t for t in found.get("templates", []) if t.get("slug") == SLUG), None)
    if not template or "Cloud" not in template.get("executionMode", []):
        raise RuntimeError("Exact cloud template not found")
    schema = template.get("inputSchema") or found.get("inputSchema") or []
    field = next((f for f in schema if f.get("field") == "youTube_Account_URLs"), {})
    if field.get("valueFormat") != "string[]":
        raise RuntimeError("Input schema changed; review it before creating a task")
    parameters = {"youTube_Account_URLs": [channel]}
    if any(f.get("required") and f.get("field") not in parameters for f in schema):
        raise RuntimeError("Template now requires additional inputs")
    state = {"template": SLUG, "channel": channel,
             "taskName": "youtube-" + uuid.uuid4().hex,
             "status": "submission_pending"}
    # Reserve before POST. An ambiguous failure leaves a marker that blocks resubmission.
    with path.open("x", encoding="utf-8") as file:
        json.dump(state, file, indent=2)
    data = api("POST", "executeTask", key,
               headers={"x-external-user-id": external_user}, timeout=60,
               json={"templateName": template.get("templateName", SLUG),
                     "taskName": state["taskName"], "targetMaxRows": 10,
                     "parameters": json.dumps(parameters)})
    state.update({k: data.get(k) for k in ("taskId", "lotNo", "status", "retryGuidance")})
    save_state(path, state)
    if not state["taskId"] or state["status"] not in {"accepted", "ready"}:
        raise RuntimeError("Task not accepted; inspect the saved identifiers before retrying")
    return state


def wait_value(data, fallback=60):
    value = float((data.get("retryGuidance") or {}).get("waitSecondsMin", fallback))
    if not math.isfinite(value) or value < 0:
        raise ValueError("Invalid server waiting interval")
    return max(1, value)


def poll_export(state, key, budget=900, attempts=30):
    deadline = time.monotonic() + budget
    wait = wait_value(state)
    params = {"taskId": state["taskId"], "exportFileType": "CSV", "previewRows": 5}
    if state.get("lotNo") is not None:
        params["lotNo"] = state["lotNo"]
    for _ in range(attempts):
        remaining = deadline - time.monotonic()
        if wait >= remaining:
            raise TimeoutError("Polling budget exhausted; resume the same state later")
        time.sleep(wait)
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise TimeoutError("Polling budget exhausted")
        data = api("GET", "exportData", key, params=params, timeout=min(30, remaining))
        if time.monotonic() >= deadline:
            raise TimeoutError("Response arrived after the polling budget; resume later")
        status = data.get("status")
        if status == "exported":
            if not data.get("exportFileUrl"):
                raise RuntimeError("Exported response has no artifact URL")
            return data
        if status in {"no_data", "failed", "invalid"}:
            raise RuntimeError(f"Export ended with {status}; keep the saved task state")
        if status not in {"collecting", "exporting"}:
            raise RuntimeError("Unknown export status; inspect current API guidance")
        wait = wait_value(data, wait)
    raise TimeoutError("Polling attempt limit reached; resume the same state later")


def validate_rows(rows):
    if not isinstance(rows, list) or not rows:
        raise ValueError("No row objects to validate")
    seen = set()
    for index, row in enumerate(rows, 1):
        if not isinstance(row, dict) or any(not str(row.get(k) or "").strip() for k in REQUIRED):
            raise ValueError(f"Row {index}: missing required field")
        url = urlparse(str(row["Video_URL"]))
        if url.scheme != "https" or url.hostname not in {"www.youtube.com", "youtube.com", "youtu.be"}:
            raise ValueError(f"Row {index}: unexpected video URL")
        if row["Video_URL"] in seen:
            raise ValueError(f"Row {index}: duplicate video URL")
        seen.add(row["Video_URL"])
    return len(rows)


def download_csv(url, destination):
    if urlparse(url).scheme != "https":
        raise ValueError("Expected an HTTPS export URL")
    # Use only the artifact URL returned for this task. Never forward API headers.
    with requests.get(url, timeout=30, stream=True, allow_redirects=False) as response:
        response.raise_for_status()
        if response.status_code != 200:
            raise RuntimeError("Inspect export response guidance before following redirects")
        chunks, size = [], 0
        for chunk in response.iter_content(65536):
            size += len(chunk)
            if size > 25 * 1024 * 1024:
                raise ValueError("Sample export exceeds 25 MiB; use a streaming pipeline")
            chunks.append(chunk)
    content = b"".join(chunks)
    rows = list(csv.DictReader(io.StringIO(content.decode("utf-8-sig"), newline="")))
    count = validate_rows(rows)
    with destination.open("xb") as file:
        file.write(content)
    return count


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--channel", required=True)
    parser.add_argument("--state", type=Path, default=Path("youtube-task.json"))
    parser.add_argument("--download-csv", type=Path)
    args = parser.parse_args()
    key = os.environ["OCTOPARSE_API_KEY"]
    user = os.environ["OCTOPARSE_EXTERNAL_USER_ID"]
    state = create_or_resume(args.state, args.channel, key, user)
    print("Task state saved:", args.state)
    data = poll_export(state, key)
    preview = data.get("sampleData")
    if preview:
        print("Validated preview rows:", validate_rows(preview))
    else:
        print("No preview returned; validate the CSV before using the data")
    if args.download_csv:
        count = download_csv(data["exportFileUrl"], args.download_csv)
        print("Validated CSV rows:", count, "Saved:", args.download_csv)
    else:
        print("Export available. Re-run with --download-csv to save and validate it")


if __name__ == "__main__":
    try:
        main()
    except (requests.RequestException, ValueError, RuntimeError, TimeoutError, OSError) as error:
        # Request exceptions can contain signed URLs; do not print them in shared logs.
        if isinstance(error, requests.RequestException):
            raise SystemExit("HTTP/network error. Keep the state file and reconcile before retrying")
        raise SystemExit(str(error))

Step 3: Run, Resume, and Save a CSV

After checking the target and your account access, run this command. If the state file does not exist, it creates a cloud task. If it contains an accepted task for the same channel, it resumes export polling. The optional download flag saves the CSV only after validating the required columns and checking for duplicate video URLs.

python youtube_api.py --channel "https://www.youtube.com/@Octoparsewebscraping/videos" --state youtube-task.json --download-csv youtube-videos.csv

Omit --download-csv to check the export and its preview without downloading the artifact. The script deliberately refuses to overwrite an existing CSV. Choose a new output filename when you need another saved snapshot. It also rejects exports larger than 25 MiB; larger datasets need a streaming validation pipeline.

A populated preview validates only those preview rows. The CSV path validates every downloaded row for nonempty Title, Video_URL, Duration, View_count, and Date, an expected YouTube URL host, and duplicate URLs. These checks do not establish that every video on a channel was collected. Keep view counts and dates in their original form until your analysis layer normalizes them.

The Export Data reference distinguishes export metadata from the downloadable artifact. This client downloads only when you supply the flag and never forwards the API key to the file host. Review any changed response guidance or redirects before adapting the example.

Step 4: Recover Without Creating Duplicate Tasks

If polling times out, run the same command with the same state file. Each invocation gets a fresh polling budget; it does not start a fresh cloud task. The client checks a 15-minute budget before and after waiting and after each request, and stops after at most 30 polling attempts. Python Requests timeouts are not a strict wall-clock cancellation mechanism: an in-flight request can overrun the budget. Stopping this client does not stop cloud extraction.

If task creation times out, the state file remains marked submission_pending. Do not delete it and retry blindly. Use Search Tasks with the same external user ID and the unique taskName in the file. Confirm the matching target and task before adding its verified taskId, lotNo when available, and accepted status to the saved state. If no match is immediately visible, that alone does not prove the POST failed; inspect the account or contact support before resubmitting.

What Did the Verified API Run Return?

In the August 14, 2026 first-party test, the URL-based template completed the asynchronous Octoparse API workflow in approximately 260 seconds and reported 50 video records. All five preview rows contained a title, video URL, duration, visible view count, and publication date. These records document the historical extraction; they do not certify the revised Python client against today’s live API.

What Did the August 30–31 MCP Live Run Return?

The historical retest followed the official MCP workflow: discover the template, execute one task, and inspect the export. It used the same free template but a different client interface from the Python HTTP example. The run started on August 30, 2026, and the latest snapshot was recorded on August 31 (Asia/Shanghai). Private task and export identifiers are omitted.

All five preview rows contained channel context, title, video and cover URLs, duration, visible view count, and date. The result verifies the MCP path for this template, account, target, and test time. It does not make 199 a fixed final total, because the status was still running when the snapshot was exported.

Sanitized MCP Preview

What Is a YouTube Scraper API?

A YouTube scraper API lets an application request structured data from a managed extraction workflow. With Octoparse, the template handles webpage collection while Python handles orchestration, validation, and storage. This guide concerns permitted public metadata, not video downloads or access-control bypasses. For the broader concept, see the web scraping API guide.

YouTube Scraper API vs. YouTube Data API

The official YouTube Data API and a scraper API solve overlapping but different problems. The official API is the natural first choice when its supported resources, fields, and policies fit your application. According to Google’s YouTube Data API overview, it requires a Cloud project and credentials, uses a quota system, and requires OAuth 2.0 for operations involving authorization or private user data.

Official quota reference (checked September 1, 2026): Google documents a separate daily allowance of 100 search.list calls, with each call costing one unit. A channels.list request costs one unit, while the default combined allowance for other endpoints is 10,000 units per day. These defaults can change, so confirm the current quotas in your Google Cloud project before choosing an architecture.

A scraper API works from information publicly displayed on webpages. It may be useful when your project needs a web-visible dataset in a format provided by an existing scraping template. It is not an “unlimited” or policy-free version of the official API.

YouTube API comparison

The two methods can also coexist. For example, a team might use the official API for supported product features and use a scraper for an internal research dataset that reflects what visitors can see on public channel pages.

What Data Can You Extract from YouTube Channels?

The verified YouTube Channel Scraper (Free) workflow is designed for public channel and video metadata. Its observed schema and sample included:

https://www.octoparse.com/template/youtube-channel-scraper-free

• Video title
• Author or channel account
• Video URL
• Publish time
• View count
• Video length

The template page may describe broader input behavior, but the verified API run used one full channel /videos URL in youTube_Account_URLs. Treat the live inputSchema and outputSchema returned by searchTemplates as the source of truth before sending one or multiple inputs. This workflow is not presented as a comments or transcript extractor.

Treat the template’s live API schema as the source of truth. Marketing copy and screenshots help you understand the product, but an integration must use the field keys and allowed options returned by searchTemplates. Never infer a parameter key from a display label such as “Usernames to Scrape”; use the machine-facing field name.

Why Use Octoparse for a YouTube Scraping Workflow?

Building a YouTube crawler yourself gives you control, but it also makes your team responsible for browser orchestration, scrolling behavior, selector changes, retries, infrastructure, and export logic. Octoparse provides a web data platform with a Templates Gallery and Octoparse Desktop for building or validating workflows, then exposes cloud task management and export through the OpenAPI. The same template boundary can also be operated through the AI Assistant (formerly Agentic Mode), MCP, or CLI beta where the relevant access is enabled; those interfaces orchestrate extraction, while your Python service validates and routes the data.

That division of labor is useful when you want to:

• Send YouTube data into an existing Python pipeline
• Run the same collection method for many public channels
• Export structured data without operating a browser cluster
• Keep application code focused on validation, orchestration, and analysis

The benefit is not “zero engineering.” You still need to select the correct template, validate parameters, secure credentials, manage task state, and handle failures. The difference is that you are integrating a managed scraping workflow instead of rebuilding the crawler layer.

Before You Schedule the Workflow

Run the revised client on one permitted target with your account before scheduling it. Confirm the actual export encoding, field names, row completeness, and current retry guidance. Local simulated-response tests check client behavior; they do not establish live extraction availability or throughput.

For a scheduled pipeline, use a durable job store rather than this single-process state file. Record the input, task ID, batch ID, collection timestamp, and delivery outcome for each intentional run. Resume an unfinished run before creating the next one, and use stable video identifiers when comparing snapshots.

Common Errors and How to Handle Them

YouTube scraper API troubleshooting table

Avoid generic retry decorators around task creation. Retries are useful for idempotent reads, but cloud-task creation can have side effects even when the client never receives the response.

How Fast Is a YouTube Channel Scraper?

There is no single reliable runtime for every channel. Channel size, page behavior, cloud load, template configuration, and retry conditions can all affect completion time. A small public channel may finish quickly, while a large catalog can require a longer extraction and export cycle.

Use a representative sample of your own target channels before setting production timeouts or service-level expectations. In Octoparse’s 2026 HubSpot YouTube competitor-analysis test, a channel task returned 364 videos in 4 minutes 24 seconds with no failed URLs. A follow-on task across 10 videos returned 124 comments in 1 minute 21 seconds with no duplicates. These are observed results for those targets and settings, not general throughput guarantees.

Practical YouTube Scraper API Use Cases

Competitor channel monitoring

Collect public video titles, dates, durations, and visible view counts on a schedule. A downstream job can calculate posting frequency, identify recurring formats, and flag unusually strong videos for human review.

Creator research

Build a consistent dataset of public channel videos for an initial creator shortlist. Scraped metadata can support discovery and comparison, but human review should remain part of any partnership decision.

Content research

Group titles by topic, compare publishing cadence, or analyze how video length varies across a channel. The API handles repeatable collection; your analysis layer handles normalization and interpretation.

For adjacent collection needs, use the YouTube channel crawler guide for no-code channel exports and the YouTube comment scraper guide for comment-specific workflows. This tutorial remains focused on the Python task lifecycle.

Dataset enrichment

Join public YouTube metadata to an existing list of companies, products, or creators. Preserve collection timestamps because values such as visible view counts change over time.

Legal and Responsible Scraping Considerations

Public visibility is not blanket permission to collect or reuse information for every purpose. YouTube’s current Terms of Service restrict automated access except in specified circumstances, including prior written permission or where applicable law permits it. They also restrict collecting information that may identify a person unless an applicable permission or legal basis exists.

Before running a scraper:

• Review YouTube’s current Terms of Service and applicable policies.
• Confirm that your collection and intended use are lawful in relevant jurisdictions.
• Limit the workflow to data you are authorized to collect.
• Avoid private, restricted, or sensitive personal information.
• Do not use the workflow to download videos or bypass access controls.
• Apply reasonable volume, retention, and security controls.
• Seek legal advice when the use case or jurisdiction creates uncertainty.

This article explains a technical integration and is not legal advice.

FAQs about YouTube Scraper API

1. Can I use this channel template for comments or transcripts?

This example validates channel-video metadata only. Comments and transcripts require a workflow designed for those outputs, with its own input schema and permitted-use checks. Use the YouTube comment scraper guide to assess comment collection; do not assume that changing the URL adds unsupported fields.

2. Does “Free” mean every API run is unlimited?

No. The template name identifies the free template, not an unlimited account entitlement. Check the current template details, account access, and applicable usage limits before scheduling runs. Also distinguish a requested row target from the observed export size: the historical test requested ten rows and reported 50.

3. Does a valid CSV prove the channel export is complete?

No. The client checks field presence, URL hosts, and duplicate URLs in the file it receives. It does not compare that file with the entire channel catalog. Check task completion, expected coverage, collection time, and any pagination or export guidance before treating a snapshot as a complete dataset.

4. How should I compare view counts across runs?

Store the raw view-count text and a collection timestamp for each snapshot. Normalize values such as abbreviated thousands in a separate analysis step, documenting the rules and locale. Match videos by stable identifiers rather than titles, which can change. A difference between snapshots is an observed change, not a guaranteed real-time measurement.

Build the Integration Around the Task Lifecycle

Start with one permitted channel, preserve the task and batch IDs, and inspect the exported rows before scheduling repeat runs. When your first CSV passes validation, connect it to the analysis or storage system you actually need. Keep the historical test results as examples, not speed or completeness guarantees.