Step-by-step tutorials for you to get started with web scraping
Download OctoparseScraping video info from Youtube
Wednesday, December 04, 2019The latest version for this tutorial is available here. Go to have a check now!
In this tutorial, we are going to show you how to scrape video information from Youtube. Also, the ready-to-use Youtube Template has been inserted in our latest version, you could check it out here: Task Templates
.
If you would like to build the scraper from scratch, you might want to use the URL in this tutorial: https://www.youtube.com
Here are the main steps in this tutorial: [Download task file here]
1."Go To Web Page"- to open the targeted web page
2. Create a "Loop Item"- to loop enter searching keywords
3. Dealing with infinitive scrolling
4.Create a "Loop Item" -to loop extract each item
5.Extract data - to select data you need to scrape
6.Run extraction - to run your task and get data
1)"Go To Web page"- to open the targeted web page
- Click "+Task" to start a new task with Advanced Mode
- Paste the URL into the "Input URL" box
- Click "Save URL" to move on
2)Create a "Loop Item" - to loop enter searching keywords
We can customize our "text list" to create a loop search action. Octoparse will automatically enter every keyword in the list into the search box, one line a time.
- Drop a "loop item" action into the workflow designer
- Go to loop mode and select "Text list"
- Click "a" to enter the keywords lists with one keywords per line. Here we'll enter "Big Data" and "Machine Learning"
- Click "OK" and "OK" when you finish entering. Then you can see your keywords in the "Loop Item"
- Click on the search box on the page in the built-in browser and select "Enter text” on "Action Tips"
When you click on the input field in the built-in browser, Octoparse can detect that you select a search box, the "Enter text" action will automatically appear on "Action tips".
- Input the first keyword "Big Data" on "Action Tips"
- Click "OK", then the"Enter Text" action will be generated in the workflow
- Drag the "Enter Text" action into the"Loop Item. Click on the "Enter Text" action
Go to "Loop Text" and select "Use the text in loop item to fill in the text box" and click "OK" to save.
- Click the search button of the web page and select "Click button” on "Action Tips", you will notice the "Click Item" action is added into the workflow.
3)Dealing with infinitive scrolling
In this case, pagination is not an option for loading the searching results, we will need to scroll down to the bottom of the page continuously to load all the contents.
- Check "Scroll down to bottom of the page when finished loading" under "Advanced Options"
- Set "Scroll times" and "Internal" you need
- Select "Scroll down to bottom of the page" as "Scroll way"
- Click "OK" button to save the result
|
Tips!
|
4) Create a "Loop Item" -to loop extract each item
When you create a list of items to scrape a website, sometimes the list may include several "Ads" items. To exclude the promotional video in this case, we can start building the "Loop Item" from the second row of the products on this page.
- Select the second block in the built-in browser
We need to make sure the whole block of the first video item is covered in blue when you curse over your mouse. Only in this way, we could see the whole item block is highlighted in green after clicking, covering all other information like video title, channel name, total reviews...etc.
- Click the third and fourth whole video item, until Octoparse identifies all other videos.
Octoparse will automatically recognize the other blocks and highlight them in green. (If not, keep clicking on the next one till all of them are selected)
- Click " Extract text of the selected element " on "Action Tips" panel.
|
Tips!
|
5) Extract data - to select data you need to scrape
- Click data you need in the item block which is highlighted in red.
- Click "Extract text of the selected element" and rename the "Field name" column if necessary.
Rename the fields by selecting from the pre-defined list or inputting on your own
- Click "OK" to save the result.
6) Run extraction - to run your task and get data
- Click "start extraction"
- Select "local extraction" to run the task on your computer
Below is the output sample:
Was this article helpful? Feel free to let us know if you have any question or need our assistance.
Contact us here !
Author: Momo
Editor：Suire
Download Octoparse to start web scraping or contact us for any
question about web scraping!
