Web Scraping Case Study | Scrolling down to scrape TweetsSaturday, October 08, 2016 1:42 AM
List features covered
(Download my extraction task of this tutorial HERE just in case you need it.)
Now, let's get started!
Step 1. Set up basic information and navigate to the target website
Step 2. Scroll down to load the web content
We are now on the search result page. Waiting until the page loaded.
Notice that, we suggest you'd better not set a relatively high number of "Scroll times", like 10,000 or more.
Step 3. Create a list of items
Move your cursor over the article with similar layout, where you would extract the content of the article.
Now, the first item has been added to the list, we need to finish adding all items to the list
Now we get all the sections added to the list with similar layout
Step 7. Select the data to be extracted
Step 8. Rename data field
All the content will be selected in Data Fields.
Step 9. Starting running your task
Octoparse will automatically extract all the data selected.
Step 10. Check the data and export
Author: The Octoparse Team
For more information about Octoparse, please click here.
Sign up today!