Blog > Customer Reviews > Post

Octoparse Web Crawler Helps Automatically Collect Twitter Data for Academic Research

Monday, May 8, 2017

Octoparase User Review By Fizah Zainudin from London- Basic Plan User 


I am currently using Octoparse to obtain Twitter data for my EngD research. I use the functionality to automatically open and log into my twitter account and perform a specific search on twitter to get the data that I require.


How does Octoparse work for my project?

I created a task using the advanced mode to initially open the twitter page and then login using my credentials. The task will then search for a specific set of lexicons that I have defined and will then show the results of the search.


The task will then scroll down the twitter search results for the predefined number of times in order to get as many tweets as possible and then the task will iterate through tweets to extract the username of the person tweeting and also the actual tweets and the timestamp. I then used the extracted data for further sentiment analysis using a separate tool.


Why do I choose Octoparse?

I find that it is very intuitive to use Octoparse. There are also many resources available on its website, and a video specifically showing how to extract tweets which is very useful. Also, there appears to be an increasing number of users which means that I expect more discussions and tips to be posted online moving forward. I also find that for a resource hungry processes, it is very useful to have the ability to run those processes in the cloud so that it will finish quicker.


I have only scratched the surface in terms of the functionality available on Octoparse and look forward to experiment with them and potentially use this tool as the strategic platform for data extraction for my research. I find other tools out there to be have very limited capabilities and are not as intuitive to use as Octoparse. Additionally, I also look forward to introducing Octoparse as the strategic platform for social media data extraction to my department at the university. As it stands, I also feel that Octoparse has the right price point which is not too prohibitive for user in a small organization as myself.


Author: The Octoparse Team


More Resources


Twitter Scraping, Text Mining, and Sentiment Analysis Using Python

Scrape Tweets from Twitter

5 Things You Need to Know Before Scraping Facebook

Scrape Instagram with Octoparse 7.X

Social Media Web Scraping Templates Take Away  



We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline