Blog > Customer Reviews > Post

Octoparse Web Crawler Helps Automatically Collect Twitter Data for Academic Research

Monday, May 08, 2017

[Octoparase User Review] By Fizah Zainudin from London- Basic Plan User 


I am currently using Octoparse to obtain Twitter data for my EngD research. I use the functionality to automatically open and log into my twitter account and perform a specific search on twitter to get the data that I require.


How Octoparse works for my project?

I created a task using advanced mode to initially open the twitter page and then login using my credentials. The task will then search for a specific set of lexicons that I have defined and will then show the results of the search.


The task will then scroll down the twitter search results for predefined number of times in order to get as many tweets as possible and then the task will iterate through tweets to extract the username of the person tweeting and also the actually tweets and the timestamp. I then used the extracted data for further sentiment analysis using a separate tool.


Why I choose Octoparse?

I find that it is very intuitive to use Octoparse. There are also many resources available on its website, and a video specifically showing how to extract tweets which is very useful. Also, there appear to be an increasing number of users which means that I expect more discussions and tips to be posted online moving forward. I also find that for a resource hungry processes, it is very useful to have the ability to run those processes on cloud so that it will finish quicker.


I have only scratched the surface in terms of the functionality available on Octoparse and look forward to experiment with them and potentially use this tool as the strategic platform for data extraction for my research. I find other tools out there to be have very limited capabilities and are not as intuitive to use as Octoparse. Additionally, I also look forward to introducing Octoparse as the strategic platform for social media data extraction to my department at the university. As it stands, I also feel that Octoparse has the right price point which is not too prohibitive for user in a small organisation as myself.


Author's Picks

How to Extract Tweets from Twitter

Scrape Websites with Infinite Scrolling (Quora, Facebook,Twitter)

Octoparse Cloud Service 


Download Octoparse Today

For more information about Octoparse, please click here.

Sign up today!




Download Octoparse to start web scraping or contact us for any
question about web scraping!

Contact Us Download