Octoparse Web Crawler Helps Automatically Collect Twitter Data for Academic Research

2 min read

Octoparase User Review 

By Fizah Zainudin from London- Basic Plan User 

I am currently using Octoparse to obtain Twitter data for my EngD research. I use the functionality to automatically open and log into my Twitter account and perform a specific search on Twitter to get the data that I require.

How does Octoparse work for my project?

I created a task using the advanced mode to initially open the Twitter page and then log in using my credentials. The task will then search for a specific set of lexicons that I have defined and will then show the results of the search.

The task will then scroll down the Twitter search results for a predefined number of times in order to get as many tweets as possible and then the task will iterate through tweets to extract the username of the person tweeting and also the actual tweets and the timestamp. I then used the extracted data for further sentiment analysis using a separate tool.

Why do I choose Octoparse?

I find that it is very intuitive to use Octoparse. There are also many resources available on its website, and a video specifically showing how to extract tweets is very useful. Also, there appears to be an increasing number of users which means that I expect more discussions and tips to be posted online moving forward. I also find that for resource-hungry processes, it is very useful to have the ability to run those processes in the cloud so that they will finish quicker.

I have only scratched the surface in terms of the functionality available on Octoparse and look forward to experimenting with them and potentially using this tool as the strategic platform for data extraction for my research. I find other tools out there to have very limited capabilities and are not as intuitive to use as Octoparse. Additionally, I also look forward to introducing Octoparse as the strategic platform for social media data extraction to my department at the university. As it stands, I also feel that Octoparse has the right price point which is not too prohibitive for users in a small organization like myself.

Hot posts

Explore topics

Get web automation tips right into your inbox
Subscribe to get Octoparse monthly newsletters about web scraping solutions, product updates, etc.

Get started with Octoparse today


Related Articles