Blog > Post

Web Scraping -- Scraping AJAX and JavaScript Websites

Monday, December 20, 2021


Handle AJAX and JavaScript

Speaking about handling AJAX and JavaScript while web scraping, sometimes it could be tricky, especially when you are a tech noob. (Hmm.. I’m sorry. A...What... What did you just say?)

I’ve received lots of questions about scraping AJAX and JavaScript lately. And I’ve compiled some questions I get asked the most often.


. How do I crawl an infinite-scrolling AJAX website?

. How do I scrape data and click on load button or next button?

. How do I scrape any websites with AJAX content(like Gumtree)?

. Can Octoparse be used to scrape dynamic content from websites that are using AJAX?

. Can I scrape data from website with pagination?

. Can I scrape websites which dynamically load data(like Facebook)?

. Can I crawl a website that loads content using Javascript?



Scraping Sites with AJAX Is Not Easy

Sometimes people see web pages and find AJAX content loading and think the site cannot be scraped. If you are learning Python and you are dipping your hand into building a web scraper. It is not going to be easy. If you are looking for an easy and quick way to do this, especially for large workloads, you may want to look into some third-party applications for scraping websites with AJAX.


Example: Scrape Websites with Infinite Scrolling

So as an example, what I’m going to show you is how to scrape websites with infinite scrolling. ( If you are experienced programmer and you just write your awesome web scraper, then just ignore my gibberish.)

Check out how to scrape websites with Infinite scrolling here.




We use cookies to enhance your browsing experience. Read about how we use cookies and how you can control them by clicking cookie settings. If you continue to use this site, you consent to our use of cookies.
Accept decline