“Octoparse did a great job not only on data scraping but also on understanding projects and the center’s needs. Data collected from online spaces would have been useless unless Octoparse understood the whole purpose of the project.”
— Jinho Jung, Research Associate, Center for Food Demand Analysis and Sustainability, Purdue University
About the Customer
The Center for Food Demand Analysis and Sustainability (CFDAS) is a research center within the College of Agriculture at Purdue University. Its mission is to improve the flow of data about consumers and food markets — helping consumers make more informed food choices, and enabling agribusinesses, policymakers, and farmers to improve the food system.
The Challenge
CFDAS needed to collect data on over 2 million grocery products from 20 online grocery chains — every single day. This required faster servers, larger data storage, and real-time data transfer at scale. The center also needed to aggregate all collected data into an interactive dashboard so audiences could monitor grocery prices daily, across regions and product categories.
Doing this manually or with fragile custom scrapers wasn’t an option. They needed a reliable, scalable solution that could keep pace with their research demands.
The Solution
CFDAS partnered with Octoparse to handle their daily web scraping needs. Octoparse now pulls data from 20 online grocery chains across 5 product categories and approximately 342 zip codes each day — aggregating up to 2.3 million products daily. The center’s data depot is directly connected to Octoparse’s data storage, enabling real-time data transfer without delay.
Why Octoaprse
Faster servers and larger storage for data
The center needed to collect data from 20 online grocery chains across 5 categories of grocery items and around 342 zip codes every day — aggregating up to 2.3 million products daily. Octoparse’s infrastructure handled the scale without compromise.
Detailed and well-structured data
Octoparse developed a scraping program for detailed grocery information such as items, categories, and geographies. With the scraped data, the center’s dashboard helps producers, agribusinesses, and policymakers make decisions that improve the food system, as well as guide research on nutrition and plant innovations.
More efficient everyday data management
Octoparse links its data storage with the center’s data depot for transferring scraped data every day, enabling the center to manage data in a timelier manner — with no manual intervention required.
Looking Ahead
CFDAS now provides agribusinesses, farmers, and policymakers with timely, accessible data and insights across food prices, food production and supply, consumer spending, and consumer preferences. By leveraging web scraping at scale, the center is building a food system that works better for everyone.
