In this article, we will show you how to extract and monitor stock prices easily from Yahoo Finance with web scraping.
Why Do We Scrape Stock Prices
Before we start our scraping process, let’s find out the reasons for scraping stock prices. Well, when you constantly extract a stock price and continuously feed the data into your research and data models, you can then train your algorithm with your machine learning code that later gives you more accurate and profitable advice in the investment market.
As you might have known, one of the applications that Octoparse can help with is price monitoring. Not only does it track and monitor prices on a page, but it also extracts raw data from your competitors and scrapes real-time data within a few mouse clicks.
4 Steps to Easily Scrape Stock Prices
Auto-detect feature in Octoparse
The unique auto-detect function makes Octoparse a strong competitor in the market. For this useful function, once you copy and paste a URL into the address bar, it automatically gets started in parsing the webpage and trying to predict what data you want to extract. If the detection doesn’t fully get all the data you want, you can switch the detect results several times to get the one that is closest to your ideal data set. We will look at it in detail later. In this case, we are going to enter the most active stocks URL from Yahoo! Finance. After you paste the URL, click the Start button and wait for a little bit while it’s loading. Now, let’s walk through how to scrape stock prices within four steps.
With the auto-detection feature, it takes 4 steps to scrape and extract stock prices. Or, you can also watch the video guide below:
Step 1: Switch Auto-detect results
When you see “Switch auto-detect results” in Tips Panel, click it through a few times until you see the table of most active stocks is highlighted red in the browser.
Step 2: Take care of the pagination and generate a workflow
We need to set up the pagination. Click the “Edit link” and select the “Next page” button on the page to let Octoparse know where exactly the next button is.
Click the confirm button once you have selected the next button. Uncheck the “Add a page scroll” and generate a workflow.
Step 3: Set up the pagination option and review the workflow
As you can see, a workflow is generated on the right-hand side. Next, we need to set up the pagination option again. Hover the mouse cursor to the Pagination bar and click the gear icon. Expand the “Exit Loop” panel and enter 2. This means we want the loop to quit after the next button on the webpage has been clicked twice.
Let’s review our workflow. First, it executes “Go to Web Page!”. Next, it goes into a pagination loop where contains the real deal of extraction. You can see the “Extract Data” step that extracts what you have preselected. Then the loop will click the next button for you until it satisfies the exit condition.
Step 4: Save and run the task
Save the task and click the “Run” button to fire off our scraping task now. There are a few options for where you want to run the task. For this example’s sake, Select “Run task on your device“.
We can see the extracted data started coming in, and it will stop after 50 lines of data are extracted.
You can read more information about Yahoo Finance web scraping by reading the post here: A Full Guide on Scraping Yahoo Finance.
Export Price Data to Spreadsheet
When it is completed, click the “Export data” button. Now open up the saved spreadsheet and compare the data with the ones you previewed. Make sure the data looks good. As we scroll the spreadsheet, the last row is 51 which makes sense because the first row is the header. 51 minus 1 is 50 rows.
Here is a related post about how to scrape the table data into Excel with 3 different methods, you can read it to learn more if you’re interested.
There you have it. In only a few minutes, we have extracted the most active stocks from Yahoo! Finance and saved the data to a spreadsheet.
