How can I parse a website using Selenium and Beautifulsoup in python?

twitch after coffee :

New to programming and figured out how to navigate to where I need to go using Selenium. I'd like to parse the data now but not sure where to start. Can someone hold my hand a sec and point me in the right direction?

Any help appreciated -

RocketDonkey :

Assuming you are on the page you want to parse, Selenium stores the source HTML in the driver's page_source attribute. You would then load the page_source into BeautifulSoup as follows:

In [8]: from bs4 import BeautifulSoup

In [9]: from selenium import webdriver

In [10]: driver = webdriver.Firefox()

In [11]: driver.get('http://news.ycombinator.com')

In [12]: html = driver.page_source

In [13]: soup = BeautifulSoup(html)

In [14]: for tag in soup.find_all('title'):
   ....:     print tag.text
   ....:     
   ....:     
Hacker News

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

How can I open a website in my web browser using Python?

How can I parse a website using Selenium and Beautifulsoup in python?

How can I take a screenshot/image of a website using Python?

How can I loop scraping data for multiple pages in a website using python and beautifulsoup4

How can I log in this specific website using Python Requests?

How can I grab multiple information from a table on a website using beautifulsoup

how can I upload image to this website with selenium in python

How can I parse two strings in a table row by using beautifulsoup?

how i can split word and number after scraping website with BeautifulSoup?

How do I scrape ::before element in a website using selenium python

How can I parse a table from a specific string using BeautifulSoup?

How can I parse in the onclick() text in Python3 BeautifulSoup?

how can i extract this value from a website, with python, selenium and chromedriver

How can I use a variable in website link using Python?

How can I scrape a product price from a website using Beautifulsoup?

How do I wait for this website to load using selenium in Python?

How can I get the tooltip text from a website using selenium and python where the text comes from a javascript

How Can i Parse XML using Python

How can I parse xml content of wsdl using Python suds?

How can I print an item in a website using selenium

How to receive website link in Python using BeautifulSoup

How to login to a website using selenium in Python?

Python-Selenium : How can I get access to this part of the website?

How to upload an image to a website using selenium python

How to Parse XML from website using python

How can i download specifcs files from a website using Python + Selenium

How can I scrape a website that does not show any HTML codes in the source using Python without Selenium

How can I take the #text part of the website with BeautifulSoup?

How to properly load website using selenium in python