I need to scrape data from this website: https://www.daraz.pk/products/hy-i189662857-s1379994759.html?spm=a2a0e.searchlistcategory.list.3.70426378Fs3yJh&search=1
and html code for this is:
<div class="mod-reviews">
<div class="item">...</div>
<div class="item">...</div>
<div class="item">...</div>
<div class="item">...</div>
</div>
Each class div has 1 review and i want to extract it. I tried following code with selenium:
driver.get(site)
time.sleep(5)
data = driver.find_elements(By.CLASS_NAME,"item")
Hypothetically this should give me all item tags but for some reason this returns nothing. Am I missing something here?
These elements only appear after you scroll down the page for a while. Try to GUI-Automate that with a javascript injection like this
for i in range(25):
driver.execute_script("window.scrollBy(0,100)")
Sadly, that scrolling does not work until you put your mouse cursor there, so there will other gui automation be necessary as well.
But wait! you can trick it:
for i in range(10):
driver.execute_script("window.scrollBy(0,100)")
for i in range(10):
driver.execute_script("window.scrollBy(0,-100)")
for i in range(25):
driver.execute_script("window.scrollBy(0,100)")
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments