So I'm trying to run a scrape of a website. The scraper runs very well. But whenever I try to write the scraped information/rows into the csv file it deletes the previous row. I end up just having the very last scrape result in the file at the end. I'm sure it's just an indentation error? I'm still new to Python so any help would be appreciated!
Code:
# create general field names
fields = ['name', 'about', 'job_title', 'location','company',
'education','accomplishments','linkedin_url']
with open('ScrapeResults.csv', 'w') as f:
# using csv.writer method from CSV package
write = csv.writer(f)
write.writerow(fields)
f.close()
# Loop-through urls to scrape multiple pages at once
for individual,link in contact_dict.items():
## assign ##
the_name = individual
the_link = link
# scrape peoples url:
person = Person(the_link, driver=driver, close_on_complete=False)
# rows to be written
rows = [[person.name, person.about, person.job_title, person.location, person.company,
person.educations, person.accomplishments, person.linkedin_url]]
# write
with open('ScrapeResults.csv', 'w') as f:
# using csv.writer method from CSV package
write = csv.writer(f)
write.writerows(rows)
f.close()
You will need to open the file in append mode.
change
with open('ScrapeResults.csv', 'w') as f:
to
with open('ScrapeResults.csv', 'a') as f:
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments