我正在尝试从该网站的首页刮到第14页：https : //cross-currents.berkeley.edu/archives?author=&title=&type=All&issue=All®ion=All这是我的代码：
import requests as r
from bs4 import BeautifulSoup as soup
import pandas
#make a list of all web pages' urls
webpages=[]
for i in range(15):
root_url = 'https://cross-currents.berkeley.edu/archives?author=&title=&type=All&issue=All®ion=All&page='+ str(i)
webpages.append(root_url)
print(webpages)
#start looping through all pages
for item in webpages:
headers = {'User-Agent': 'Mozilla/5.0'}
data = r.get(item, headers=headers)
page_soup = soup(data.text, 'html.parser')
#find targeted info and put them into a list to be exported to a csv file via pandas
title_list = [title.text for title in page_soup.find_all('div', {'class':'field field-name-node-title'})]
title = [el.replace('\n', '') for el in title_list]
#export to csv file via pandas
dataset = {'Title': title}
df = pandas.DataFrame(dataset)
df.index.name = 'ArticleID'
df.to_csv('example31.csv',encoding="utf-8")
输出的csv文件仅包含最后一页的目标信息。当我打印“网页”时，它表明所有页面的网址均已正确地放入列表中。我究竟做错了什么？先感谢您！
您只需覆盖所有页面的相同输出CSV文件，就可以.to_csv()在“追加”模式下调用以将新数据添加到现有文件的末尾：
df.to_csv('example31.csv', mode='a', encoding="utf-8", header=False)
或者，最好将标题收集到标题列表中，然后转储到CSV中一次：
#start looping through all pages
titles = []
for item in webpages:
headers = {'User-Agent': 'Mozilla/5.0'}
data = r.get(item, headers=headers)
page_soup = soup(data.text, 'html.parser')
#find targeted info and put them into a list to be exported to a csv file via pandas
title_list = [title.text for title in page_soup.find_all('div', {'class':'field field-name-node-title'})]
titles += [el.replace('\n', '') for el in title_list]
# export to csv file via pandas
dataset = [{'Title': title} for title in titles]
df = pandas.DataFrame(dataset)
df.index.name = 'ArticleID'
df.to_csv('example31.csv', encoding="utf-8")
本文收集自互联网，转载请注明来源。
如有侵权，请联系 [email protected] 删除。
我来说两句