我正在从遵循相同结构的多个div抓取标题,描述,链接和人们的名字。我正在使用BeautifulSoup,并且能够将所有内容从第一格中刮除。但是,我无法从一长串的div中进行抓取,并且无法以CSV或JSON之类的可移植格式获取数据。
我如何从长长的div列表中抓取每个项目,并将该信息存储在JSON对象中以用于每个mp3?
div看起来像这样:
<div class="audioBoxWrap clearBoth">
<h3>Title 1</h3>
<p>Description 1</p>
<div class="info" style="line-height: 1px; height: 1px; font-size: 1px;"></div>
<div class="audioBox" style="display: none;">
stuff
</div>
<div> [ <a href="link1.mp3">Right-click to download</a>] </div>
</div>
<div class="audioBoxWrap clearBoth">
<h3>Title 2</h3>
<p>Description 2</p>
<div class="info" style="line-height: 1px; height: 1px; font-size: 1px;"></div>
<div class="audioBox" style="display: none;">
stuff
</div>
<div> [ <a href="link2.mp3">Right-click to download</a>] </div>
</div>
我已经想出了如何从第一个div抓取,但是我无法获取每个div的信息。例如,下面的代码只会一遍又一遍地吐出第一个div的h3。
我知道我可以为标题,描述等创建一个python列表,但是如何保持JSON之类的元数据结构,以使title1,link1和description1以及title2的信息保持在一起。
with open ('soup.html', 'r') as myfile:
html_doc = myfile.read()
soup = BeautifulSoup(html_doc, 'html.parser')
audio_div = soup.find_all('div', {'class':"audioBoxWrap clearBoth"})
print len(audio_div)
#create dictionary for storing scraped data. I don't know how to store the values for each mp3 separately.
for i in audio_div:
print soup.find('h3').text
我希望我的JSON看起来像这样:
{
"podcasts":[
{
"title":"title1",
"description":"description1",
"link":"link1"
},
{
"title":"title2",
"description":"description2",
"link":"link2"
}
]
}
遍历每条轨道并进行特定于上下文的搜索:
from pprint import pprint
from bs4 import BeautifulSoup
data = """
<div>
<div class="audioBoxWrap clearBoth">
<h3>Title 1</h3>
<p>Description 1</p>
<div class="info" style="line-height: 1px; height: 1px; font-size: 1px;"></div>
<div class="audioBox" style="display: none;">
stuff
</div>
<div> [ <a href="link1.mp3">Right-click to download</a>] </div>
</div>
<div class="audioBoxWrap clearBoth">
<h3>Title 2</h3>
<p>Description 2</p>
<div class="info" style="line-height: 1px; height: 1px; font-size: 1px;"></div>
<div class="audioBox" style="display: none;">
stuff
</div>
<div> [ <a href="link2.mp3">Right-click to download</a>] </div>
</div>
</div>"""
soup = BeautifulSoup(data, "html.parser")
tracks = soup.find_all('div', {'class':"audioBoxWrap clearBoth"})
result = {
"podcasts": [
{
"title": track.h3.get_text(strip=True),
"description": track.p.get_text(strip=True),
"link": track.a["href"]
}
for track in tracks
]
}
pprint(result)
印刷:
{'podcasts': [{'description': 'Description 1',
'link': 'link1.mp3',
'title': 'Title 1'},
{'description': 'Description 2',
'link': 'link2.mp3',
'title': 'Title 2'}]}
本文收集自互联网,转载请注明来源。
如有侵权,请联系 [email protected] 删除。
我来说两句