使用Scrapy从表中收集数据

坦齐布·侯赛因·尼尔乔尔

我是第一次尝试Scrapy。经过大量研究后，我掌握了基础知识。现在，我试图获取表的数据。它不起作用。请在下面查看源代码。

items.py

from scrapy.item import Item, Field

class Digi(Item):

    sl = Field()
    player_name = Field()
    dismissal_info = Field()
    bowler_name = Field()
    runs_scored = Field()
    balls_faced = Field()
    minutes_played = Field()
    fours = Field()
    sixes = Field() 
    strike_rate = Field()

digicric.py

from scrapy.spider import Spider
from scrapy.selector import Selector
from crawler01.items import Digi

class DmozSpider(Spider):
    name = "digicric"
    allowed_domains = ["digicricket.marssil.com"]
    start_urls = ["http://digicricket.marssil.com/match/MatchData.aspx?op=2&match=1250"]

    def parse(self, response):

        sel = Selector(response)
        sites = sel.xpath('//*[@id="ctl00_ContentPlaceHolder1_divData"]/table[3]/tr')
        items = []

        for site in sites:
            item = Digi()
            item['sl'] = sel.xpath('td/text()').extract()
            item['player_name'] = sel.xpath('td/a/text()').extract()
            item['dismissal_info'] = sel.xpath('td/text()').extract()
            item['bowler_name'] = sel.xpath('td/text()').extract()
            item['runs_scored'] = sel.xpath('td/text()').extract()
            item['balls_faced'] = sel.xpath('td/text()').extract()
            item['minutes_played'] = sel.xpath('td/text()').extract()
            item['fours'] = sel.xpath('td/text()').extract()
            item['sixes'] = sel.xpath('td/text()').extract()
            item['strike_rate'] = sel.xpath('td/text()').extract()
            items.append(item)
        return items

关键问题是您sel在循环内部使用。另一个关键问题是XPath表达式指向td元素，而您需要td按索引获取元素并将其与item字段相关联。

工作解决方案：

def parse(self, response):
    sites = response.xpath('//*[@id="ctl00_ContentPlaceHolder1_divData"]/table[3]/tr')[1:-2]

    for site in sites:
        item = Digi()
        item['sl'] = site.xpath('td[1]/text()').extract()
        item['player_name'] = site.xpath('td[2]/a/text()').extract()
        item['dismissal_info'] = site.xpath('td[3]/text()').extract()
        item['bowler_name'] = site.xpath('td[4]/text()').extract()
        item['runs_scored'] = site.xpath('td[5]/b/text()').extract()
        item['balls_faced'] = site.xpath('td[6]/text()').extract()
        item['minutes_played'] = site.xpath('td[7]/text()').extract()
        item['fours'] = site.xpath('td[8]/text()').extract()
        item['sixes'] = site.xpath('td[9]/text()').extract()
        item['strike_rate'] = site.xpath('td[10]/text()').extract()
        yield item

它正确输出11个项目实例。

本文收集自互联网，转载请注明来源。

如有侵权，请联系 [email protected] 删除。