如何使用 Scrapy 和 Selenium 从使用 javascript 和 php 的网站获取数据？

路易斯·彼得

我正在尝试从提供事故信息的网站获取数据。我为此使用了 Scrapy 和 Selenium，但它不起作用。我对此很陌生，并试图了解发生了什么。我在 venv 中安装了 Scrapy 和 Selenium。该网站的结构有点陈旧，很难理解。

任何帮助，将不胜感激！

我使用的是 Firefox，所以在设置中我使用了这个：

SELENIUM_DRIVER_NAME = 'firefox'
SELENIUM_DRIVER_EXECUTABLE_PATH = which('geckodriver.exe')
SELENIUM_DRIVER_ARGUMENTS=\['-headless'\] \

我的代码如下所示：

import scrapy
from selenium.webdriver import firefox
from http.server import executable
from lib2to3.pgen2 import driver
from scrapy.utils.project import get_project_settings


class MeldingenSpider(scrapy.Spider):
    name = '112meldingen'

    def start_requests(self):
        settings = get_project_settings
        driver_path = settings.get('SELENIUM_DRIVER_EXECUTABLE_PATH')
        driver = firefox(executable_path=driver_path)

        driver.get('http://ftp.112meldingen.nl/index.php')
        xpath = '//*[@id="divContentAlerts"]'
        link_elements = driver.find_elements_by_xpath(xpath)


    def parse(self, response):
        articles = response.css('table::attr(id.alerts)')
        for article in articles:
            #if "haven" in article.css('div.title a::text').get():
                yield {
                    'headline': article.css('td.bold a::text').get() ,
                    'timestamp': article.css('td.bold span').get(),
                    'location' : article.xpath('td > td').get()[3]
                    }

F·霍克

您可以借助SeleniumRequest.

脚本：

import scrapy
from scrapy import Selector
from scrapy_selenium import SeleniumRequest
#from selenium.common.exceptions import NoSuchElementException

class MeldingenSpider(scrapy.Spider):
    name = 'dingen'

    responses = []

    def start_requests(self):
        yield SeleniumRequest(
            url='http://ftp.112meldingen.nl/index.php',
            callback=self.parse
        )

    def parse(self, response):
        driver = response.meta['driver']
        intial_page = driver.page_source
        self.responses.append(intial_page)
        driver.implicitly_wait(2)

        for resp in self.responses:
            r = Selector(text=resp)
            articles = r.css('table#alerts')
            for article in articles:
            #if "haven" in article.css('div.title a::text').get():
                    yield {
                        'headline': article.css('td.bold a::text').get() ,
                        'timestamp': article.css('td.bold span::text').get().replace('\xa0\xa021',''),
                        'location' : [x.replace('\xa0',' ') for x in article.xpath('.//tr/td[@class="bold center"]/following-sibling::td//text()').getall()][-1]
           

         }

您必须从 settings.py 文件中更改以下指令

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Middleware

DOWNLOADER_MIDDLEWARES = {
    'scrapy_selenium.SeleniumMiddleware': 800
}

# Selenium
from shutil import which
SELENIUM_DRIVER_NAME = 'chrome'
SELENIUM_DRIVER_EXECUTABLE_PATH = which('chromedriver')
# '--headless' if using chrome instead of firefox
SELENIUM_DRIVER_ARGUMENTS = ['--headless']

输出：

{'headline': 'B2 ', 'timestamp': '18:42:39-03-22', 'location': '.'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A1 BAARLE-NASSAU RIT: 37170 (DIRECTE INZET: JA)', 'timestamp': '18:42:22-03-22', 'location': '1220499   Monitor Regionale Ambulancevoorziening Midden- en West-Brabant'}      
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 CALLENBURGHPLANTSOEN VOORSC DIRECTE INZET 16185', 'timestamp': '18:42:15-03-22', 'location': '1523185   Ambulance-16-185 Hollands Midden'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 DP5 WESTLAND GALGEPAD NAALDW ', 'timestamp': '18:42:00-03-22', 'location': '.'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2  ', 'timestamp': '18:41:31-03-22', 'location': '0120999   Monitor Regionale Ambulancevoorziening Amsterdam-Amstelland'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 ', 'timestamp': '18:41:09-03-22', 'location': '1123128   Ambulance-22-128 Helmond'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A1 AMBU 17104 VERBOOMSTRAAT 3082JC ROTTERDAM ROTTDM BON 40895', 'timestamp': '18:40:43-03-22', 'location': '1420999   Monitor Regionale Ambulancevoorziening Rotterdam-Rijnmond'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 BEST RIT: 31448', 'timestamp': '18:40:20-03-22', 'location': '1123124   Ambulance-22-124 Helmond'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 MIDDELBURG RIT: 37168 (DIRECTE INZET: JA)', 'timestamp': '18:40:11-03-22', 'location': '1320101   Ambulance-19-101 Goes'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': None, 'timestamp': '18:39:49-03-22', 'location': '1420999   Monitor Regionale Ambulancevoorziening Rotterdam-Rijnmond'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'ONGEVAL MATERIEEL NIEUWE GRACHT HAARLEM', 'timestamp': '18:39:49-03-22', 'location': '0127850   Persinformatie Politie Kennemerland'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': None, 'timestamp': '18:39:48-03-22', 'location': '1420023   Ambulance-17-123 Rotterdam-Rijnmond'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 BAARLE-NASSAU RIT: 37167 (DIRECTE INZET: JA)', 'timestamp': '18:39:37-03-22', 'location': '1220646   Ambulance-20-146 Tilburg-Noord'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A1 EINDHOVEN RIT: 31447', 'timestamp': '18:39:29-03-22', 'location': '1123101   
Ambulance-22-101 Eindhoven'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'B1 NIEUWEGEIN 29751', 'timestamp': '18:39:26-03-22', 'location': '0726137   Ambulance-09-137 Amersfoort'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'B1 AMBU 06152 - AALTEN RIT 27711', 'timestamp': '18:38:22-03-22', 'location': '0820152   Ambulance-06-128 Noord- en Oost-Gelderland'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': None, 'timestamp': '18:38:01-03-22', 'location': '0108999   Monitor Brandweer Veiligheidsregio Kennemerland'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': None, 'timestamp': '18:37:12-03-22', 'location': '0520000   Regionaal Proefalarm 
GHOR Drenthe'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 DP1 HARNASCHPOLDER VRIJ-HARNASCH DENHZH ', 'timestamp': '18:37:12-03-22', 'location': '.'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 11134 RIT 38636 ', 'timestamp': '18:36:30-03-22', 'location': '0126999   Monitor Regionale Ambulancevoorziening Kennemerland'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A1 ZUIDERBEEKWEG 6862EM OOSTERBEEK 41964', 'timestamp': '18:36:28-03-22', 'location': '0920118   Ambulance-07-118 Barneveld'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 SOEST 29750', 'timestamp': '18:36:18-03-22', 'location': '0726104   Ambulance-09-148 Utrecht'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2 RAAMSDONKSVEER RIT: 37166', 'timestamp': '18:36:14-03-22', 'location': '1220626   Ambulance-20-126 Breda-Zuid'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A1 11125 RIT 38634 ELINE VEREPLANTSOEN ZAANDAM', 'timestamp': '18:35:20-03-22', 
'location': '0126999   Monitor Regionale Ambulancevoorziening Kennemerland'}
2022-03-21 23:43:07 [scrapy.core.scraper] DEBUG: Scraped from <200 http://ftp.112meldingen.nl/index.php>
{'headline': 'A2  ', 'timestamp': '18:35:07-03-22', 'location': '1423397   Ambulance-18-197 Papendrecht'}

硒请求

本文收集自互联网，转载请注明来源。

如有侵权，请联系 [email protected] 删除。

编辑于 2022-03-24

我来说两句

0 条评论

登录后参与评论

上一篇：键入放大后的警告消息“DeprecationWarning: Invalid 'main' field in .....”

如何使用 Scrapy 和 Selenium 从使用 javascript 和 php 的网站获取数据？

如何使用 Scrapy 和 Selenium 从使用 javascript 和 php 的网站获取数据？

UITableView的项目向下滚动后更改颜色，然后快速备份

Linux的官方Adobe Flash存储库是否已过时？

用日期数据透视表和日期顺序查询

应用发明者仅从列表中选择一个随机项一次

Mac OS X更新后的GRUB 2问题

验证REST API参数

Java Eclipse中的错误13，如何解决？

带有错误“ where”条件的查询如何返回结果？

ggplot：对齐多个分面图-所有大小不同的分面

尝试反复更改屏幕上按钮的位置 - kotlin android studio

如何从视图一次更新多行（ASP.NET - Core）

计算数据帧中每行的NA

蓝屏死机没有修复解决方案

在 Python 2.7 中。如何从文件中读取特定文本并分配给变量

离子动态工具栏背景色

VB.net将2条特定行导出到DataGridView

通过 Git 在运行 Jenkins 作业时获取 ClassNotFoundException

在Windows 7中无法删除文件（2）

python中的boto3文件上传

当我尝试下载 StanfordNLP en 模型时，出现错误

Node.js中未捕获的异常错误，发生调用