y痒：蜘蛛退出所有错误产生之前没有错误消息

1。

您共享的GitHub上的代码a46b598可能与您在本地为示例作业提供的确切版本不同。例如，我还没有观察到日志行的任何行<timestamp> [guba] INFO: <url>。但是，我认为没有太大的区别。

2。

遇到任何问题时，建议将日志级别配置为DEBUG。

3。

如果您已将日志级别配置为DEBUG，则可能会看到类似以下内容的内容： 2018-10-26 15:25:09 [scrapy.downloadermiddlewares.redirect] DEBUG: Discarding <GET http://guba.eastmoney.com/topic,600000_22.html>: max redirections reached 更多行：https : //gist.github.com/starrify/b2483f0ed822a02d238cdf9d32dfa60e

That happens because you're passing the full response.meta dict to the following requests (related code), and Scrapy's RedirectMiddleware relies on some meta values (e.g. "redirect_times" and "redirect_ttl") to perform the check.

And the solution is simple: pass only the values you need into next_request.meta.

4.

It's also observed that you're trying to rotate the user agent strings, possibly for avoiding web crawl bans. But there's no other action taken. That would make your requests fishy still, because:

• Scrapy's cookie management is enabled by default, which would use a same cookie jar for all your requests.
• All your requests come from a same source IP address.

因此，我不确定对您来说正确地刮取整个站点是否足够好，尤其是当您不限制请求时。