我正在努力构建具有结构化为不同url的现有代码的代理旋转器。
下面的代码示例中提供了我想要的URL。我试图让提供的脚本调用所需的URL,并在代理类型为“ HTTPS”时获取所有的“ IP:PORT”(当前脚本限制为十个)。
可以在xpath或bs4中完成。我更喜欢bs4。
我理解逻辑,但是在如何构造它上却失败了。首先,我尝试剥离字符串并尝试调用特定的td元素,但是它不起作用。
#URLs I want
url_list = ['http://spys.one/free-proxy-list/US/','http://spys.one/free-proxy-list/US/1/']
#code I have
from lxml.html import fromstring
import requests
from itertools import cycle
import traceback
def get_proxies():
url = 'https://free-proxy-list.net/'
response = requests.get(url)
parser = fromstring(response.text)
proxies = set()
for i in parser.xpath('//tbody/tr')[:10]:
if i.xpath('.//td[7][contains(text(),"yes")]'):
proxy = ":".join([i.xpath('.//td[1]/text()')[0], i.xpath('.//td[2]/text()')[0]])
proxies.add(proxy)
return proxies
proxies = get_proxies()
proxy_pool = cycle(proxies)
proxy = next(proxy_pool)
response = requests.get(url,proxies={"http": proxy, "https": proxy})
我希望了解提供的代码是如何针对2个所需的URL构造的,当代理类型为HTTPS时返回所有IP:PORT号
一种方法是在循环中发出特定于端口的POST请求。您可以修改以添加到一个最终列表中。端点已经是https专用的。
import requests
from bs4 import BeautifulSoup as bs
def get_proxies(number, port, p):
r = requests.post('http://spys.one/en/https-ssl-proxy/', data = {'xpp': 5, 'xf4': number})
proxies = [':'.join([str(i),port]) for i in p.findall(r.text)]
return proxies
ports = ['3128', '8080', '80']
p = re.compile(r'spy14>(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})<script')
proxies = []
for number, port in enumerate(ports,1):
proxies+=get_proxies(number, port, p)
print(proxies)
结果示例:
针对特定国家/地区:
import requests
from bs4 import BeautifulSoup as bs
def get_proxies(number, port, p, country):
r = requests.post('http://spys.one/en/https-ssl-proxy/', data = {'xpp': 5, 'xf4': number})
soup = bs(r.content, 'lxml')
proxies = [':'.join([p.findall(i.text)[0], port]) for i in soup.select('table table tr:has(.spy14:contains("' + country + '")) td:has(script) .spy14')]
return proxies
ports = ['3128', '8080', '80']
p = re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})document')
proxies = []
for number, port in enumerate(ports,1):
proxies+=get_proxies(number, port, p, 'United States')
print(proxies)
对于您所说的内容,我将引用我的原始答案:
from bs4 import BeautifulSoup as bs
import requests
def get_proxies():
r = requests.get('https://free-proxy-list.net/')
soup = bs(r.content, 'lxml')
proxies = {tr.td.text + ':' + tr.td.next_sibling.text for tr in soup.select('tr:has(.hx:contains(yes))')}
return proxies
get_proxies()
本文收集自互联网,转载请注明来源。
如有侵权,请联系 [email protected] 删除。
我来说两句