在英超联赛球员统计信息中应用过滤器时,我正在尝试复制请求。我注意到在为2019/20赛季应用过滤器时,URL添加了组件'?co = 1&se = 274'
https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274
代替
https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats
但是当做
requests.get('https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274')
然后抓取内容,就好像没有应用过滤器一样抓取内容。如何在应用网页过滤器的地方发出请求?
通过深入了解,我了解到它受到CloudFront的保护,这意味着在发布请求之前,将剥离所有查询参数。有没有解决的办法?
这是我如何抓取数据:
from bs4 import BeautifulSoup as soup
import requests
from tqdm import tqdm
from pprint import pprint
players_url =['https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274']
# this is dict where we store all information:
players = {}
for i in tqdm(players_url):
player_page = requests.get(i)
cont = soup(player_page.content, 'lxml')
time.sleep(2)
data = dict((k.contents[0].strip(), v.get_text(strip=True)) for k, v in zip(cont.select('.topStat span.stat, .normalStat span.stat'), cont.select('.topStat span.stat > span, .normalStat span.stat > span')))
clud_ele = cont.find('div', attrs={'class' : 'info'})
club = {"Club" : clud_ele.get_text(strip=True)}
position = {"Position": clud_ele.find_next('div', attrs={'class' : 'info'}).get_text(strip=True)}
data.update(club)
data.update(position)
players[cont.select_one('.playerDetails .name').get_text(strip=True)] = data
pprint(players)
在输出中,我可以清楚地看到未应用过滤器,因为本赛季没有45场比赛
{'Virgil van Dijk': {'Accurate long balls': '533',
'Aerial battles lost': '207',
'Aerial battles won': '589',
'Appearances': '122',
'Assists': '2',
'Big chances created': '11',
'Blocked shots': '23',
'Clean sheets': '45',
您可以通过尝试按季节过滤时复制完成的后台请求来解决此问题。我已经使用requests
库来获取所有玩家的统计信息。
这个过程主要涉及三个网址,
(ex. 274)
https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100
(ex. Name: Virgil van Dijk, ID: 5140)
https://footballapi.pulselive.com/football/players
(ex. 5140)
https://footballapi.pulselive.com/football/stats/player/
完整脚本
import requests
import json
response = requests.get('https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100').json() # request to obtain the id values and corresponding season
id = int(response["content"][0]["id"]) # converts current season id which is a decimal point value to interger
players = {} # dictionary to store players data
playersAndStats = {} # dictionary to store player name and associated stats
numEntries = 100
page = 0
# loop to get player name and id
while True:
params = (
('pageSize', '100'),
('compSeasons', str(id)),
('altIds', 'true'),
('page', str(page)),
('type', 'player'),
('id', '-1'),
('compSeasonId', str(id)),
)
response = requests.get('https://footballapi.pulselive.com/football/players',params=params).json()
playersData = response["content"]
for playerData in playersData:
players[playerData["id"]] = playerData["name"]["display"]
page += 1
if page == response["pageInfo"]["numPages"]:
break
print("Total no. of players :",len(players))
count = 0
total = len(players)
# loop to get player stats
for player in players:
count += 1
print(count,"/",total)
params = (
('comps', '1'),
('compSeasons', str(id)), # setting season id to current season id
)
playerId = str(int(player))
# gets the stat of the player using playerId
response = requests.get('https://footballapi.pulselive.com/football/stats/player/'+playerId,params=params).json()
stats = response["stats"]
# creating a stat dict for the player
playersAndStats[players[player]] = {}
# loop to store each stat associated with the player
for stat in stats:
playersAndStats[players[player]][stat["name"].replace("_"," ").capitalize()] = int(stat["value"])
# to store data to a json file
f = open("data.json","w")
# pretty prints and writes the same to the json file
f.write(json.dumps(playersAndStats,indent=4, sort_keys=True))
f.close()
print("Saved to data.json")
样本输出
"Aaron Connolly": {
"Accurate back zone pass": 1,
"Accurate fwd zone pass": 1,
"Accurate pass": 2,
"Aerial lost": 1,
"Appearances": 1,
"Attempts conceded ibox": 2,
"Attempts conceded obox": 1,
"Backward pass": 1,
"Ball recovery": 1,
"Duel lost": 1,
"Duel won": 0,
"Final third entries": 1,
"Goals conceded": 1,
"Goals conceded ibox": 1,
"Losses": 1,
"Mins played": 24,
"Open play pass": 1,
"Poss won mid 3rd": 1,
"Rightside pass": 1,
"Successful final third passes": 1,
"Successful open play pass": 1,
"Total back zone pass": 1,
"Total final third passes": 1,
"Total fwd zone pass": 1,
"Total offside": 2,
"Total pass": 2,
"Total sub on": 1,
"Total tackle": 0,
"Touches": 3,
"Won tackle": 0
}
data.json文件包含所有播放器的数据。
与每个玩家相关的统计信息根据他们的比赛位置而变化。
本文收集自互联网,转载请注明来源。
如有侵权,请联系 [email protected] 删除。
我来说两句