在网页上应用过滤器时如何复制请求?

先生黄油

英超联赛球员统计信息中应用过滤器时,我正在尝试复制请求我注意到在为2019/20赛季应用过滤器时,URL添加了组件'?co = 1&se = 274'

https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274

代替

https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats

但是当做

requests.get('https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274')

然后抓取内容,就好像没有应用过滤器一样抓取内容。如何在应用网页过滤器的地方发出请求?

通过深入了解,我了解到它受到CloudFront的保护,这意味着在发布请求之前,将剥离所有查询参数。有没有解决的办法?

这是我如何抓取数据:

from bs4 import BeautifulSoup as soup
import requests
from tqdm import tqdm
from pprint import pprint
players_url =['https://www.premierleague.com//players/5140/Virgil-van-Dijk/stats?co=1&se=274']


# this is dict where we store all information:
players = {}
for i in tqdm(players_url):
    player_page = requests.get(i)
    cont = soup(player_page.content, 'lxml')
    time.sleep(2)
    data = dict((k.contents[0].strip(), v.get_text(strip=True)) for k, v in zip(cont.select('.topStat span.stat, .normalStat span.stat'), cont.select('.topStat span.stat > span, .normalStat span.stat > span')))
    clud_ele = cont.find('div', attrs={'class' : 'info'})
    club = {"Club" : clud_ele.get_text(strip=True)}
    position = {"Position": clud_ele.find_next('div', attrs={'class' : 'info'}).get_text(strip=True)}
    data.update(club)
    data.update(position)
    players[cont.select_one('.playerDetails .name').get_text(strip=True)] = data

pprint(players)

在输出中,我可以清楚地看到未应用过滤器,因为本赛季没有45场比赛

{'Virgil van Dijk': {'Accurate long balls': '533',
                     'Aerial battles lost': '207',
                     'Aerial battles won': '589',
                     'Appearances': '122',
                     'Assists': '2',
                     'Big chances created': '11',
                     'Blocked shots': '23',
                     'Clean sheets': '45',
CodeIt

您可以通过尝试按季节过滤时复制完成的后台请求来解决此问题。我已经使用requests库来获取所有玩家的统计信息。

这个过程主要涉及三个网址,

  1. 获取季节ID值。 (ex. 274)

https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100

  1. 获取玩家名称和ID (ex. Name: Virgil van Dijk, ID: 5140)

https://footballapi.pulselive.com/football/players

  1. 使用玩家ID获取玩家统计信息 (ex. 5140)

https://footballapi.pulselive.com/football/stats/player/

完整脚本

import requests
import json

response = requests.get('https://footballapi.pulselive.com/football/competitions/1/compseasons?page=0&pageSize=100').json() # request to obtain the id values and corresponding season 

id = int(response["content"][0]["id"]) # converts current season id which is a decimal point value to interger

players = {} # dictionary to store players data

playersAndStats = {} # dictionary to store player name and associated stats

numEntries = 100

page = 0

# loop to get player name and id
while True: 

  params = (
      ('pageSize', '100'),
      ('compSeasons', str(id)), 
      ('altIds', 'true'),
      ('page', str(page)),
      ('type', 'player'),
      ('id', '-1'),
      ('compSeasonId', str(id)),
  )

  response = requests.get('https://footballapi.pulselive.com/football/players',params=params).json()

  playersData = response["content"]

  for playerData in playersData:
    players[playerData["id"]] = playerData["name"]["display"]

  page += 1

  if page == response["pageInfo"]["numPages"]:
    break

print("Total no. of players :",len(players))

count = 0 
total = len(players)

# loop to get player stats 
for player in players:

  count += 1
  print(count,"/",total)

  params = (
      ('comps', '1'),
      ('compSeasons', str(id)), # setting season id to current season id
  )

  playerId = str(int(player))

  # gets the stat of the player using playerId 
  response = requests.get('https://footballapi.pulselive.com/football/stats/player/'+playerId,params=params).json()

  stats = response["stats"]

  # creating a stat dict for the player
  playersAndStats[players[player]] = {}

  # loop to store each stat associated with the player
  for stat in stats:
    playersAndStats[players[player]][stat["name"].replace("_"," ").capitalize()] = int(stat["value"])

# to store data to a json file 
f = open("data.json","w")

# pretty prints and writes the same to the json file 
f.write(json.dumps(playersAndStats,indent=4, sort_keys=True))
f.close()

print("Saved to data.json")

样本输出

"Aaron Connolly": {
        "Accurate back zone pass": 1,
        "Accurate fwd zone pass": 1,
        "Accurate pass": 2,
        "Aerial lost": 1,
        "Appearances": 1,
        "Attempts conceded ibox": 2,
        "Attempts conceded obox": 1,
        "Backward pass": 1,
        "Ball recovery": 1,
        "Duel lost": 1,
        "Duel won": 0,
        "Final third entries": 1,
        "Goals conceded": 1,
        "Goals conceded ibox": 1,
        "Losses": 1,
        "Mins played": 24,
        "Open play pass": 1,
        "Poss won mid 3rd": 1,
        "Rightside pass": 1,
        "Successful final third passes": 1,
        "Successful open play pass": 1,
        "Total back zone pass": 1,
        "Total final third passes": 1,
        "Total fwd zone pass": 1,
        "Total offside": 2,
        "Total pass": 2,
        "Total sub on": 1,
        "Total tackle": 0,
        "Touches": 3,
        "Won tackle": 0
    }

data.json文件包含所有播放器的数据。

与每个玩家相关的统计信息根据他们的比赛位置而变化。

本文收集自互联网,转载请注明来源。

如有侵权,请联系 [email protected] 删除。

编辑于
0

我来说两句

0 条评论
登录 后参与评论

相关文章

页面加载时如何应用过滤器?

在 r 上应用过滤器时 ifelse 不起作用

使用 searchview 应用过滤器时 RecyclerView 上的空白输出

在数据帧上应用过滤器时出错 - PySpark

如何使用EspoCRM REST API在datetime属性上使用过滤器进行GET请求?

在Hibernate的链接表上应用过滤器

在数据帧上应用过滤器函数

必要时重新应用过滤器

应用过滤器时更正搜索语句

应用过滤器时Meshlab崩溃

应用过滤器时,StreamingMessageSource不断触发

如何在python中的2D数组/列表上应用过滤器功能

如何使用快速测量MoM%并在视觉上应用过滤器以排除当月

如何在 chrome.windows.onFocusChanged 上应用过滤器

在表上应用过滤器后如何将数据绑定到下拉列表

如何在 jsonb 对象数组上应用过滤器 - 聚合后?

如何在 django ManyToManyField 上应用过滤器,以便字段的多个值遵循条件?

如何在字典数组中的内部数组上应用过滤器?

如何在数据透视表上应用过滤器?

无论如何在btrfs子卷快照时应用过滤器?

在 JavaScript 中为 ArcGis 使用对象时如何为多个图层应用过滤器

仅当存在嵌套映射时如何应用过滤器

如何组成两列以应用过滤器?

如何在Vuejs组件中应用过滤器?

应用过滤器后,如何更新个人计数?

弹性搜索中如何应用过滤器?

如何在PowerBI WebApp中应用过滤器

如何使用Softlayer的票证API应用过滤器?

如何一起应用过滤器?