How to set up Accept-Encoding to gzip in Python pyppeteer and print pyppeteer headers?

xin.chen
  1. how to set headers for pyppeteer for example:Accept-Encoding: gzip
  2. how to print pyppeteer headers in python. i know java
const response = await page.goto('https://example.org/')
console.log(response.headers)

result

{ date: 'Sun, 29 Oct 2017 13:35:59 GMT',
'content-encoding': 'gzip',
'last-modified': 'Fri, 09 Aug 2013 23:54:35 GMT',
server: 'ECS (lga/1318)',
etag: '"359670651+gzip"',
vary: 'Accept-Encoding',
'x-cache': 'HIT',
'content-type': 'text/html',
status: '200',
'cache-control': 'max-age=604800',
'content-length': '606',
expires: 'Sun, 05 Nov 2017 13:35:59 GMT' }
yupoog
import pyppeteer
import asyncio
from pyppeteer.network_manager import Request, Response

async def req_intercept(req: Request):
    print(f'Original header: {req.headers}')
    req.headers.update({'Accept-Encoding': 'gzip'})
    await req.continue_(overrides={'headers': req.headers})

async def resp_intercept(resp: Response):
    print(f"New header: {resp.request.headers}")


async def test():
    browser = await pyppeteer.launch()
    page = await browser.newPage()
    await page.setRequestInterception(True)
    page.on('request', req_intercept)
    page.on('response', resp_intercept)
    resp = await page.goto('https://example.org/')
    print(resp.headers)

asyncio.get_event_loop().run_until_complete(test())

result:

Original header: {'upgrade-insecure-requests': '1', 'user-agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'}
New header:      {'upgrade-insecure-requests': '1', 'user-agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8', 'Accept-Encoding': 'gzip'}
{'status': '200', 'content-encoding': 'gzip', 'accept-ranges': 'bytes', 'cache-control': 'max-age=604800', 'content-type': 'text/html; charset=UTF-8', 'date': 'Sat, 13 Apr 2019 03:07:49 GMT', 'etag': '"1541025663"', 'expires': 'Sat, 20 Apr 2019 03:07:49 GMT', 'last-modified': 'Fri, 09 Aug 2013 23:54:35 GMT', 'server': 'ECS (dcb/7F84)', 'vary': 'Accept-Encoding', 'x-cache': 'HIT', 'content-length': '606'}

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

In Python, how do I decode GZIP encoding?

Accept-Encoding: gzip on iOS

How to set up headers "h:Reply-To" with Mailgun php API

Python: Pyppeteer with asyncio

Python: Pyppeteer clicking on pop up window

Python: keep open browser in pyppeteer and create CDPSession

Missing content-encoding in Response Header for Request headers with accept-encoding: gzip, deflate, br (in that precise order) for HTTPS Requests

Accept gzip encoding using httr R

pyppeteer and javascript query

Scraping content using pyppeteer in association with asyncio

How to disable Images/CSS in Pyppeteer?

Pyppeteer. Chromium browser stops loading pages after set period of time

Pyppeteer: Browser closed unexpectedly in AWS Lambda

In response header Content-encoding: gzip is present, but in request header Accept-encoding: gzip, deflate is missing

x-gzip token in accept encoding header

PHP: how to use HTTP_ACCEPT_ENCODING to pass gzip acceptance when using fileget_contents()?

Nginx: how to set up multiple servers with accept_filter=httpready

How set up headers in ajax POST request to include CSRF token

Is there a way to scroll to end of page in pyppeteer

How to speed up of pocessing of gzip file in Python

How to fetch a url asynchronously with pyppeteer(One browser many tabs)

Pyppeteer (python) - clink a tag and after scraping the page

Using Pyppeteer to download CSV / Excel file from Vanguard via JavaScript

Issue puppeteer/pyppeteer headless doesn't render SSR page

Connect to a browser with Pyppeteer

How to set up hydra config to accept a custom enum?

pyppeteer.errors.ElementHandleError: Evaluation failed: SyntaxError: Unexpected token return

RuntimeError: Event loop is closed. Pyppeteer

How to correctly define Accept-Encoding: gzip header in RAML?