Python: urllib.error.HTTPError: HTTP Error 525: Origin SSL Handshake Error

Tom

I am using Python 3 to crawl many web pages on one website with urllib.request.build_opener. Each web_page_url is opened like below:

_masterOpener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(CookieJar()))
_masterOpener.addheaders = [('Cookie', some_cookie)]
request = _masterOpener.open(web_page_url)
content = request.read()

It always works smoothly when crawling the first hundreds of pages for about 10 minutes (I tried a few times), and then an error like below occurs:

File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 525: Origin SSL Handshake Error

I searched on the Web and failed to find a solution. How to solve the problem 'urllib.error.HTTPError: HTTP Error 525: Origin SSL Handshake Error' as described?

plaes

HTTP Status 5xx errors indicate an error in the server, and it is your responsibility to handle them gracefully (for example, not to crash your crawler).

In this case, the error 525 issue seems to be CloudFlare-specific, where connection to original site via CloudFlare has timed out.

So just add the try...except clause to handle this error gracefully:

try:
    _masterOpener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(CookieJar()))
    _masterOpener.addheaders = [('Cookie', some_cookie)]
    request = _masterOpener.open(web_page_url)
    content = request.read()
except urllib.error.HTTPError as e:
    # Possible issue with CloudFlare, just fall through
    if e.code == 525:
        # TODO: Log warning about broken url
        pass
    # TODO: ... handle all the other 5xx errors
    # Raise the original exception
    raise

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Nginx with Cloudflare: Error 525 SSL Handshake failed

urllib.error.HTTPError: HTTP Error 503: Service Unavailable python

Python: urllib.error.HTTPError: HTTP Error 404: Not Found

Python urllib.error.HTTPError: HTTP Error 404: Not Found

urllib.error.HTTPError: HTTP Error 502: Bad Gateway PYTHON

python urllib.error.HTTPError

Python: urllib2.HTTPError: HTTP Error 300: Multiple Choices

urllib2.HTTPError: HTTP Error 400: Bad Request - Python

Pytube error: urllib.error.HTTPError: HTTP Error 404: Not Found

Pytube : urllib.error.HTTPError: HTTP Error 410: Gone

Again urllib.error.HTTPError: HTTP Error 400: Bad Request

Beautiful Soup - urllib.error.HTTPError: HTTP Error 403: Forbidden

urllib.error.HTTPError: HTTP Error 403: Forbidden for urlretrieve

urllib.error.HTTPError: HTTP Error 404: Not Found

urllib.error.HTTPError: HTTP Error 404: Not Found yfinance library

urllib.error.HTTPError: HTTP Error 404: Not Found Python while scraping data from Metacritic

Python 3.6.2 url.request.urlopen() urllib.error.HTTPError: HTTP Error 403: Forbidden

urllib.error.HTTPError: HTTP Error 400: Bad Request in Python function

JSON from webpage into Python script: urllib.error.HTTPError: HTTP Error 403: Forbidden

urllib2.HTTPError: HTTP Error 403: Forbidden

urllib2.HTTPError: HTTP Error 403: Forbidden

Python urllib2.HTTPError: HTTP Error 503: Service Unavailable on valid website

urllib.error.HTTPError: HTTP Error 403: Forbidden with urllib.requests

How to fix error urllib.error.HTTPError: HTTP Error 400: BAD REQUEST?

pytube error while downloading video: urllib.error.HTTPError: HTTP Error 404: Not Found

BioPython "urllib.error.HTTPError: HTTP Error 400: Bad Request" error message

SSL Error: Bad handshake

SSL handshake error

SSL Error: Bad handshake (Python requests)