I haven't found a solution to my specific issue on here, so here goes. I want to scrape news article metadata from different searches on the news site. I did this before for a single search, running the code over all the search result pages.
Here is my code:
urls = ["url1&page=", "url2&page=", "url3&page="]
for url in urls:
for page in range(2):
browser.get(url + str(page))
What I want to achieve is that it goes through all the page numbers for all the urls, but I get "TypeError: can only concatenate list (not "str") to list".
The desired output would be: "url1&page=0", "url1&page=1", "url2&page=0", "url2&page=1"... FYI I have about 10 urls like this and the range is 20 pages.
Thanks for any help!
Edit: here's the full error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-2-b0fdd2b9dc7f> in <module>
15 for page in range(21):
16
---> 17 browser.get(url + str(page))
18 #browser.get(f'{url}{page}')
19
TypeError: can only concatenate list (not "str") to list
Edit 2: thanks @balderman and @darkknight, it worked! I had another label called 'url' further down which was causing some additional issues but luckily I spotted it quickly :) Now looking at a beautiful csv of my data, thanks again!
Here you go:
NPAGES=2
urls = ["url1&page=", "url2&page=", "url3&page="]
for p in range(NPAGES):
for url in urls:
browser.get(f'{url}{p}')
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments