Below is a program that makes multiple get requests and writes the response images to my directory. These get requests are meant to be in separate threads, and thus be quicker than w/o threads but I'm not seeing the performance difference.
Printing active_count() shows there are 9 threads created. However, the performance time still takes around 40 seconds whether or not I use threading.
Below is me using threading.
from threading import active_count
import requests
import time
import concurrent.futures
img_urls = [
'https://images.unsplash.com/photo-1516117172878-fd2c41f4a759',
'https://images.unsplash.com/photo-1532009324734-20a7a5813719',
'https://images.unsplash.com/photo-1524429656589-6633a470097c',
'https://images.unsplash.com/photo-1530224264768-7ff8c1789d79',
'https://images.unsplash.com/photo-1564135624576-c5c88640f235',
'https://images.unsplash.com/photo-1541698444083-023c97d3f4b6',
'https://images.unsplash.com/photo-1522364723953-452d3431c267',
'https://images.unsplash.com/photo-1513938709626-033611b8cc03',
'https://images.unsplash.com/photo-1507143550189-fed454f93097',
'https://images.unsplash.com/photo-1493976040374-85c8e12f0c0e',
'https://images.unsplash.com/photo-1504198453319-5ce911bafcde',
'https://images.unsplash.com/photo-1530122037265-a5f1f91d3b99',
'https://images.unsplash.com/photo-1516972810927-80185027ca84',
'https://images.unsplash.com/photo-1550439062-609e1531270e',
'https://images.unsplash.com/photo-1549692520-acc6669e2f0c'
]
t1 = time.perf_counter()
def download_image(img_url):
img_bytes = requests.get(img_url).content
img_name = img_url.split('/')[3]
img_name = f'{img_name}.jpg'
with open(img_name, 'wb') as img_file:
img_file.write(img_bytes)
print(f'{img_name} was downloaded...')
with concurrent.futures.ThreadPoolExecutor() as executor:
executor.map(download_image, img_urls)
print(active_count())
t2 = time.perf_counter()
print(f'Finished in {t2-t1} seconds')
Below is without threading
def download_image(img_url):
img_bytes = requests.get(img_url).content
img_name = img_url.split('/')[3]
img_name = f'{img_name}.jpg'
with open(img_name, 'wb') as img_file:
img_file.write(img_bytes)
print(f'{img_name} was downloaded...')
for img_url in img_urls:
download_image(img_url)
Could someone explain why this is happening? Thanks
This is the result i got with your piece of code, with start and end time next to the download. The overall time is around the same (on my "normal network", not the slow one i talked in my comment)
The reason is that multiple thread doesn't increase I/O or bandwith, the limitation could also be the website itself. This looks like the issue is not from your code.
EDIT (misleading statement) : as mentionned by MisterMiyagi in the comment below (read his comment, he explain why), it should increase I/O, that's the reason i get 10s increase on a slow network (limited connection on my work lab). This doesn't increase the I/O or bandwith in that specific case (with full bandwith on my "normal" connection), and this may be from a lot of source, but in my opinion, not the code itself.
I also tried with max_workers=5, the same overall time appears.
photo-1516117172878-fd2c41f4a759.jpg was downloaded... 1.0464828 - 1.7136098
photo-1532009324734-20a7a5813719.jpg was downloaded... 1.7140197 - 5.6327612
photo-1524429656589-6633a470097c.jpg was downloaded... 5.6339666 - 8.3146478
photo-1530224264768-7ff8c1789d79.jpg was downloaded... 8.3160157 - 10.474087
photo-1564135624576-c5c88640f235.jpg was downloaded... 10.4749598 - 11.2431941
photo-1541698444083-023c97d3f4b6.jpg was downloaded... 11.2436369 - 15.6939695
photo-1522364723953-452d3431c267.jpg was downloaded... 15.6954112 - 18.3257819
photo-1513938709626-033611b8cc03.jpg was downloaded... 18.3269668 - 21.0607191
photo-1507143550189-fed454f93097.jpg was downloaded... 21.0621265 - 22.2371699
photo-1493976040374-85c8e12f0c0e.jpg was downloaded... 22.2375931 - 26.4375676
photo-1504198453319-5ce911bafcde.jpg was downloaded... 26.4393404 - 28.3477933
photo-1530122037265-a5f1f91d3b99.jpg was downloaded... 28.348679 - 30.4626719
photo-1516972810927-80185027ca84.jpg was downloaded... 30.4636931 - 32.2621345
photo-1550439062-609e1531270e.jpg was downloaded... 32.2628976 - 34.7331719
photo-1549692520-acc6669e2f0c.jpg was downloaded... 34.7341393 - 35.5910094
Finished in 34.545366900000005 seconds
21
photo-1516117172878-fd2c41f4a759.jpg was downloaded... 35.5960486 - 46.1692758
photo-1564135624576-c5c88640f235.jpg was downloaded... 35.6110777 - 47.3780254
photo-1507143550189-fed454f93097.jpg was downloaded... 35.6265503 - 47.4433963
photo-1549692520-acc6669e2f0c.jpg was downloaded... 35.6692061 - 49.7097683
photo-1516972810927-80185027ca84.jpg was downloaded... 35.6420564 - 57.2326763
photo-1504198453319-5ce911bafcde.jpg was downloaded... 35.6340008 - 61.4597509
photo-1550439062-609e1531270e.jpg was downloaded... 35.6637577 - 62.0488296
photo-1530224264768-7ff8c1789d79.jpg was downloaded... 35.6072146 - 63.4139648
photo-1513938709626-033611b8cc03.jpg was downloaded... 35.6223106 - 63.8149815
photo-1524429656589-6633a470097c.jpg was downloaded... 35.6032493 - 63.8284464
photo-1530122037265-a5f1f91d3b99.jpg was downloaded... 35.6352735 - 65.0513042
photo-1522364723953-452d3431c267.jpg was downloaded... 35.6182243 - 65.5005548
photo-1532009324734-20a7a5813719.jpg was downloaded... 35.5994888 - 66.2930857
photo-1541698444083-023c97d3f4b6.jpg was downloaded... 35.6144996 - 67.8115219
photo-1493976040374-85c8e12f0c0e.jpg was downloaded... 35.6301133 - 68.5357319
Finished in 32.946069800000004 seconds
EDIT 2 (more testing) : I tried with one of my webserver (Same code, just different image list), and I got an overall decrease of 60-70% of downloading time. Work best with limited workers in that case. The problem come from the website, not your code.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments