What's the fastest way to check thousands of urls?

To say that multiprocessing is always the best choice is incorrect, multiprocessing is best only for heavy computations!

The best choice for actions which do not require heavy computations, but only IN/OUT operations like database requets or requests of remote webapp api, is module threading. Threading can be faster than multiprocessing since multiprocessing need to serialize data to send it to child process, meanwhile trheads use the same memory stack.

Threading module

Typical activity in the case is to create input queue.Queue and put task (urls in you case in it) and create several workers to take tasks from the Queue:

import threading as thr
from queue import Queue


def work(input_q):
    """the function take task from input_q and print or return with some code changes (if you want)"""
    while True:
        item = input_q.get()
        if item == "STOP":
            break

        # else do some work here
        print("some result")


if __name__ == "__main__":
    input_q = Queue()
    urls = [...]
    threads_number = 8
    workers = [thr.Thread(target=work, args=(input_q,),) for i in range(threads_number)]
    # start workers here
    for w in workers:
        w.start

    # start delivering tasks to workers 
    for task in urls:
        input_q.put(task)

    # "poison pillow" for all workers to stop them:

    for i in range(threads_number):
        input_q.put("STOP")

    # join all workers to main thread here:

    for w in workers:
        w.join

    # show that main thread can continue

    print("Job is done.")