Trying to understand concurrent.futures.ThreadPoolExecutor

[Laxminarsaiah]

Hi There

I am trying to understand ThreadPoolExecutor api in python

Here is my code.

import concurrent.futures
import threading
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']


def load_url_one(url, timeout):
    print('Current thread in load_url_one()', threading.current_thread().getName())
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()


def load_url_two(url, timeout):
    print('Current thread in load_url_two()', threading.current_thread().getName())
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()


def get_data_from_load_url_one():
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        future_to_url = {executor.submit(load_url_one, url, 60): url for url in URLS}
        for future in concurrent.futures.as_completed(future_to_url):
            url = future_to_url[future]
            try:
                data = future.result()
            except Exception as exc:
                print('%r generated an exception: %s' % (url, exc))
            else:
                print('%r page is %d bytes' % (url, len(data)))


def get_data_from_load_url_two():
    with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
        future_to_url = {executor.submit(load_url_two, url, 60): url for url in URLS}
        for future in concurrent.futures.as_completed(future_to_url):
            url = future_to_url[future]
            try:
                data = future.result()
            except Exception as exc:
                print('%r generated an exception: %s' % (url, exc))
            else:
                print('%r page is %d bytes' % (url, len(data)))


# Now get_data_from_load_url_one() and get_data_from_load_url_two() functions will triggered in a
# loop
for i in range(1, 5):
    get_data_from_load_url_one()
    get_data_from_load_url_two()

# once i ran this output i can the output as below (click to open):

Hide/Show

Output:
Current thread in load_url_one() ThreadPoolExecutor-0_0
Current thread in load_url_one() ThreadPoolExecutor-0_1
Current thread in load_url_one() ThreadPoolExecutor-0_2
Current thread in load_url_one() ThreadPoolExecutor-0_3
Current thread in load_url_one() ThreadPoolExecutor-0_4
'http://some-made-up-domain.com/' generated an exception: <urlopen error [Errno 11001] getaddrinfo failed>
'http://www.foxnews.com/' page is 234938 bytes
'http://www.cnn.com/' page is 1132646 bytes
'http://www.bbc.co.uk/' page is 246255 bytes
'http://europe.wsj.com/' page is 2906320 bytes
Current thread in load_url_two() ThreadPoolExecutor-1_0
Current thread in load_url_two() ThreadPoolExecutor-1_1
Current thread in load_url_two() ThreadPoolExecutor-1_2
Current thread in load_url_two() ThreadPoolExecutor-1_3
Current thread in load_url_two() ThreadPoolExecutor-1_4
'http://some-made-up-domain.com/' generated an exception: <urlopen error [Errno 11001] getaddrinfo failed>
'http://www.foxnews.com/' page is 234938 bytes
'http://www.cnn.com/' page is 1132646 bytes
'http://www.bbc.co.uk/' page is 246257 bytes
'http://europe.wsj.com/' page is 2906216 bytes
Current thread in load_url_one() ThreadPoolExecutor-2_0
Current thread in load_url_one() ThreadPoolExecutor-2_1
Current thread in load_url_one() ThreadPoolExecutor-2_2
Current thread in load_url_one() ThreadPoolExecutor-2_3
Current thread in load_url_one() ThreadPoolExecutor-2_4
'http://some-made-up-domain.com/' generated an exception: <urlopen error [Errno 11001] getaddrinfo failed>
'http://www.foxnews.com/' page is 234939 bytes
'http://www.cnn.com/' page is 1132646 bytes
'http://europe.wsj.com/' page is 2907443 bytes
'http://www.bbc.co.uk/' page is 246277 bytes
Current thread in load_url_two() ThreadPoolExecutor-3_0
Current thread in load_url_two() ThreadPoolExecutor-3_1
Current thread in load_url_two() ThreadPoolExecutor-3_2
Current thread in load_url_two() ThreadPoolExecutor-3_3
Current thread in load_url_two() ThreadPoolExecutor-3_4
'http://some-made-up-domain.com/' generated an exception: <urlopen error [Errno 11001] getaddrinfo failed>
'http://www.foxnews.com/' page is 234939 bytes
'http://www.cnn.com/' page is 1132646 bytes
'http://europe.wsj.com/' page is 2906118 bytes
'http://www.bbc.co.uk/' page is 246277 bytes
Current thread in load_url_one() ThreadPoolExecutor-4_0
Current thread in load_url_one() ThreadPoolExecutor-4_1
Current thread in load_url_one() ThreadPoolExecutor-4_2
Current thread in load_url_one() ThreadPoolExecutor-4_3
Current thread in load_url_one() ThreadPoolExecutor-4_4
'http://some-made-up-domain.com/' generated an exception: <urlopen error [Errno 11001] getaddrinfo failed>
'http://www.foxnews.com/' page is 234939 bytes
'http://www.cnn.com/' page is 1132646 bytes
'http://www.bbc.co.uk/' page is 246255 bytes
'http://europe.wsj.com/' page is 2898463 bytes
Current thread in load_url_two() ThreadPoolExecutor-5_0
Current thread in load_url_two() ThreadPoolExecutor-5_1
Current thread in load_url_two() ThreadPoolExecutor-5_2
Current thread in load_url_two() ThreadPoolExecutor-5_3
Current thread in load_url_two() ThreadPoolExecutor-5_4
'http://some-made-up-domain.com/' generated an exception: <urlopen error [Errno 11001] getaddrinfo failed>
'http://www.foxnews.com/' page is 234939 bytes
'http://www.cnn.com/' page is 1132646 bytes
'http://www.bbc.co.uk/' page is 246277 bytes
'http://europe.wsj.com/' page is 2905334 bytes
Current thread in load_url_one() ThreadPoolExecutor-6_0
Current thread in load_url_one() ThreadPoolExecutor-6_1
Current thread in load_url_one() ThreadPoolExecutor-6_2
Current thread in load_url_one() ThreadPoolExecutor-6_3
Current thread in load_url_one() ThreadPoolExecutor-6_4
'http://some-made-up-domain.com/' generated an exception: <urlopen error [Errno 11001] getaddrinfo failed>
'http://www.foxnews.com/' page is 234939 bytes
'http://www.cnn.com/' page is 1132646 bytes
'http://www.bbc.co.uk/' page is 246276 bytes
'http://europe.wsj.com/' page is 2897096 bytes
Current thread in load_url_two() ThreadPoolExecutor-7_0
Current thread in load_url_two() ThreadPoolExecutor-7_1
Current thread in load_url_two() ThreadPoolExecutor-7_2
Current thread in load_url_two() ThreadPoolExecutor-7_3
Current thread in load_url_two() ThreadPoolExecutor-7_4
'http://some-made-up-domain.com/' generated an exception: <urlopen error [Errno 11001] getaddrinfo failed>
'http://www.foxnews.com/' page is 234938 bytes
'http://www.cnn.com/' page is 1132646 bytes
'http://europe.wsj.com/' page is 2907057 bytes
'http://www.bbc.co.uk/' page is 246257 bytes

From this output i am trying to understand is that,
Q1. What is mean by
ThreadPoolExecutor-0_0,ThreadPoolExecutor-0_1,ThreadPoolExecutor-0_2......
ThreadPoolExecutor-1_0,ThreadPoolExecutor-1_1,ThreadPoolExecutor-1_2......
ThreadPoolExecutor-2_0,ThreadPoolExecutor-2_1,ThreadPoolExecutor-2_2......
ThreadPoolExecutor-3_0,ThreadPoolExecutor-1_1,ThreadPoolExecutor-3_2......
Q2. is it really running parallelly?
Q3. are multiple ThreadPools getting created?

can any one please make me understand ThreadPoolExecutor concept clearly.

Thanks
Laxmi