Python Concurrency and Future

#python/concurrency

This note describes how to use concurrency and future in python3. (Applies to Python3.7 and above)

Tutorial

Running Tasks

The concurrent.futures module provides a high-level interface for asynchronously executing callables. It supports both thread-based and process-based parallelism.

What is the difference between thread-based and process-based ?

Thread-based: parallel execution within the same process (process)
Process-based: parallel execution across multiple processor

Importing the required modules.

from concurrent.futures import ThreadPoolExecutor
import time

Submit tasks to the executor.

def my_task(seconds):
    print(f'Sleeping for {seconds} second(s)...')
    time.sleep(seconds)
    return f'Finished sleeping for {seconds} second(s)'

# Submitting tasks to the executor
with ThreadPoolExecutor() as executor:
	future_1 = executor.submit(my_task, 2)
	future_2 = executor.submit(my_task, 4)
Manipulate the future object !

future_1 = executor.submit(my_task, 2)

where future_1 is the returned future object. You can get the task status by the following methods.

  • task1.running() - is task running ?
  • task1.cancelled() - is task cancelled ?
  • task1.done() - is task done ?
  • task1.result() - result returned from the task !

Handle results from submitted tasks.

# Retrieving results from submitted tasks
print(task1.result())  # Blocking call - waits for task completion and returns result
print(task2.result())

Using as_completed to retrieve results as tasks complete

# Using as_completed to retrieve results as tasks complete
results = [executor.submit(my_task, seconds) for seconds in [2, 4, 1]]
for future in concurrent.futures.as_completed(results):
    print(future.result())

Mapping multiple arguments using map()

# Mapping multiple arguments using map()
results = executor.map(my_task, [2, 4, 1])
for result in results:
    print(result)

Or we can use ProcessPoolExecutor as the executor.

with concurrent.futures.ProcessPoolExecutor() as executor:
    # Your code here (similar to ThreadPoolExecutor)

That's it! You've now learned the basics of Python concurrency using the concurrent.futures module.

Remember to shutdown the executor !

Remember to handle exceptions and gracefully shutdown the executor by calling .shutdown() or using a context manager (with statement) when you're done with it.