17. Concurrent Execution

The modules described in this chapter provide support for concurrent execution of code. The appropriate choice of tool will depend on the task to be executed (CPU bound vs IO bound) and preferred style of development (event driven cooperative multitasking vs preemptive multitasking). Here’s an overview:

• 17.1. threading — Thread-based parallelism
  • active_count()
  • current_thread()
  • get_ident()
  • enumerate()
  • main_thread()
  • settrace()
  • setprofile()
  • stack_size()
  • TIMEOUT_MAX
  • 17.1.1. Thread-Local Data
    • local
  • 17.1.2. Thread Objects
    • Thread
      • Thread.start()
      • Thread.run()
      • Thread.join()
      • Thread.name
      • Thread.getName()
      • Thread.setName()
      • Thread.ident
      • Thread.is_alive()
      • Thread.daemon
      • Thread.isDaemon()
      • Thread.setDaemon()
  • 17.1.3. Lock Objects
    • Lock
      • Lock.acquire()
      • Lock.release()
  • 17.1.4. RLock Objects
    • RLock
      • RLock.acquire()
      • RLock.release()
  • 17.1.5. Condition Objects
    • Condition
      • Condition.acquire()
      • Condition.release()
      • Condition.wait()
      • Condition.wait_for()
      • Condition.notify()
      • Condition.notify_all()
  • 17.1.6. Semaphore Objects
    • Semaphore
      • Semaphore.acquire()
      • Semaphore.release()
    • BoundedSemaphore
    • 17.1.6.1. Semaphore Example
  • 17.1.7. Event Objects
    • Event
      • Event.is_set()
      • Event.set()
      • Event.clear()
      • Event.wait()
  • 17.1.8. Timer Objects
    • Timer
      • Timer.cancel()
  • 17.1.9. Barrier Objects
    • Barrier
      • Barrier.wait()
      • Barrier.reset()
      • Barrier.abort()
      • Barrier.parties
      • Barrier.n_waiting
      • Barrier.broken
    • BrokenBarrierError
  • 17.1.10. Using locks, conditions, and semaphores in the with statement
• 17.2. multiprocessing — Process-based parallelism
  • 17.2.1. Introduction
    • 17.2.1.1. The Process class
    • 17.2.1.2. Contexts and start methods
    • 17.2.1.3. Exchanging objects between processes
    • 17.2.1.4. Synchronization between processes
    • 17.2.1.5. Sharing state between processes
    • 17.2.1.6. Using a pool of workers
  • 17.2.2. Reference
    • 17.2.2.1. Process and exceptions
      • Process
        • Process.run()
        • Process.start()
        • Process.join()
        • Process.name
        • Process.is_alive()
        • Process.daemon
        • Process.pid
        • Process.exitcode
        • Process.authkey
        • Process.sentinel
        • Process.terminate()
      • ProcessError
      • BufferTooShort
      • AuthenticationError
      • TimeoutError
    • 17.2.2.2. Pipes and Queues
      • Pipe()
      • Queue
        • Queue.qsize()
        • Queue.empty()
        • Queue.full()
        • Queue.put()
        • Queue.put_nowait()
        • Queue.get()
        • Queue.get_nowait()
        • Queue.close()
        • Queue.join_thread()
        • Queue.cancel_join_thread()
      • SimpleQueue
        • SimpleQueue.empty()
        • SimpleQueue.get()
        • SimpleQueue.put()
      • JoinableQueue
        • JoinableQueue.task_done()
        • JoinableQueue.join()
    • 17.2.2.3. Miscellaneous
      • active_children()
      • cpu_count()
      • current_process()
      • freeze_support()
      • get_all_start_methods()
      • get_context()
      • get_start_method()
      • set_executable()
      • set_start_method()
    • 17.2.2.4. Connection Objects
      • Connection
        • Connection.send()
        • Connection.recv()
        • Connection.fileno()
        • Connection.close()
        • Connection.poll()
        • Connection.send_bytes()
        • Connection.recv_bytes()
        • Connection.recv_bytes_into()
    • 17.2.2.5. Synchronization primitives
      • Barrier
      • BoundedSemaphore
      • Condition
      • Event
      • Lock
        • Lock.acquire()
        • Lock.release()
      • RLock
        • RLock.acquire()
        • RLock.release()
      • Semaphore
    • 17.2.2.6. Shared ctypes Objects
      • Value()
      • Array()
      • 17.2.2.6.1. The multiprocessing.sharedctypes module
        • RawArray()
        • RawValue()
        • Array()
        • Value()
        • copy()
        • synchronized()
    • 17.2.2.7. Managers
      • multiprocessing.Manager()
      • BaseManager
        • BaseManager.start()
        • BaseManager.get_server()
        • BaseManager.connect()
        • BaseManager.shutdown()
        • BaseManager.register()
        • BaseManager.address
      • SyncManager
        • SyncManager.Barrier()
        • SyncManager.BoundedSemaphore()
        • SyncManager.Condition()
        • SyncManager.Event()
        • SyncManager.Lock()
        • SyncManager.Namespace()
        • SyncManager.Queue()
        • SyncManager.RLock()
        • SyncManager.Semaphore()
        • SyncManager.Array()
        • SyncManager.Value()
        • SyncManager.dict()
        • SyncManager.list()
      • Namespace
      • 17.2.2.7.1. Customized managers
      • 17.2.2.7.2. Using a remote manager
    • 17.2.2.8. Proxy Objects
      • BaseProxy
        • BaseProxy._callmethod()
        • BaseProxy._getvalue()
        • BaseProxy.__repr__()
        • BaseProxy.__str__()
      • 17.2.2.8.1. Cleanup
    • 17.2.2.9. Process Pools
      • Pool
        • Pool.apply()
        • Pool.apply_async()
        • Pool.map()
        • Pool.map_async()
        • Pool.imap()
        • Pool.imap_unordered()
        • Pool.starmap()
        • Pool.starmap_async()
        • Pool.close()
        • Pool.terminate()
        • Pool.join()
      • AsyncResult
        • AsyncResult.get()
        • AsyncResult.wait()
        • AsyncResult.ready()
        • AsyncResult.successful()
    • 17.2.2.10. Listeners and Clients
      • deliver_challenge()
      • answer_challenge()
      • Client()
      • Listener
        • Listener.accept()
        • Listener.close()
        • Listener.address
        • Listener.last_accepted
      • wait()
      • 17.2.2.10.1. Address Formats
    • 17.2.2.11. Authentication keys
    • 17.2.2.12. Logging
      • get_logger()
      • log_to_stderr()
    • 17.2.2.13. The multiprocessing.dummy module
  • 17.2.3. Programming guidelines
    • 17.2.3.1. All start methods
    • 17.2.3.2. The spawn and forkserver start methods
  • 17.2.4. Examples
• 17.3. The concurrent package
• 17.4. concurrent.futures — Launching parallel tasks
  • 17.4.1. Executor Objects
    • Executor
      • Executor.submit()
      • Executor.map()
      • Executor.shutdown()
  • 17.4.2. ThreadPoolExecutor
    • ThreadPoolExecutor
    • 17.4.2.1. ThreadPoolExecutor Example
  • 17.4.3. ProcessPoolExecutor
    • ProcessPoolExecutor
    • 17.4.3.1. ProcessPoolExecutor Example
  • 17.4.4. Future Objects
    • Future
      • Future.cancel()
      • Future.cancelled()
      • Future.running()
      • Future.done()
      • Future.result()
      • Future.exception()
      • Future.add_done_callback()
      • Future.set_running_or_notify_cancel()
      • Future.set_result()
      • Future.set_exception()
  • 17.4.5. Module Functions
    • wait()
    • as_completed()
  • 17.4.6. Exception classes
    • CancelledError
    • TimeoutError
    • BrokenProcessPool
• 17.5. subprocess — Subprocess management
  • 17.5.1. Using the subprocess Module
    • run()
    • CompletedProcess
      • CompletedProcess.args
      • CompletedProcess.returncode
      • CompletedProcess.stdout
      • CompletedProcess.stderr
      • CompletedProcess.check_returncode()
    • DEVNULL
    • PIPE
    • STDOUT
    • SubprocessError
    • TimeoutExpired
      • TimeoutExpired.cmd
      • TimeoutExpired.timeout
      • TimeoutExpired.output
      • TimeoutExpired.stdout
      • TimeoutExpired.stderr
    • CalledProcessError
      • CalledProcessError.returncode
      • CalledProcessError.cmd
      • CalledProcessError.output
      • CalledProcessError.stdout
      • CalledProcessError.stderr
    • 17.5.1.1. Frequently Used Arguments
    • 17.5.1.2. Popen Constructor
      • Popen
    • 17.5.1.3. Exceptions
  • 17.5.2. Security Considerations
  • 17.5.3. Popen Objects
    • Popen.poll()
    • Popen.wait()
    • Popen.communicate()
    • Popen.send_signal()
    • Popen.terminate()
    • Popen.kill()
    • Popen.args
    • Popen.stdin
    • Popen.stdout
    • Popen.stderr
    • Popen.pid
    • Popen.returncode
  • 17.5.4. Windows Popen Helpers
    • STARTUPINFO
      • STARTUPINFO.dwFlags
      • STARTUPINFO.hStdInput
      • STARTUPINFO.hStdOutput
      • STARTUPINFO.hStdError
      • STARTUPINFO.wShowWindow
    • 17.5.4.1. Constants
      • STD_INPUT_HANDLE
      • STD_OUTPUT_HANDLE
      • STD_ERROR_HANDLE
      • SW_HIDE
      • STARTF_USESTDHANDLES
      • STARTF_USESHOWWINDOW
      • CREATE_NEW_CONSOLE
      • CREATE_NEW_PROCESS_GROUP
  • 17.5.5. Older high-level API
    • call()
    • check_call()
    • check_output()
  • 17.5.6. Replacing Older Functions with the subprocess Module
    • 17.5.6.1. Replacing /bin/sh shell backquote
    • 17.5.6.2. Replacing shell pipeline
    • 17.5.6.3. Replacing os.system()
    • 17.5.6.4. Replacing the os.spawn family
    • 17.5.6.5. Replacing os.popen(), os.popen2(), os.popen3()
    • 17.5.6.6. Replacing functions from the popen2 module
  • 17.5.7. Legacy Shell Invocation Functions
    • getstatusoutput()
    • getoutput()
  • 17.5.8. Notes
    • 17.5.8.1. Converting an argument sequence to a string on Windows
• 17.6. sched — Event scheduler
  • scheduler
  • 17.6.1. Scheduler Objects
    • scheduler.enterabs()
    • scheduler.enter()
    • scheduler.cancel()
    • scheduler.empty()
    • scheduler.run()
    • scheduler.queue
• 17.7. queue — A synchronized queue class
  • Queue
  • LifoQueue
  • PriorityQueue
  • Empty
  • Full
  • 17.7.1. Queue Objects
    • Queue.qsize()
    • Queue.empty()
    • Queue.full()
    • Queue.put()
    • Queue.put_nowait()
    • Queue.get()
    • Queue.get_nowait()
    • Queue.task_done()
    • Queue.join()

The following are support modules for some of the above services:

• 17.8. dummy_threading — Drop-in replacement for the threading module
• 17.9. _thread — Low-level threading API
  • error
  • LockType
  • start_new_thread()
  • interrupt_main()
  • exit()
  • allocate_lock()
  • get_ident()
  • stack_size()
  • TIMEOUT_MAX
  • lock.acquire()
  • lock.release()
  • lock.locked()
• 17.10. _dummy_thread — Drop-in replacement for the _thread module