Addressing multithreading and multiprocessing in transparent and Pythonic ways
[EuroPython 2018 - Talk - 2018-07-27 - PyCharm [PyData]]
[Edinburgh, UK]
By David Liu
With the increase in computing power, harnessing and controlling one’s code out of the single-threaded realm becomes an ever-increasing problem, coupled with the desire to stay in the Python layer. With the recent tools and frameworks that have been published, escaping the GIL cleanly is much easier than before, allow one’s Python code to effectively utilize multi-core and many core architectures in the most Pythonic ways possible. In this talk, learn about how to utilize static multiprocessing for process pinning, and effectively balancing thread pools with a monkey-patched import of threading modules.
Overview:
Introduction to multithreading and multiprocessing in Python
History of multithreading+multiprocessing in Python, classic frameworks
Problems that can occur (oversubscription, nested parallelism issues, process hopping, pool resource on shared machines)
Python accessing bigger hardware over the last few years (28+ cores, etc)
When to stay in the GIL, and when to escape it
The advantages and safety of the GIL
Python-level exiting of the GIL; analysis of when to return to single-threaded, and when threading is a deceivingly bad idea
Accountability of frameworks that natively exit the GIL
The new multithreading and multiprocessing libraries and techniques
static multiprocessing module (smp) (and monkey patching of multiprocessing)
thread pool control with command line calls of Python ( python -m tbb -p 8)
Putting it all together
Examples of using static multiprocessing on a large machine to stop oversubscription
Example of pseudo-daemon process on 4-core machine by processor pinning
Thread pool control on a simple NumPy example
Summary - Best practices for using above methods to control multithreading+multiprocessing
What needs to be done in the space (frameworks and things that need to be exposed)
Problems that still exist in the area
Q&A