Python Bytes: #197 Structured concurrency in Python

<p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href="https://training.talkpython.fm/"><strong>courses at Talk Python Training</strong></a></li> <li><a href="https://testandcode.com/">Test &amp; Code</a> Podcast</li> </ul> <p><strong>Michael #1:</strong> <a href="https://mattwestcott.co.uk/blog/structured-concurrency-in-python-with-anyio"><strong>Structured concurrency in Python with AnyIO</strong></a></p> <ul> <li><a href="https://github.com/agronholm/anyio">AnyIO</a> is a Python library providing structured concurrency primitives on top of asyncio.</li> <li><strong>Structured concurrency</strong> is a <a href="https://en.wikipedia.org/wiki/Programming_paradigm">programming paradigm</a> aimed at improving the clarity, quality, and development time of a <a href="https://en.wikipedia.org/wiki/Computer_program">computer program</a> by using a structured approach to <a href="https://en.wikipedia.org/wiki/Concurrent_computing">concurrent programming</a>. The core concept is the encapsulation of concurrent threads of execution (here encompassing kernel and userland threads and processes) by way of control flow constructs that have clear entry and exit points and that ensure all spawned threads have completed before exit. — Wikipedia</li> <li>The best overview is <a href="https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statement-considered-harmful/">Notes on structured concurrency</a> by Nathaniel Smith (or his <a href="https://www.youtube.com/watch?v=oLkfnc_UMcE">video</a> if you prefer).</li> <li>Python has three well-known concurrency libraries built around the async/await syntax: <a href="https://docs.python.org/3/library/asyncio.html">asyncio</a>, <a href="https://github.com/dabeaz/curio">Curio</a>, and <a href="https://github.com/python-trio/trio">Trio</a>. (WHERE IS <a href="https://asherman.io/projects/unsync.html">unsync</a>?!?! 🙂 )</li> <li>Since it's the default, the overwhelming majority of async applications and libraries are written with asyncio.</li> <li>The second and third are attempts to improve on asyncio, by David Beazley and Nathaniel Smith respectively</li> <li>The <a href="https://github.com/agronholm/anyio">AnyIO</a> library by Alex Grönholm describes itself as follows: &gt; an asynchronous compatibility API that allows applications and libraries written against it to run unmodified on asyncio, curio and trio.</li> </ul> <p>Example:</p> <pre><code> import anyio async def task(n): await anyio.sleep(n) async def main(): try: async with anyio.create_task_group() as tg: await tg.spawn(task, 1) await tg.spawn(task, 2) finally: # e.g. release locks print('cleanup') anyio.run(main) </code></pre> <ul> <li>AnyIO also provides other primitives to replace the native asyncio ones if you want to benefit from structured concurrency's cancellation semantics:</li> <li><a href="https://anyio.readthedocs.io/en/latest/synchronization.html">Synchronisation primitives (locks, events, conditions)</a></li> <li><a href="https://anyio.readthedocs.io/en/latest/streams.html">Streams (similar to queues)</a></li> <li><a href="https://anyio.readthedocs.io/en/latest/api.html#timeouts-and-cancellation">Timeouts (e.g.</a> <code>[move_on_after](https://ift.tt/35bheE2 href="https://anyio.readthedocs.io/en/latest/api.html#timeouts-and-cancellation">,</a> <code>[fail_after](https://ift.tt/35bheE2 href="https://anyio.readthedocs.io/en/latest/api.html#timeouts-and-cancellation">)</a></li> <li><a href="https://ift.tt/2Za2Vvy">... and more</a></li> </ul> <p><strong>Brian #2:</strong> <a href="https://data-apis.org/blog/announcing_the_consortium/"><strong>The Consortium for Python Data API Standards</strong></a></p> <ul> <li>One unintended consequence of the advances in multiple frameworks for data science, machine learning, deep learning and numerical computing is fragmentation and differences in common function signatures.</li> <li>The Consortium for Python Data API Standards aims to tackle this fragmentation by developing API standards for arrays (a.k.a. tensors) and dataframes. </li> <li>They intend to work with library maintainers and the community and have a review process.</li> <li>One example of the problem, “mean”. Five different interfaces over 8 frameworks:</li> </ul> <pre><code> numpy: mean(a, axis=None, dtype=None, out=None, keepdims=[HTML_REMOVED]) dask.array: mean(a, axis=None, dtype=None, out=None, keepdims=[HTML_REMOVED]) cupy: mean(a, axis=None, dtype=None, out=None, keepdims=False) jax.numpy: mean(a, axis=None, dtype=None, out=None, keepdims=False) mxnet.np: mean(a, axis=None, dtype=None, out=None, keepdims=False) sparse: s.mean(axis=None, keepdims=False, dtype=None, out=None) torch: mean(input, dim, keepdim=False, out=None) tensorflow: reduce_mean(input_tensor, axis=None, keepdims=None, name=None, reduction_indices=None, keep_dims=None) </code></pre> <ul> <li>They are going to start with array API</li> <li>Then dataframes</li> <li>Also, it’s happening fast, hoping to make traction in next few months.</li> </ul> <p><strong>Michael #3:</strong> <a href="https://switowski.com/blog/ask-for-permission-or-look-before-you-leap"><strong>Ask for Forgiveness or Look Before You Leap?</strong></a></p> <ul> <li>via PyCoders</li> <li>Think C++ style vs Python style of error handling </li> <li>Or any exception-first/only language vs. some hybrid thing</li> <li>If you “look before you leap”, you first check if everything is set correctly, then you perform an action.</li> <li>Example:</li> </ul> <pre><code> from pathlib import Path if Path("/path/to/file").exists(): ... </code></pre> <ul> <li>With “ask for forgiveness,” you don’t check anything. You perform whatever action you want, but you wrap it in a <code>try/catch</code> block.</li> </ul> <pre><code> try: with open("path/to/file.txt", "r") as input_file: return input_file.read() except IOError: # Handle the error or just ignore it </code></pre> <ul> <li>Their example, “Look before you leap” is around 30% slower (155/118≈1.314). Testing for subclass basically with no errors</li> <li>But if there are errors: The tables have turned. “Ask for forgiveness” is now over <strong>four times</strong> as slow as “Look before you leap” (562/135≈4.163). That’s because this time, our code throws an exception. And <strong>handling exceptions is expensive</strong>.</li> <li>If you expect your code to fail often, then “Look before you leap” might be much faster.</li> <li>Michael’s counter example: <a href="https://gist.github.com/mikeckennedy/00828db1d49d2cd2dac8fa0295e54c23">gist.github.com/mikeckennedy/00828db1d49d2cd2dac8fa0295e54c23</a></li> </ul> <p><strong>Brian #4:</strong> <a href="https://myrepos.branchable.com/"><strong>myrepos</strong></a></p> <ul> <li>“You have a lot of version control repositories. Sometimes you want to update them all at once. Or push out all your local changes. You use special command lines in some repositories to implement specific workflows. Myrepos provides a <code>mr</code> command, which is a tool to manage all your version control repositories.”</li> <li>Run <code>mr register</code> for all repos under a shared directory.</li> <li>Then be able to do common operations on a subtree of repos, like <code>mr status</code>, <code>mr update</code>, <code>mr diff</code>, or really anything.</li> <li>See also: <a href="https://adamj.eu/tech/2020/04/02/maintaining-multiple-python-projects-with-myrepos/"><strong>Maintaining Multiple Python Projects With myrepos</strong></a> <strong>-</strong> Adam Johnson</li> </ul> <p><strong>Michael #5:</strong> <a href="https://pythonspeed.com/articles/official-python-docker-image/"><strong>A deep dive into the official Docker image for Python</strong></a></p> <ul> <li>by <a href="#">Itamar Turner-Trauring</a>, via PyCoders</li> <li>Wait, there’s <a href="https://hub.docker.com/_/python">an official Docker image</a> for Python</li> <li>The base image is Debian GNU/Linux 10, the current stable release of the Debian distribution, also known as Buster because Debian names all their releases after characters from Toy Story</li> <li>Next, environment variables are added: <code>ENV PATH /usr/local/bin:$PATH</code></li> <li>Next, the locale is set: <code>ENV LANG C.UTF-8</code> </li> <li>There’s also an environment variable that tells you the current Python version: <code>ENV PYTHON_VERSION 3.8.5</code> </li> <li>In order to run, Python needs some additional packages (the dreaded certificates, etc)</li> <li>Next, a compiler toolchain is installed, Python source code is downloaded, Python is compiled, and then the unneeded Debian packages are uninstalled. Interestingly, The packages—<code>gcc</code> and so on—needed to compile Python are removed once they are no longer needed.</li> <li>Next, <code>/usr/local/bin/python3</code> gets an alias <code>/usr/local/bin/python</code>, so you can call it either way</li> <li>the <code>Dockerfile</code> makes sure to include that newer <code>pip</code></li> <li>Finally, the Dockerfile specifices the entrypoint: <code>CMD ["python3"]</code> Means docker run launches into the REPL:</li> </ul> <pre><code> $ docker run -it python:3.8-slim-buster Python 3.8.5 (default, Aug 4 2020, 16:24:08) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. &gt;&gt;&gt; </code></pre> <p><strong>Brian #6:</strong> <strong>“Only in a Pandemic” section</strong> <a href="https://www.ethanrosenthal.com/2020/08/25/optimal-peanut-butter-and-banana-sandwiches/"><strong>nannernest: Optimal Peanut Butter and Banana Sandwiches</strong></a></p> <ul> <li>Ethan Rosenthal</li> <li>Computer vision, deep learning, machine learning, and Python come together to make sandwiches.</li> <li>Just a really fun read about problems called “nesting” or “packing” and how to apply it to banana slices and bread.</li> </ul> <p>Extras:</p> <p>Brian:</p> <ul> <li><a href="https://www.patreon.com/pythonbytes">Patreon link</a></li> </ul> <p>Michael:</p> <ul> <li>Sign up for the free <a href="https://www.crowdcast.io/e/tips-and-techniques-to-move-from-excel-to-python">Excel to Python webcast on Sept 29</a>.</li> <li>Check out the early access version of <a href="https://talkpython.fm/mem">the memory course</a>.</li> </ul> <p>Joke</p> <p>via <a href="https://twitter.com/EduardoOrochena/status/1291427366867263489">Eduardo Orochena</a></p> <p><img src="https://ift.tt/3gWIMiA" alt="" /></p>

from Planet Python
via read more