Python Bytes: #184 Too many ways to wait with await?

<p>Sponsored by <strong>DigitalOcean</strong>: <a href="https://pythonbytes.fm/digitalocean"><strong>pythonbytes.fm/digitalocean</strong></a> - $100 credit for new users to build something awesome.</p> <p><strong>Michael #1:</strong> <a href="https://hynek.me/articles/waiting-in-asyncio/"><strong>Waiting in asyncio</strong></a></p> <ul> <li>by <a href="https://hynek.me/">Hynek Schlawack</a></li> <li>One of the main appeals of using Python’s <code>asyncio</code> is being able to fire off many coroutines and run them concurrently. How many ways do you know for waiting for their results?</li> <li>The simplest case is to <em>await</em> your coroutines:</li> </ul> <pre><code> result_f = await f() result_g = await g() </code></pre> <ul> <li>Drawbacks: <ol> <li>The coroutines do <strong>not</strong> run concurrently. <code>g</code> only starts executing after <code>f</code> has finished.</li> <li>You can’t cancel them once you started awaiting.</li> </ol></li> <li><code>[asyncio.Task](https://ift.tt/2Xytv0W href="https://docs.python.org/3/library/asyncio-task.html#asyncio.Task">s</a> wrap your coroutines and get independently <em>scheduled</em> for execution by the event loop whenever you yield control to it</li> </ul> <pre><code> task_f = asyncio.create_task(f()) task_g = asyncio.create_task(g()) await asyncio.sleep(0.1) # &lt;- f() and g() are already running! result_f = await task_f result_g = await task_g </code></pre> <ul> <li>Your tasks now run <em>concurrently</em> and if you decide that you don’t want to wait for <code>task_f</code> or <code>task_g</code> to finish, you can cancel them using <code>task_f.cancel()</code></li> <li><code>[asyncio.gather()](https://ift.tt/3gWJZHH> takes 1 or more awaitables as <code>*args</code>, wraps them in tasks if necessary, and waits for all of them to finish. Then it returns the <strong>results</strong> of all awaitables <strong>in the same order</strong></li> </ul> <pre><code> result_f, result_g = await asyncio.gather(f(), g()) </code></pre> <ul> <li><code>[asyncio.wait_for()](https://ift.tt/30iDiKt> allows for passing a time out</li> <li>A more elegant approach to timeouts is the <a href="https://pypi.org/project/async-timeout/"><em>async-timeout</em></a> <a href="https://pypi.org/project/async-timeout/">package</a> on PyPI. It gives you an asynchronous context manager that allows you to apply a <em>total</em> timeout even if you need to execute the coroutines <strong>sequentially</strong></li> </ul> <pre><code> async with async_timeout.timeout(5.0): await f() await g() </code></pre> <ul> <li><code>[asyncio.as_completed()](https://ift.tt/3cIn7IY> takes an iterable of awaitables and returns an iterator that yields <code>[asyncio.Future](https://ift.tt/3gTD5TT in the order the awaitables are done</li> </ul> <pre><code> for fut in asyncio.as_completed([task_f, task_g], timeout=5.0): try: await fut print("one task down!") except Exception: print("ouch") </code></pre> <ul> <li>Michael’s <a href="http://talkpython.fm/async">Async Python course</a>.</li> </ul> <p><strong>Brian #2:</strong> <strong>virtualenv is faster than venv</strong></p> <ul> <li><a href="https://virtualenv.pypa.io/en/latest/">virtualenv docs</a>: “<code>virtualenv</code> is a tool to create isolated Python environments. Since Python <code>3.3</code>, a subset of it has been integrated into the standard library under the <a href="https://docs.python.org/3/library/venv.html">venv module</a>. The <code>venv</code> module does not offer all features of this library, to name just a few more prominent: <ul> <li>is slower (by not having the <code>app-data</code> seed method),</li> <li>is not as extendable,</li> <li>cannot create virtual environments for arbitrarily installed python versions (and automatically discover these),</li> <li>is not upgrade-able via <a href="https://pip.pypa.io/en/stable/installing/">pip</a>,</li> <li>does not have as rich programmatic API (describe virtual environments without creating them).”</li> </ul></li> <li>pro: faster: under 0.5 seconds vs about 2.5 seconds</li> <li>con: the <code>--prompt</code> is weird. I like the parens and the space, and 3.9’s magic “.” option for prompt to name it after the current directory.</li> <li>pro: the pip you get in your env is already updated</li> <li>conclusion: <ul> <li>I’m on the fence for my own use. Probably leaning more toward keeping built in. But not having to update pip is nice.</li> <li>For teaching, I’ll stick with the built in <code>venv</code>.</li> <li>The “extendable” and “has an API” parts really don’t matter much to me. </li> </ul></li> </ul> <pre><code> $ time python3.9 -m venv venv --prompt . real 0m2.698s user 0m2.055s sys 0m0.606s $ source venv/bin/activate (try) $ deactivate $ rm -fr venv $ time python3.9 -m virtualenv venv --prompt "(try) " ... real 0m0.384s user 0m0.202s sys 0m0.255s $ source venv/bin/activate (try) $ </code></pre> <p><strong>Michael #3:</strong> <a href="https://nullprogram.com/blog/2020/05/24/">Latency in Asynchronous Python</a></p> <ul> <li>Article by Chris Wellons</li> <li>Was debugging a misbehaving Python program that makes significant use of <a href="https://docs.python.org/3/library/asyncio.html">Python’s asyncio</a>.</li> <li>The program would eventually take very long periods of time to respond to network requests.</li> <li>The program’s author had made a couple of fundamental mistakes using asyncio.</li> <li>Scenario: <ul> <li>Have a “heartbeat” async method that beats once every ms: <ul> <li>heartbeat delay = 0.001s</li> <li>heartbeat delay = 0.001s</li> <li>…</li> </ul></li> <li>Have a computational amount of work that takes 10ms</li> <li>Need to run a bunch of these computational things (say 200).</li> <li>But starting the heartbeat blocks the asyncio event loop</li> <li>See my example at https://ift.tt/2Ue1XM2> </ul></li> <li><a href="https://github.com/alex-sherman/unsync">Unsync</a> fixes this and improves the code! Here’s my example: https://ift.tt/374gdfB> <li>Need to limit the number of “active” tasks at a time.</li> <li><strong>Solving it with a job queue:</strong> Here’s what does work: a <a href="https://docs.python.org/3/library/asyncio-queue.html">job queue</a>. Create a queue to be populated with coroutines (not tasks), and have a small number of tasks run jobs from the queue.</li> </ul> <p><strong>Brian #4:</strong> <a href="https://www.dampfkraft.com/code/how-to-deprecate-a-pypi-package.html"><strong>How to Deprecate a PyPI Package</strong></a></p> <ul> <li>Paul McCann, <a href="https://twitter.com/polm23">@polm23</a></li> <li>A collection of options of how to get people to stop using your package on PyPI. Also includes code samples ore example packages that use some of these methods.</li> <li>Options: <ul> <li><strong>Add deprecation warnings:</strong> Useful for parts of your package you want people to stop using, like some of the API, etc.</li> <li><strong>Delete it:</strong> Deleting a package or version ok for quick oops mistakes, but allows someone else to grab the name, which is bad. Probably don’t do this.</li> <li><strong>Redirect shim:</strong> Add a setup.py shim that just installs a different package. Cool idea, but a bit creepy. </li> <li><strong>Fail during install:</strong> Intentionally failing during install and redirecting people to use a different package or just explain why this one is dead. I think I like this the best.</li> </ul></li> </ul> <p><strong>Michael #5:</strong> <a href="https://pypi.org/project/enlighten/"><strong>Another progress bar library: Enlighten</strong></a></p> <ul> <li>by Avram Lubkin</li> <li>A few unique features:</li> <li><strong>Multicolored progress bars - It's like many progress bars in one!</strong> You could use this in testing, where red is failure, green is success, and yellow is an error. Or maybe when loading something in stages such as loaded, started, connected, and the percentage of the bar for each color changes as the services start up. Has 24-bit color support.</li> <li><strong>Writing to stdout and stderr just works!</strong> There are a lot of progress bars. Most of them just print garbage if you write to the terminal when they are running.</li> <li><strong>Automatically handles resizing! (except on Windows)</strong></li> <li>See the animation on the home page.</li> </ul> <p><strong>Brian #6:</strong> <a href="https://codeocean.com/"><strong>Code Ocean</strong></a></p> <ul> <li>Contributed by Daniel Mulkey</li> <li>From Daniel “a peer-reviewed journal I read (SPIE's Optical Engineering) has a recommended platform for associating code with your article. It looks like it's focused on reproducibility in science. “</li> <li>Code Ocean is a research collaboration platform that supports researchers from the beginning of a project through publication.</li> <li>This is a paid service, but has a free tier.</li> <li>Supports: <ul> <li>C/C++</li> <li>Fortran</li> <li>Java </li> <li>Julia</li> <li>Lua</li> <li>MATLAB</li> <li>Python (including jupyter) (why is this listed so low? should be at the top!) </li> <li>R</li> <li>Stata</li> </ul></li> <li>From the “About Us” page: <ul> <li>“We built a platform that can help give researchers back 20% of the time they spend troubleshooting technology in order to run and reproduce past work before completing new experiments.”</li> <li>“Code Ocean is an open access platform for code and data where users can develop, share, publish, and download code through a web browser, eliminating the need to install software on personal computers. Our mission is to make computational research easier, more collaborative, and durable.”</li> </ul></li> </ul> <p>Extras:</p> <p>Brian:</p> <ul> <li><a href="https://pythoninsider.blogspot.com/2020/05/python-390b1-is-now-available-for.html">Python 3.9.0b1 is available for testing</a></li> </ul> <p>Michael:</p> <ul> <li>SpaceX launch, lots of Python in action.</li> </ul> <p>Joke:</p> <ul> <li>Sent over by <a href="https://twitter.com/StevenCHowell">Steven Howell</a></li> <li><p><a href="https://twitter.com/tecmint/status/1260251832905019392">https://twitter.com/tecmint/status/1260251832905019392</a></p></li> <li><p>From Bert, <a href="https://twitter.com/schilduil/status/1264869362688765952">https://twitter.com/schilduil/status/1264869362688765952</a></p></li> <li>But modified for my own experience:</li> <li>“What does pyjokes have in common with Java? It gets updated all the time, but never gets any better.”</li> </ul>

from Planet Python
via read more