Python Bytes: #249 All of Linux as a Python API

<p><strong>Watch the live stream:</strong></p> <a href='https://www.youtube.com/watch?v=djSI88HGPq8' style='font-weight: bold;'>Watch on YouTube</a><br> <br> <p><strong>About the show</strong></p> <p>Sponsored by <strong>us:</strong></p> <ul> <li>Check out the <a href="https://training.talkpython.fm/courses/all"><strong>courses over at Talk Python</strong></a></li> <li>And <a href="https://pythontest.com/pytest-book/"><strong>Brian’s book too</strong></a>!</li> </ul> <p>Special guest: <strong><a href="https://twitter.com/Hellsen83">Erik Christiansen</a></strong></p> <p><strong>Michael #1:</strong> <a href="https://github.com/trailofbits/fickling"><strong>Fickling</strong></a></p> <ul> <li>via Oli</li> <li>A Python pickling decompiler and static analyzer</li> <li>Pickled ML models are becoming the data exchange and workflow of ML</li> <li>Analyses pickle files for security risks - It can also remove or insert [malicious] code into pickle files... </li> <li>Created by a security firm, it can be a useful defensive or offensive tool.</li> <li>Perhaps it is time to screen all pickles?</li> </ul> <pre><code> &gt;&gt;&gt; import ast &gt;&gt;&gt; import pickle &gt;&gt;&gt; from fickling.pickle import Pickled &gt;&gt;&gt; print(ast.dump(Pickled.load(pickle.dumps([1, 2, 3, 4])).ast, indent=4)) Module( body=[ Assign( targets=[ Name(id='result', ctx=Store())], value=List( elts=[ Constant(value=1), Constant(value=2), Constant(value=3), Constant(value=4)], ctx=Load()))]) </code></pre> <ul> <li>You can test for common patterns of malicious pickle files with the <code>--check-safety</code> option</li> <li>You can also safely trace the execution of the Pickle virtual machine without exercising any malicious code with the <code>--trace</code> option.</li> <li>Finally, you can inject arbitrary Python code that will be run on unpickling into an existing pickle file with the <code>--inject</code> option.</li> <li>See <strong><a href="https://risky.biz/RB635/">Risky Biz's episode for more details</a></strong>.</li> </ul> <p><strong>Brian #2:</strong> <a href="https://hynek.me/til/python-project-local-venvs/"><strong>Python Project-Local Virtualenv Management</strong></a></p> <ul> <li><strong>Hynek Schlawack</strong></li> <li>Only works on UNIX-like systems. MacOS, for example.</li> <li>Instructions <ul> <li>Install direnv. (ex: brew install direnv)</li> <li>Put this into a <code>.envrc</code> file in your project root: </li> <li><code>layout python python3.9</code></li> </ul></li> <li>Now <ul> <li>when you <code>cd</code> into that directory or a subdirectory, your virtual environment is loaded.</li> <li>when you cd out of it, the venv is unloaded</li> </ul></li> <li>Notes: <ul> <li>Michael covered direnv on <a href="https://pythonbytes.fm/episodes/show/185/this-code-is-snooping-on-you-a-good-thing">Episode 185</a>. But it wasn’t until Hynek spelled it out for me how to use it with venv that I understood the simplicity and power. </li> <li>Not really faster than creating a venv, but when flipping between several projects, it’s way faster than deactivating/activating.</li> <li>You can also set env variables per directory (kinda the point of direnv)</li> </ul></li> </ul> <p><strong>Erik #3:</strong> <strong><a href="https://github.com/testcontainers/testcontainers-python">Testcontainers</a></strong></p> <p>“Python port for testcontainers-java that allows using docker containers for functional and integration testing. Testcontainers-python provides capabilities to spin up docker containers (such as a database, Selenium web browser, or any other container) for testing. “ (pypi description).</p> <ul> <li>Provides cloud native services, many databases and the like (e.g. Google Cloud Pub/Sub, Kafka..)</li> <li>Originally a java project, still a way to go for us python programmers to implement all services</li> <li>Provides an example for use in CI/CD by leveraging Docker in Docker</li> </ul> <pre><code> import sqlalchemy from testcontainers.mysql import MySqlContainer with MySqlContainer('mysql:5.7.17') as mysql: engine = sqlalchemy.create_engine(mysql.get_connection_url()) version, = engine.execute("select version()").fetchone() print(version) # 5.7.17 </code></pre> <p><strong>Michael #4:</strong> <a href="https://github.com/kellyjonbrazil/jc"><strong>jc</strong></a></p> <ul> <li>via Garett</li> <li>CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.</li> <li>Run it as <code>COMMAND ARGS | jc --C</code>OMMAND</li> <li>Commands include: <code>systemctl</code>, <code>passwd</code>, <code>ls</code>, <code>jobs</code>, <code>hosts</code>, <code>du</code>, and <code>cksum</code>.</li> </ul> <p><strong>Brian #5:</strong> <a href="https://florian-dahlitz.de/articles/what-is-pythons-ellipsis-object"><strong>What is Python's Ellipsis Object?</strong></a></p> <ul> <li>Florian Dahlitz</li> <li><code>Ellipsis</code> or <code>…</code> is a constant <a href="https://docs.python.org/3/library/constants.html#Ellipsis">defined in Python</a> <ul> <li>“Ellipsis: The same as the ellipsis literal “...”. Special value used mostly in conjunction with extended slicing syntax for user-defined container data types.”</li> </ul></li> <li>Can be used in type hinting <ul> <li>Func returns two int tuple</li> </ul></li> </ul> <pre><code> def return_tuple() -&gt; tuple[int, int]: pass </code></pre> <ul> <li>Func returns one or more integer:</li> </ul> <pre><code> def return_tuple() -&gt; tuple[int, ...]: pass </code></pre> <ul> <li>Replacement for <code>pass</code>:</li> </ul> <pre><code> def my_function(): ... </code></pre> <ul> <li>Ellipsis in the wild, “if you want to implement a certain feature where you need a non-used literal, you can use the ellipsis object.” <ul> <li>FastAPI : <a href="https://fastapi.tiangolo.com/tutorial/query-params-str-validations/#make-it-required">Ellipsis used to make parameters required</a></li> <li>Typer: <a href="https://typer.tiangolo.com/tutorial/arguments/optional/">Same</a></li> </ul></li> </ul> <p><strong>Erik #6:</strong> <strong><a href="https://pytorch-forecasting.readthedocs.io/en/latest/">PyTorch Forecasting</a></strong> PyTorch Forecasting aims to ease state-of-the-art timeseries forecasting with neural networks for both real-world cases and research alike. The goal is to provide a high-level API with maximum flexibility for professionals and reasonable defaults for beginners. </p> <ul> <li>basically tries to achieve for time series what fast.ai has achieved for computer vision and natural language processing</li> <li>The package is built on PyTorch Lightning to allow training on CPUs, single and multiple GPUs out-of-the-box.</li> <li><a href="https://arxiv.org/abs/1912.09363">Implements of Temporal Fusion Transformers</a> <ul> <li>interpretable - can calculate feature importance</li> </ul></li> <li>Hyperparameter tuning with <a href="https://optuna.readthedocs.io/">optuna</a></li> </ul> <p><strong>Extras</strong></p> <p>Brian</p> <ul> <li><a href="https://blog.python.org/2021/09/python-3100rc2-is-available.html">Python 3.10rc2</a> available. 3.10 is about a month away</li> </ul> <p>Michael</p> <ul> <li><strong>GoAccess</strong> follow up</li> <li><strong>Caffinate more</strong> - via Nathan Henrie: you mentioned the MacOS /usr/bin/caffeinate tool on "<a href="https://pythonbytes.fm/episodes/show/247/do-you-dare-to-press-.">https://pythonbytes.fm/episodes/show/247/do-you-dare-to-press-.</a>". Follow caffeinate with long-running command to keep awake until done (<code>caffeinate python -c 'import time; time.sleep(10)')</code>, or <code>caffeinate -w "$PID"</code> for an already running task. </li> <li><a href="https://twitter.com/smtibor/status/1433870246276907009"><strong>Python Keyboard</strong></a> (via Sean Tabor)</li> <li><a href="https://www.cnbc.com/2021/09/03/mongodb-tops-30-billion-market-cap-in-banner-week-for-open-source.html"><strong>Open source is booming</strong></a> (via Mark Little)</li> <li><a href="https://ffmpegwasm.netlify.app/"><strong>FFMPEG.WASM ffmpeg.wasm is a pure WebAssembly</strong></a> via Jim Anderson</li> <li>Everything is fine: <a href="https://www.theregister.com/2021/08/02/in_brief_security/"><strong>PyPI packages</strong></a></li> <li><a href="https://twitter.com/pyblogsal/status/1435611382976684039"><strong>Python 3.10 RC 2 is out</strong></a></li> </ul> <p><strong>Joke:</strong> <a href="https://www.reddit.com/r/ProgrammerHumor/comments/jnb9fa/when_you_only_validate_the_http_code_of_the/"><strong>200 == 400</strong></a></p>

from Planet Python
via read more