Sumana Harihareswara

What do you need from pip, PyPI, and packaging?

Python packaging has seen relatively quick development in recent years as a result of increased funding; most famously the new PyPI.org website was launched in 2018. The current work in progress includes malware detection and signed packages on PyPI, a new dependency resolver for pip, and a revamp of virtualenv. Much of this work is funded by grants from companies. (Details on the Working Group page.) Sumana Harihareswara from the Packaging Working Group is prolific grant proposal writer; she presented ideas for further development.

Read more 2020 Python Language Summit coverage.

Python packaging ideas for the future include:

Better specifications, toolchain, and services for building distributions
Robust interoperability testing
Revamp PyPI API
Make setuptools the reference implementation of the distutils API
Provide more standardized editable installations
Add support for pyproject.toml as a way to configure setuptools
Create & maintain a generic wheel-builder
PEP 480: secure the developer-to-user supply chain
Implement perennial manylinux (PEP 600)

Harihareswara solicited packaging ideas from the audience to inform the Python Packaging Authority roadmap and the Fundable Packaging Improvements page, asked them to add their complaints to the packaging problems list, and requested help writing grant proposals.

Since a revamp of virtualenv was listed among the Packaging Working Group's current projects, Barry Warsaw wondered what advantages virtualenv has over venv, which is now built in to all supported Python versions. Bernat Gabor, who maintains virtualenv, answered that virtualenv is faster, provides a richer API for tools built on it, and serves as a laboratory for ideas that might be merged into venv.

Ernest Durbin provided a status update on malware checking: the framework is in place but only two checks have been implemented, "mainly for demonstration." He has invited security researchers to implement more checks.

David Mertz asked whether pip's new dependency resolver would be able to resolve dependency conflicts. Paul Moore said he is still researching what users want pip to do in the case of conflicts, and what solutions are provided by resolver algorithms. The new resolver is about to be released, but it is still alpha-level and will be turned off by default.

Eric Snow

A Retrospective on My "Multi-Core Python" Project

Of all the schemes for freeing CPython from the Global Interpreter Lock, the frontrunner is Eric Snow's plan to give each subinterpreter its own lock. He proposed the idea in 2015, began discussing and prototyping the idea intensely, and burned out the next year. "I was trying to do too much on my own," said Snow. In 2017 he resumed development, this time with dozens of collaborators, and wrote PEP 554 to expose subinterpreters to pure Python programs, which will ease testing for the multi-core project. He presented his progress to the Language Summit in 2018 and in a 2019 PyCon talk. His TalkPython interview last year was especially effective at drawing attention to the project.

Snow's immediate blocker is PEP 554's acceptance and implementation, but much work remains after that. He told the 2020 Language Summit, "I've just been chugging along, little by little. Lots of little pieces to get this project done!" Hard problems include passing data safely between subinterpreters, the "grind" of removing all the global variables, and reaching the actual goal: creating a distinct GIL per subinterpreter. Snow predicts the split GIL won't land in Python 3.9, but "3.10 for sure."
Snow thanked a large list of contributors, many of them outside the core developer team.

Kyle Stanley asked whether daemon threads should be still be allowed in subinterpreters or not. (Victor Stinner tried to ban them but had to revert his change.) Snow replied that daemon threads in subinterpreters lead to many finalization problems, and their use should be discouraged, but removing them entirely has proven too disruptive for the core team to accomplish any time soon.