Python Software Foundation: Lightning Talks Part 1 - Python Language Summit 2020

Sumana Harihareswara

What do you need from pip, PyPI, and packaging?

Python packaging has seen relatively quick development in recent years as a result of increased funding; most famously the new PyPI.org website was launched in 2018. The current work in progress includes malware detection and signed packages on PyPI, a new dependency resolver for pip, and a revamp of virtualenv. Much of this work is funded by grants from companies. (Details on the Working Group page.) Sumana Harihareswara from the Packaging Working Group is prolific grant proposal writer; she presented ideas for further development.

Read more 2020 Python Language Summit coverage.

---

Python packaging ideas for the future include:

• Better specifications, toolchain, and services for building distributions
  
• Robust interoperability testing
  
• Revamp PyPI API
  
• Make setuptools the reference implementation of the distutils API
  
• Provide more standardized editable installations
  
• Add support for pyproject.toml as a way to configure setuptools
  
• Create & maintain a generic wheel-builder
  
• PEP 480: secure the developer-to-user supply chain
  
• Implement perennial manylinux (PEP 600)

Harihareswara solicited packaging ideas from the audience to inform the Python Packaging Authority roadmap and the Fundable Packaging Improvements page, asked them to add their complaints to the packaging problems list, and requested help writing grant proposals.

Since a revamp of virtualenv was listed among the Packaging Working Group's current projects, Barry Warsaw wondered what advantages virtualenv has over venv, which is now built in to all supported Python versions. Bernat Gabor, who maintains virtualenv, answered that virtualenv is faster, provides a richer API for tools built on it, and serves as a laboratory for ideas that might be merged into venv.

Ernest Durbin provided a status update on malware checking: the framework is in place but only two checks have been implemented, "mainly for demonstration." He has invited security researchers to implement more checks.

David Mertz asked whether pip's new dependency resolver would be able to resolve dependency conflicts. Paul Moore said he is still researching what users want pip to do in the case of conflicts, and what solutions are provided by resolver algorithms. The new resolver is about to be released, but it is still alpha-level and will be turned off by default.

Eric Snow

A Retrospective on My "Multi-Core Python" Project

Of all the schemes for freeing CPython from the Global Interpreter Lock, the frontrunner is Eric Snow's plan to give each subinterpreter its own lock. He proposed the idea in 2015, began discussing and prototyping the idea intensely, and burned out the next year. "I was trying to do too much on my own," said Snow. In 2017 he resumed development, this time with dozens of collaborators, and wrote PEP 554 to expose subinterpreters to pure Python programs, which will ease testing for the multi-core project. He presented his progress to the Language Summit in 2018 and in a 2019 PyCon talk. His TalkPython interview last year was especially effective at drawing attention to the project.

Snow's immediate blocker is PEP 554's acceptance and implementation, but much work remains after that. He told the 2020 Language Summit, "I've just been chugging along, little by little. Lots of little pieces to get this project done!" Hard problems include passing data safely between subinterpreters, the "grind" of removing all the global variables, and reaching the actual goal: creating a distinct GIL per subinterpreter. Snow predicts the split GIL won't land in Python 3.9, but "3.10 for sure."
Snow thanked a large list of contributors, many of them outside the core developer team.

Kyle Stanley asked whether daemon threads should be still be allowed in subinterpreters or not. (Victor Stinner tried to ban them but had to revert his change.) Snow replied that daemon threads in subinterpreters lead to many finalization problems, and their use should be discouraged, but removing them entirely has proven too disruptive for the core team to accomplish any time soon.

from Planet Python
via read more

Python Software Foundation: The 2020 Python Language Summit

The Python Language Summit is a small gathering of Python language implementers (both the core developers of CPython and alternative Pythons), as well third-party library authors and other Python community members. The summit features short presentations followed by group discussions. In 2020, the Summit was held over two days by videoconference; questions were asked by a combination of voice and chat.

Summaries of all presentations will be posted here as they are completed.

Thanks to MongoDB for sponsoring the Python Language Summit.

---

Day 1

All Strings Become f-strings
Eric V. Smith

Replacing CPython’s Parser with a PEG-based parser
Pablo Galindo, Lysandros Nikolaou, Guido van Rossum

A Formal Specification for the (C)Python Virtual Machine
Mark Shannon

HPy: a Future-Proof Way of Extending Python?
Antonio Cuni

CPython Documentation: The Next 5 Years
Carol Willing, Ned Batchelder

---

Day 2

Lightning talks (pre-selected)

The Path Forward for Typing
Guido van Rossum

Property-Based Testing for Python Builtins and the Standard Library
Zac Hatfield-Dodds

Core Workflow Updates
Mariatta Wijaya

CPython on Mobile Platforms
Russell Keith-Magee

Lighting talks (sign-up during the summit)

---

Image: Natal Rock Python

from Planet Python
via read more

Python Software Foundation: The path forward for typing - Python Language Summit 2020

"There are a lot of PEPs about typing!" said Guido van Rossum at the Language Summit. Since 2014 there have been ten PEPs approved for Python's type-checking features. Two of them have been approved already this year: the relatively "esoteric" PEP 613: Explicit Type Aliases, and another that will have widespread impact, PEP 585: Type Hinting Generics In Standard Collections, written by Łukasz Langa and mainly implemented by Van Rossum. Thanks to this PEP, types which had been defined like List[int] can now be spelled list[int], with a lowercase "L". As Van Rossum told the Python Language Summit, "We want to avoid a world where users have to remember, 'Here I have to use a capital-L List and here I use a lowercase-L list.'"

Read more 2020 Python Language Summit coverage.

---

A "generic" is a type that can be parameterized with other types. Generics are usually container types. Since Python 3.5, the typing module has provided "type aliases" like List, which can be parametrized with the type of values it contains, like List[str] in this type-annotated function definition:

from typing import List
def greet_all(names: List[str]) -> None:
    for name in names:
        print("Hello", name)

Van Rossum showed the Summit the following code, demonstrating that the ordinary built-in list and dict classes can now be used as generics for type annotations:

>>> p = list[int]
>>> p
list[int]
>>> p.__origin__
<class 'list'>
>>> p.__args__
(<class 'int'>,)
>>> p((1, 2, 3))
[1, 2, 3]
>>> from typing import TypeVar; T = TypeVar("T")
>>> dict[str, T][int]
Dict[str, int]

The syntax list[int] is enabled by implementing __class_getitem__ on list. The built-in containers such as tuple, dict, list and set are supported, along with some standard library containers and abstract base classes, including collections.deque, collections.abc.Iterable, queue.Queue, and re.Pattern. The effect for everyday coders is mainly a matter of spelling, yet as Van Rossum said, "It's probably going to affect everyone's code, or everyone will encounter code like this." Fewer users will have to import type aliases such as List from the typing module; it will be required only for advanced annotations. Van Rossum asked the Summit, "How much of this do we want to make built in?"

Python's approach to type-checking is to add type annotations in the source code, but to check types neither during compilation nor at runtime. Instead, programmers use a separate type-checker (such as mypy or PyCharm). The new PEP 585 type annotations are the same: they do no checking at all, so "nonsense" annotations like list[str, str] are permitted. It is the type checker's job to reject them.

Annotations are not completely free at runtime, however: by default an annotation like List[int] is evaluated to create a type object when it is encountered, usually at module-load time. This can noticeably hurt startup times for big type-annotated programs. PEP 563 Postponed Evaluation of Annotations was introduced in Python 3.7 to solve this problem: type annotations are saved as strings, and evaluated only when a type checker such as mypy requests it. This optimization is currently guarded behind from __future__ import annotations. Van Rossum asked whether postponed evaluation should become the default in Python 3.9, which will be released imminently, or 3.10.

Also in Python 3.10 will be PEP 604, which permits the current Union[t1, t2] annotation to be spelled as t1 | t2, using the vertical bar to express a union of types. The PEP's scope might expand to add syntax that even programs without type annotations would enjoy. For example, isinstance(x, (t1, t2)) could be written isinstance(x, t1 | t2), and an exception handler could be written like except t1 | t2.

Yury Selivanov noted that typing.Optional[t] could be replaced with t | None, and asked whether it could be shortened further as t?. "Every year," replied Van Rossum, "there's another feature that people want to use the question mark for." In his opinion, t | None is convenient enough, and another syntax would be redundant. (Although the new PEG parser would make it easy to implement.)

Stéphane Wirtel asked if Python would ever have exception annotations. "Ouch!" said Van Rossum. The consensus is that Java's checked exceptions were a bad idea, and would probably be bad in Python too. "I don't think I have the stomach for that."

The standard library and most PyPI packages have no type annotations. Type-hinted "package stubs" for this code are hosted in the typeshed repository, but storing all those stubs in a monolithic distribution doesn't scale, and the problem will grow worse. In a GitHub issue thread, Jukka Lehtosalo predicted that in two years, stubs for third-party packages will outnumber those for the standard library, and in five years, typeshed will include more than 1000 third-party packages. As Van Rossum told the Language Summit, Lehtosalo's proposal will split typeshed into separate distributions so users can easily download just the stubs they need, consistent with PEP 561.

Brett Cannon asked whether the standard library's annotations should be shipped with Python, either as stub files or in the code itself. Van Rossum said new stdlib code should be written with annotations inline, but old code includes optimizations and strange legacy behaviors that defy static typing. Currently mypy does not analyze standard library code because "it assumes that the standard library is full of untyped shit," it looks in typeshed instead. If indigenous type annotations grew in the standard library, the core team would have to coordinate with type checker authors to manage the change.

Van Rossum offered an update on mypy. He admitted he hadn't been active on mypy recently, and "my former colleagues at Dropbox have not been able to make as much progress as we did in the past." Support for NumPy is stalled. The same goes for decorators, although once PEP 612 is approved it will provide a prerequisite for decorator support in mypy. Raymond Hettinger asked if mypy development needs funding. Michael Sullivan, a mypy contributor from Dropbox, replied that Dropbox considers mypy mostly complete, and has moved on to projects like their Python 3 migration. Van Rossum said funding could help. Personally he has "moved on to retirement." The Python static typing mailing list is quieter than Van Rossum would like, interested people should join.

There's better news about mypyc, an experimental project to translate type-annotated Python into C. The translator's main use for now is converting mypy to C for speed. There is work in progress to allow a mix of Python and Python-translated-to-C in the same program, and to write documentation. The mypyc project expects a Google Summer of Code student this summer.

from Planet Python
via read more

Python Software Foundation: CPython Documentation: The Next 5 Years - Python Language Summit 2020

"Documentation is the way we communicate with each other," said Willing. "Historically, we've done a great job with documentation." But the environment is changing: Python's BDFL has retired, and Python's user base is expanding, becoming more global, and moving away (to some degree) from lower-level programming to higher-level applications. These changes impose new documentation burdens on the small core team. Willing said, "We don't scale well."

Read more 2020 Python Language Summit coverage.

---

Willing and Ned Batchelder proposed a new Python Steering Council workgroup called the "Documentation Editorial Board". Its members would include core developers and community members; they would write style guides, manage translations into non-English languages, and create a landing page that guides different kinds of users to the documentation that suits them. (Daniele Procida had shared earlier a guide to writing docs for a variety of users' needs.) The core team need not write all the docs themselves—they should be owned and written by the community, along with the core team, overseen by the new Editorial Board.

In the Editorial Board's first year it would focus on governance, translations, the new landing page, and tutorials. Willing was inspired by the core team's overhaul of the asyncio docs; they added tutorials and split the high-level information from the low-level. The rest of the standard library would serve users better with more tutorials like asyncio's. Style guides would ensure consistency and best practices. As Ned Batchelder pointed out, Python has two PEPs for code style (one for C and one for Python), but none for documentation.

In its second year, the Board would measure its effectiveness so far, and begin running documentation sprints. Willing recommends the Board begin annual editorial reviews, seeking patterns of user confusion: "When users ask questions on mailing lists and the bug tracker, it means something's not clear to them." Updating the documentation to fix common misunderstandings would save time in the long run for users and the core team.

Batchelder observed that "twenty-five years ago, our main audience seemed to be refugees from C," but most readers of the Python docs today are not career software developers at all; they need different docs.

Raymond Hettinger asked, "Any thoughts on why no one stepped up to write any docs for the walrus operator? I'm not seeing people volunteering for major documentation efforts. Mostly the contributions are minor, micro edits." Willing replied that the walrus operator specifically was a "hot potato" that deterred volunteers. In general, the Python core team doesn't encourage others to lead big documentation projects; community members don't have a sense of ownership over the docs, nor the authority to merge their changes, so skilled writers take their efforts elsewhere. The proposed new Editorial Board would help to change that.

Sumana Harihareswara asked how documentation work would be funded, and whether professional technical writers might be involved. Willing replied that the PSF will fund some work, but she emphasized recruiting volunteers from the community. Several in the audience asked about making a "core documenter" role analogous to "core developer"; Batchelder replied that fine-grained roles and permissions in open source projects are counterproductive. People who write excellent documentation should simply be promoted to core developers.

from Planet Python
via read more

Python Software Foundation: HPy: a future-proof way of extending Python? - Python Language Summit 2020

Antonio Cuni presented HPy (pronounced "aitch pi"), an attempt at a replacement C API that is compatible and performant across several interpreter implementations. The idea was born at EuroPython last year from a discussion among CPython, PyPy, and Cython developers.

Read more 2020 Python Language Summit coverage.

---

CPython's API for C extensions is tightly coupled with the interpreter's internals. Another interpreter such as Jython, if it wants to support the same C extensions, must emulate these internals, pretending to the C extension that the interpreter works the same as CPython. Even CPython suffers: any of its internals that are exposed in the C API can't be improved without breaking compatibility. Python objects in the C API are pointers to CPython's PyObject structs, whose internal layout is partly exposed to extensions. Extensions expect each PyObject pointer to be constant for the object's lifetime, which prevents a memory manager like PyPy's from moving objects during garbage collection.

Most prominently, the C API requires extensions to control objects' lifetimes by incrementing and decrementing their reference counts. Any Python implementation that does not have a reference-counting memory manager, such as PyPy, must emulate refcounts for the sake of the C API. Cuni calls this a "massive amount of precious developer hours wasted," an impediment to performance, and the main obstacle for Larry Hastings's GILectomy.

Victor Stinner has already outlined the design for a better C API that hides some internals, but still depends on reference-counting; Cuni confronts the same questions and gives a more radical answer.

HPy is a new C API that is interpreter-agnostic. Instead of PyObject pointers, HPy presents handles (hence the "H" in its name). Where a C API user would incref a PyObject when copying a reference to it, an HPy user would duplicate a handle. Each handle is distinct and must be closed independently. Cuni showed this code example:

/* C API */
PyObject *a = PyLong_FromLong(42);
PyObject *b = a;
Py_INCREF(b);
Py_DECREF(a);
Py_DECREF(a); // Ok
/* HPy */
HPy a = HPyLong_FromLong(ctx, 42);
HPy b = HPy_Dup(ctx, a);
HPy_Close(a);
HPy_Close(a); // WRONG!

Handles are HPy's basic departure from the C API. The independence of handles' liftimes, said Cuni, decouples HPy from CPython's ref-counting memory manager, and makes HPy a natural, fast C interface for other Pythons. PyPy, for example, will maintain a map of handles to objects; when its garbage collector moves objects in memory, it only needs to update the map. Handles permit precise debugging: if a handle is leaked, HPy prints the line number where it was created. (The HPy context parameter ctx that is passed everywhere allows for subinterpreters, and perhaps other features in the future.)

Brett Cannon asked whether HPy will be a minimalist API for extension development, or if it will include specialized APIs for speed. For example, the current C API has a generic PyObject_GetItem, and a fast PyDict_GetItem specialized for dicts. Cuni said he prefers a smaller API, but benchmarks would guide him.

Cannon asked whether a tool could semi-automatically port C code from the C API to HPy. It could not, according to Cuni, because the problem of closing each handle exactly once must be solved carefully by a human. HPy's debug messages will be a great help. "In theory," Cuni said, "it should be as easy as adding an 'H' to all C API calls, renaming Py_INCREF to HPy_Dup, putting HPy_Close here and there, and then see if the debug mode is happy or complains."

Victor Stinner asked whether his draft proposal to incrementally modify the C API to hide internals would eventually solve PyPy's problems with C extensions. Cuni replied, "It's not enough for PyPy because of reference counting and the fact that PyObject* is not a good representation for objects that can move in memory." But he acknowledged that Stinner's proposal goes in the right direction.

Cuni said the "HPy strategy to conquer the world" is to create a zero-overhead façade that maps HPy to the C API (using compile-time macros), then port third-party C extensions to pure HPy, one function at a time. It must be faster on alternative implementations than their existing C API emulations; early benchmarks show a 3x speedup on PyPy and 2x on GraalPython, a JVM-based Python.

HPy is currently missing type objects, but Cuni said it "basically works." An HPy extension can be compiled to the CPython ABI or to an "HPy universal ABI" that allows the same compiled extension to work with multiple interpreters. In the future, a new Cython backend will target HPy instead of the C API. Cuni and his collaborators have ported ujson to HPy; they plan to port a subset of NumPy next, and eventually to write a PEP and merge HPy into the official CPython distribution, where it will live alongside the existing C API. Cuni hopes the core developers will endorse HPy for third-party C extension development; in a "hypothetical sci-fi future" CPython might port its standard library C modules to HPy.

from Planet Python
via read more

Python Software Foundation: A formal specification for the (C)Python virtual machine - Python Language Summit 2020

Mark Shannon began his presentation saying, "This should actually be titled A Semi-Formal Specification. I don't think we'll ever get to the stage of ML," a language described with mathematical rigor. However, a more formal definition of Python would be useful. Authors of alternative Python implementations would have a standard to go by, besides reading CPython's source and testing for equivalent behavior on a variety of programs. "It seems to work for Java so I don't see why it wouldn't work for Python," said Shannon. It would be particularly helpful for anyone who wants to examine a part of the language. Today, one cannot understand CPython's bytecode interpreter without studying all its quirky interactions with the C API, the code loader, and so on; a Python specification would break down the language into domains that could be grasped separately.

A specification would clean up edge cases. Shannon said there are "too many weird bugs relating to interactions of threads and generators and locals and various other bits and pieces." If the language were defined, developers would at least know how it's supposed to behave.

Read more 2020 Python Language Summit coverage.

---

Shannon proposed to split Python's specifications into major components: code loading (including imports and parsing), execution, and the C API. The components' specs would be allowed to refer to each other, but only at a high level. In his Language Summit presentation, Shannon focused on the execution component.

A Python execution spec would define the virtual machine (VM) as a set of variables, such as the list of threads and their call stacks, and it would define small steps that transition from one VM state to the next. For example, a Python function call involves these steps:

• Create a new stack frame
  
• Move arguments from current frame to the new one
  
• Save the next instruction pointer
  
• Push the new frame onto the call stack
  
• Continue

Every definition ends with "continue", meaning that the interpreter proceeds to the next step.
Writing a specification for Python is a chance to rethink how its parts relate. CPython's generators are implemented in terms of iterators, because iterators were built first. But generators are a more basic and flexible concept; starting from scratch it's natural to define iterators in terms of generators. Shannon presented a sketch of such a definition.

The next steps for Shannon's project are to define the spec's format, list its components, and decide what to do about "awkward" language features such as sys.settrace(). He concluded that a semi-formal spec of Python would help alternative implementations match CPython, would make PEPs less ambiguous, and would clarify whether any existing "odd behavior is a feature or a bug." It would be possible to reason about the correctness of optimizations. However, writing the spec is work, and it could deter good PEPs in the future if authors are daunted by writing their proposals in terms of the spec.

The audience's questions revealed their confusion about Shannon's idea. Larry Hastings admitted he had "lots of trouble following this." Was Shannon proposing a new language implementation based on the spec? Shannon said no, the spec's purpose was to describe and reason about CPython and other implementations. A. Jesse Jiryu Davis wondered if the spec would be written in a formal modeling language like TLA+ or Alloy, but Shannon felt that would discourage contributors. English would be the language of the spec; the JVM spec demonstrates this approach.

Brett Cannon asked if PEPs for new language features would require spec patches. Shannon replied that PEPs for deep changes, similar to the introduction of yield from in Python 3.3, would benefit if they were described in terms of the spec.

The presentation ended with the attendees saying a Python specification might be a good idea, but struggling to envision it and how it would be used in practice.

from Planet Python
via read more

Python Software Foundation: Replacing CPython’s parser - Python Language Summit 2020

Since its start, Python’s grammar has been LL(1): it needs only a left-to-right parser that looks one token ahead to resolve ambiguities. The standard CPython parser is produced by a simple custom parser generator. There are some costs to this simplicity, however. First, the official Python grammar file does not capture the language exactly. There are invalid constructs allowed by the grammar, for example, this assignment expression (using the new walrus operator):

[x for x in y] := [1, 2, 3]

This expression is illegal, because the left side of the walrus operator must be a name, not an arbitrary sub-expression like [x for x in y]. Python’s LL(1) parser is not powerful enough to enforce this rule, though, so it must be enforced after parsing by special-case logic that runs while transforming the parse tree into the abstract syntax tree (AST).

Worse, there is Python code that we would like to write but cannot, because it can’t be parsed. Parenthesized with-statements look reasonable but they’re currently prohibited:

with (
    open("a_really_long_foo") as foo,
    open("a_really_long_baz") as baz,
    open("a_really_long_bar") as bar
):
    ...

Read more 2020 Python Language Summit coverage.

---

Guido van Rossum, Pablo Galindo, and Lysandros Nikolaou wrote a new Python parser to excise these warts, and proposed PEP 617 to adopt it in CPython. The new parser is written in a more powerful style called a parsing expression grammar (PEG), so the project is named “PEGEN”.

When a PEG parser reads the beginning of a token sequence that could match several grammar rules, the parser tries each of those rules, from left to right, until one succeeds. For example, given a set of rules:

rule: A | B | C

The parser first tries to apply rule A to its input sequence. If A succeeds, the parser ignores B and C. Otherwise, the parser moves on and tries to apply rule B, and so on. Unlike an LL(1) parser, a PEG parser can look ahead as far as necessary to disambiguate a sequence. The grammar is deterministic: in cases where multiple rules match, the leftmost wins. Some sequences would require exponential time to find the matching rule; to prevent this, most PEG parsers (including the new Python parser) implement the “packrat” method to cache intermediate results. A packrat parser can process any input in linear time, but spends some memory on its cache.

PEGEN’s grammar is far more legible than the old one’s, and the grammar exactly matches all legal Python programs.

CPython’s old parser progresses in stages: the tokenizer feeds the LL(1) parser, which produces a concrete syntax tree (CST), which is transformed into the AST. The CST stage is necessary because the parser does not support left recursion. Consider this expression:

a + b + c

The old parser’s CST is too flat, as the presenters showed in this slide:

In real life, the expression will be evaluated at runtime by first adding a to b, then adding c. Unfortunately the CST does not encode this nested evaluation, so the LL(1) parser must transform the program a second time to turn the CST into an AST in the properly nested form, which can then generate bytecode. PEGEN, on the other hand, directly generates the AST. Depending on the program, PEGEN may use more or less memory: what it spends on the packrat method, it may save by skipping the CST.

Emily Morehouse tested PEGEN against the old parser by verifying they produce the same AST for every module in the standard library and the top 3800 PyPI packages. (PEGEN parses the standard library slightly faster than the old parser, but uses 10% more memory.)

Once Python has a non-LL(1) parser, its syntax may grow more grammatically complex; Victor Stinner asked if third-party Python parsers (such as linters and IDEs) might have difficulty. Van Rossum felt certain they could adopt more powerful parsers themselves, if necessary.

The authors plan to switch CPython to the new parser cautiously. In Python 3.9 alpha 6 the new parser will be opt-in; it will become the default in beta 1 and in the official 3.9 release, with the old parser still available by a command line switch. As soon as the new parser is enabled, parenthesized with-statements will be allowed! In Python 3.10 the old parser will be deleted. PEP 617 must still be approved, however, and the new code needs a final review.

The proposal met no opposition; in fact, several in the audience asked whether it could become the default sooner.

from Planet Python
via read more

Python Software Foundation: All Strings Become f-strings - Python Language Summit 2020

The first language change proposed this year was the most radical: to make f-strings the default. Eric V. Smith, who wrote the PEP for f-strings in 2015, said they are the killer feature of Python 3.6, and they are the motivation for many of his clients to move to Python 3. However, they are error-prone. It's common to forget the "f" prefix:

x = 1
# Forgot the f prefix.
print("The value is {x}")

Smith has observed programmers f-prefixing all strings, whether they include substitutions or not, just to avoid this mistake.

Read more 2020 Python Language Summit coverage.

---

When f-strings were added in 3.6, it was suggested to make them the default, but this was too big a breaking change. Besides, replacing all literal brace characters with double braces would be ugly:

"A single pair of braces: "

In this year's Summit, Smith proposed again to make f-strings the default. The following kinds of strings would become f-strings:

• "string" — an f-string
  
• f"string" — still an f-string
  
• r"string" — a raw f-string
  

Binary literals like b"string" would not become f-strings. Smith would add a new "p" string prefix for plain strings, which would behave like ordinary strings today.

• p"string" — a plain string
  

Performance would not be affected: the runtime behavior of a string without any substitutions would be the same as today. Plain strings would still have some uses; for example, regular expressions that include braces, or as the input to str.format. In Smith's opinion, f-strings have superseded str.format, but several in the audience objected that str.format with a plain string allows for late binding, and f-strings don't obviate str.format_map.

Smith acknowledged some problems with his idea. It would introduce yet another string prefix. Flipping the master switch to enable f-mode would break some code, so there must be a way to gradually enable the change module by module, perhaps like:

from __future__ import all_fstrings

He was concerned the change was so drastic that the Python core developers would never have the nerve to enable it without requiring a future import. If so, the idea should be abandoned right away.

Yarko Tymciurak asked via chat: "How do you describe to beginners what p'why is this needed' is?" Smith conceded that p-strings make the language more complicated, but, he said, "There's going to be very few p's in the wild, and I think their explanation will be fairly obvious."

Several attendees were enthusiastic to make the change. Brett Cannon said that removing the need for f-prefix would make the language easier for beginners.

Larry Hastings pointed out that PHP strings are format strings by default and "the script kids love it." However, he wrote, "It seems to me this is solving the problem of 'oh I forgot to put an f in front of my string', and not noticing until it's too late. Is that problem bad enough that we have to change the language?
" Many agreed that f-strings by default would have been a good idea if Python were beginning from scratch; however, Paul Moore, Guido van Rossum, and others feared the disruption would outweigh the benefits. The group concluded that Smith should send his PEP to the mailing list for further debate.

from Planet Python
via read more

Lightning Talks Part 1 - Python Language Summit 2020

Sumana Harihareswara

What do you need from pip, PyPI, and packaging?

Python packaging has seen relatively quick development in recent years as a result of increased funding; most famously the new PyPI.org website was launched in 2018. The current work in progress includes malware detection and signed packages on PyPI, a new dependency resolver for pip, and a revamp of virtualenv. Much of this work is funded by grants from companies. (Details on the Working Group page.) Sumana Harihareswara from the Packaging Working Group is prolific grant proposal writer; she presented ideas for further development.

Read more 2020 Python Language Summit coverage.

---

Python packaging ideas for the future include:

• Better specifications, toolchain, and services for building distributions
  
• Robust interoperability testing
  
• Revamp PyPI API
  
• Make setuptools the reference implementation of the distutils API
  
• Provide more standardized editable installations
  
• Add support for pyproject.toml as a way to configure setuptools
  
• Create & maintain a generic wheel-builder
  
• PEP 480: secure the developer-to-user supply chain
  
• Implement perennial manylinux (PEP 600)

Harihareswara solicited packaging ideas from the audience to inform the Python Packaging Authority roadmap and the Fundable Packaging Improvements page, asked them to add their complaints to the packaging problems list, and requested help writing grant proposals.

Since a revamp of virtualenv was listed among the Packaging Working Group's current projects, Barry Warsaw wondered what advantages virtualenv has over venv, which is now built in to all supported Python versions. Bernat Gabor, who maintains virtualenv, answered that virtualenv is faster, provides a richer API for tools built on it, and serves as a laboratory for ideas that might be merged into venv.

Ernest Durbin provided a status update on malware checking: the framework is in place but only two checks have been implemented, "mainly for demonstration." He has invited security researchers to implement more checks.

David Mertz asked whether pip's new dependency resolver would be able to resolve dependency conflicts. Paul Moore said he is still researching what users want pip to do in the case of conflicts, and what solutions are provided by resolver algorithms. The new resolver is about to be released, but it is still alpha-level and will be turned off by default.

Eric Snow

A Retrospective on My "Multi-Core Python" Project

Of all the schemes for freeing CPython from the Global Interpreter Lock, the frontrunner is Eric Snow's plan to give each subinterpreter its own lock. He proposed the idea in 2015, began discussing and prototyping the idea intensely, and burned out the next year. "I was trying to do too much on my own," said Snow. In 2017 he resumed development, this time with dozens of collaborators, and wrote PEP 554 to expose subinterpreters to pure Python programs, which will ease testing for the multi-core project. He presented his progress to the Language Summit in 2018 and in a 2019 PyCon talk. His TalkPython interview last year was especially effective at drawing attention to the project.

Snow's immediate blocker is PEP 554's acceptance and implementation, but much work remains after that. He told the 2020 Language Summit, "I've just been chugging along, little by little. Lots of little pieces to get this project done!" Hard problems include passing data safely between subinterpreters, the "grind" of removing all the global variables, and reaching the actual goal: creating a distinct GIL per subinterpreter. Snow predicts the split GIL won't land in Python 3.9, but "3.10 for sure."
Snow thanked a large list of contributors, many of them outside the core developer team.

Kyle Stanley asked whether daemon threads should be still be allowed in subinterpreters or not. (Victor Stinner tried to ban them but had to revert his change.) Snow replied that daemon threads in subinterpreters lead to many finalization problems, and their use should be discouraged, but removing them entirely has proven too disruptive for the core team to accomplish any time soon.

from Python Software Foundation News
via read more