If you have ever looked at a project that has a lot of wheels (like numpy), you may have wondered what the part that comes after the project name and version mean. Well, they are known as platform compatibility tags and they are primarily defined in PEP 425. For someone like me whose personal projects are all written in pure Python, I never really paid much thought to what those tags meant since the wheel tags for my projects are all py3-none-any (you will find out what that means later in this post). So what led me from not caring to learning as much as I could about wheel tags and what did I learn along the way?
I get sucked into packaging projects too easily
For some reason I want to help Python end up with a good packaging story. Now I'm not deep into packaging because my day job doesn't require it and my work on Python itself is enough to keep me occupied with my open source time. But for some reason I still try to keep up with what is happening with the PyPA and that inevitably leads me into getting sucked into some new project on a semi-regular basis. A perfect example of this is PEP 518 and pyproject.toml: this didn't actually solve a problem I was having in general, I just knew it would be a good thing to have for packaging in the Python community and would help solve a problem that had existed for well over a decade.
One day I got it into my head that one should be able to script the installation of something from PyPI entirely via open source packages and their programmatic APIs which implemented some specification (in other words running pip through subprocess didn't count, nor did something that followed some old practice that wasn't covered by a PEP). And so I outlined the steps to go from package and version name to installed package and figured out where the gaps were. One of the gaps that I found in the outline was there was no way to easily tell what wheel to download from PyPI for a project as there wasn't a package to help with wheel tags. And so I dove in.
What was the state of things when I started?
Digging into this and talking with PyPA folks I learned that there was a module named pep425tags which various projects used:
Unfortunately every one of those projects had a vendored copy that had deviated from the original code (which I think came from zc.buildout). When looking at the APIs of all four versions it became apparent rather quickly that the module's feature API was rather small and most of the exposed API was actually for the module's own benefit. So that at least meant it wouldn't be difficult to come up with an API which should work for all projects.
As for semantics, it was recommended to me to look at pip's code since it was recently updated to deal with some bugs. And so I used pip's output from its copy of pep425tags as my starting point. But before I dive into what I found out I should probably explain how wheel compatibility tags work.
Compatibility tags
What a wheel filename is trying to communicate to you is what platform(s) it is compatible with without requiring that you open the file in any way. This is because reading a filename is much easier, faster, and something a human being can easily do from their OS compared to having to pull out some metadata contained within a wheel. It also prevents filename collisions in cases where a project has 22 wheel files like numpy.
As a consumer of wheels, what you're after is the most specific wheel which will work for your Python installation. You want that because the assumption is if a wheel is more specific it will perform better, provide more options, etc. than the most general wheel that your platform supports. It should also be noted that wheels represent what Python installation the wheel should be compatible with. There's always a possibility that a wheel won't work on a platform even if the tags match simply because there's a limit in how much information can be provided in a filename (and compatibility with compiled C code is surprisingly hard).
And to communicate what wheels are compatible with what Python installation setups, a tag triple is used.
Interpreter
The first tag is for the interpreter. This represents the Python interpreter version the wheel is compatible with. Well-known interpreters like CPython or PyPy have predefined abbreviations like cp and pp, respectively. That means that if you see a tag for cp37 then it's considered compatible with CPython 3.7. For pp360 it means it's compatible with PyPy3 6.0. The version number is meant to represent the major and minor version of the interpreter version, hence why PyPy3 has a version number that doesn't necessarily match the version of Python it implements.
Now obviously it's possible to write Python code that is interpreter-agnostic and these example tags don't reflect that. This is why there's a py abbreviation to represent the Python language; you can think of it like an abstract interpreter for a version of the language. So if a wheel has a tag of py37 then it's compatible with any interpreter that supports Python 3.7.
The py "interpreter" is also often used to represent just the major version of Python, like py3 for wheels that are not strictly tied to a major/minor version of Python. Remember earlier when I said that wheels represent what should work for your Python installation? The py3 tag is a good example of that: it should work on Python 3, but if you're running a really old version like Python 3.0 the wheel is not guaranteed to work for you.
ABI
The next tag is the ABI required of the Python interpreter. This is traditionally a CPython-specific thing, but there's technically nothing stopping PyPy3 from declaring it supports CPython's API or even defining it's own ABI.
Specifically for CPython 3.7, you may see the following ABI tags:
cp37*cp37mcp37dmcp37dcp37
abi3none
The cp37* tags are the most common one for CPython-specific wheels, and specifically cp37m is what nearly all wheels on PyPI will be for CPython. The cp37 bit is just like in the interpreter tag: it represents the interpreter version that the wheel expects. The various letter suffixes mean the following:
m: compiled with pymallocd: compiled for a debug build of CPython (we're currently discussing on python-dev how to make this less necessary, if at all)
There's also a u letter for Python 2.7, but since that won't be much an issue in less than 7 months as I write this, I won't bother going into it. Suffice it to say that Python 3 is better for a reason. 😊
The abi3 tag is for the stable ABI. Unfortunately that ABI hasn't been useful due to its design. We unfortunately made it an opt-out action to not have new APIs go into the stable ABI versus opt-in like it should have been, and so we have accidentally expanded it on numerous occasions. Plus I think enough people want C extension modules for performance that they don't want any restrictions in what part of the CPython API they use, and so people just build a bunch of cp wheels and don't worry about minimizing the number of wheels they provide that way.
Finally, there's the none ABI which represent the case of not caring. This is typically seen with py interpreter tags since you shouldn't care about what ABI an interpreter supports if you're targetting just the Python language and not a specific interpreter.
Platform
The platform tag represents what operating system a wheel supports. Now OSs being OSs, what labels exist and why for Windows, macOS, and Linux vary.
Windows
For Windows, there's just two:
win_amd64win32
In other words the backwards-compatibility obsession that Windows has means you only have to care about whether you're running a 32- or 64-bit CPU. Simple.
macOS
For macOS, it's rather extensive due to the history of the OS. There's x86 or PowerPC CPU architectures. There's 32- versus 64-bit. There's whether the wheel contains a fat binary for 32-bit, 64-bit, Intel, or either CPU architectures.
And then there's the OS version. Certain versions support only certain bitness for CPUs. Same goes for architecture. This means that what version of macOS that a Python installation might support can be rather long. And this doesn't talk about shifts in macOS details which can shift what a wheel supports.
Luckily it isn't a big deal to support multiple macOS versions with a single wheel thanks to compressed tag sets (which will be discussed later in this post).
Linux
The hardest OS to support is definitely Linux. Unlike Windows which has a hardcore backwards-compatibility guarantee or macOS which updates as a single unit, Linux distributions all do their own thing and each are rather unique. This varies from updating packages aggressively and independently of other packages, as well as simply supporting different packages period. There's also the simple fact that Linux as an operating system is more of a concept than concrete thing that one can easily specify as a baseline build target for a wheel.
And so some very smart people came up with the manylinux concept. What this defines is a baseline Linux OS built on top of the oldest supported CentOS version available along with what libraries every Linux distribution would be reasonably expected to have. The idea is that if you can build with whatever old glibc version that the oldest supported CentOS has then any newer glibc will also work and thus work for your Linux distribution because there are very few Linux distributions with support lifetimes as long as CentOS. (The spec supports specifying 32-bit or 64-bit x86 support.)
Currently there's two versions of manylinux. The manylinux1 spec is in PEP 513 and the manylinux2010 spec is in PEP 571. Each target a different version of CentOS. There is an active discussion going on about how to handle the future of the manylinux spec in terms of whether there should be a perennial manylinux definition or if tagging by year should continue (this stems from the fact it took literally years to get the manylinux2010 spec finished and supported by the appropriate tools). And if the year-style tagging stays there's also a discussion of how to make it more useful to more people as some wheels on PyPI actually don't follow the spec appropriately on purpose due to various issues their code has with the spec. In other words this is all being actively discussed in order to make it work better for everyone involved going forward.
Compressed tag sets
As I mentioned when discussing platform tags for macOS, there's a way to specify support for multiple tags in a single wheel file. Probably one of the most common compressed tag sets you will have seen is the py2.py3 interpreter tag. That dot signifies that the tag has multiple values that are all accurate. You will also see this tag used a lot for macOS as it's not hard to support multiple versions at once, e.g. macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64. The key point is that you can have a single wheel represent compatibility with multiple tags without issue.
Tag triple priority quirks
With all of this knowledge, hopefully you can roughly follow what the following tag triples represent (this is an abbreviated list of what pip gave me for release build of CPython 3.7 on macOS 10.13; I've left out all the other versions of macOS that are also supported as that makes this list 515 entries long):
('cp37', 'cp37m', 'macosx_10_13_x86_64')('cp37', 'abi3', 'macosx_10_13_x86_64')('cp37', 'none', 'macosx_10_13_x86_64')('cp36', 'abi3', 'macosx_10_13_x86_64')('cp35', 'abi3', 'macosx_10_13_x86_64')('cp34', 'abi3', 'macosx_10_13_x86_64')('cp33', 'abi3', 'macosx_10_13_x86_64')('cp32', 'abi3', 'macosx_10_13_x86_64')('py37', 'none', 'macosx_10_13_x86_64')('py3', 'none', 'macosx_10_13_x86_64')('py36', 'none', 'macosx_10_13_x86_64')('py35', 'none', 'macosx_10_13_x86_64')('py34', 'none', 'macosx_10_13_x86_64')('py33', 'none', 'macosx_10_13_x86_64')('py32', 'none', 'macosx_10_13_x86_64')('py31', 'none', 'macosx_10_13_x86_64')('py30', 'none', 'macosx_10_13_x86_64')('cp37', 'none', 'any')('cp3', 'none', 'any')('py37', 'none', 'any')('py3', 'none', 'any')('py36', 'none', 'any')('py35', 'none', 'any')('py34', 'none', 'any')('py33', 'none', 'any')('py32', 'none', 'any')('py31', 'none', 'any')('py30', 'none', 'any')
If you look at this list closely you will notice some interesting quirks. First is that the ABI varies first, but not in a consistent way. You will notice that the ABI varies while the cp37 interpreter tag stays consistent. But then after that, the interpreter tag goes from cp36 to cp30 while keeping the abi3 tag. And then after that it shifts to the py interpreter tag with the none ABI tag.
The next quirk is that goes from py37 to py3 to py36 and on down to py30. I believe the expectation here is that a wheel which claims py3 compatibility will be more compatible then e.g. py30 as chances are a py3 wheel has been developed to be compatible with newer versions of Python than py30.
The last quirk is ('cp3', 'none', 'any'). That honestly makes no real sense since writing C code that somehow works with any version of CPython and has no declared ABI wouldn't ever make sense as you would have done it with abi3 instead and still declared a CPython that you are minimally supporting or with compressed tag sets.
The end result (so far)
After diving into all of this I decided to try and get support for compatibility tags into the packaging package so that there was a central place to support these tags to minimize duplication of work and make it easier to spread proper tag support everywhere. It took a while, but at PyCon US 2019 my PR got merged. The reason I'm excited about this work is I think it will help with compatibility for everyone. I tweeted with excitement as I was working on this and seeing how many gaps I could fill in the tag triples:
Got my first complete result on my PEP 425 library! CPython 3.7 on macOS 10.13 leads to 860 tags (up from the 515 tags pip 18 generates, mostly from adding *-none-* tag combinations)
— Brett Cannon (@brettsky) October 21, 2018
Just added pypy3 support to my PEP 425 library: on macOS 10.13 the tag count goes from 172 under pip to 458 (this is mostly by filling in py3*-none-* tags)
— Brett Cannon (@brettsky) October 27, 2018
That's a 66% increase for CPython 3.7 on my Mac and a 166% increase for PyPy3 on the same computer. I don't know when packaging will do a new release with the new packaging.tags submodule being included and when pip will pick up on it, but hopefully when that all happens users will be able to use wheels that they originally didn't know they could.
from Planet Python
via read more
No comments:
Post a Comment