I have a particular dislike of nonspecific negativity, where nothing can be done to address its source because the reasons underlying it are never explicitly described. In the context of Mitogen, there has been a consistent stream of this sort of negativity originating from a particular camp in public forums, and despite efforts to bring specifics out into the open, still it continues to persist.
For that reason I'd like to try a new strategy: justify the negativity and give it a face by providing all the fuel it needs to burn. Therefore in this post, in the interests of encouraging honesty, I will critique my own work.
Mitogen is threaded
Mitogen is a prehistoric design, with the earliest code dating all the way back to 2007, a time when threading seemed cool and the 'obvious' solution to the problem the library intended to solve. Through mountains of words I have since then justified the use of threading, as if the use of threads were a mandatory feature. In some scenarios this is partially true, but just as often entirely incorrect.
This code does everything your mother told you never to do when using threads, and suffers endless asynchrony issues as a result. Historically it failed to lock mutable data structures, and some remnants of that past remain to the present day.
If you're going to throw tomatoes, threading is a great place to start. If there is any legitimate reason to have a problem with integrating this library, look no further.
The documentation sucks
If this guy gets hit by a bus, what happens to the library? Valid concern! Documentation is currently inconsistent at best, out of date and nonexistent at worst. I care a lot about good documentation, but not quite as much as keeping users happy. Therefore as a result, a description of the internals detailed enough to allow someone to take over the work, or even fix some kinds of bugs, does not as yet exist.
It's not that Mitogen is exactly rocket science to maintain, but through poor and inconcise documentation, coupled with a mysterious name, a simple solution is cloaked in a mysterious aura of inscrutability. That incites fear in many who attempt to grasp it.
Mitogen isn't tested properly
Due to the endless asynchrony issues created by threading and running across multiple processes, Mitogen is a nightmare to test. In the common case where some important test passes, there is little to no guarantee that test will continue to pass, say, if the moon turned blue or a flock of migratory pink flamingos happened to pass overhead.
Addressing testing in Mitogen is more than a full-time job, it's an impossible nightmare literally exceeding the boundaries of computer science. Due to the library's design, and the underlying use of threading, a combinatorial explosion of possible program states are produced for which no good testing strategy can or ever will exist.
Many of these aspects can be tested only partially by throwing the library at the wall in a loop and ensuring it sticks, and due to the problem it tries to solve, this will likely remain true in perpetuity. The sound byte you're looking for to justify your loathing is "Mitogen is spaghetti code that's impossible to test"
Mitogen does things it shouldn't do.
If you look closely enough you'll find some modules importing ctypes and doing all kinds of crazy things to work around deficiencies in Python's interfaces. Who the hell writes cowboy code like this? What happens if I run this on some ancient 286 running Xenix? While such atrocities are carefully sandboxed, the reality is that testing in these scenarios has never been done. The computer might catch fire, or the moon might coming crashing into the earth.
Mitogen is riddled with internationalization issues
As a side effect of being written by a monolingual developer, Mitogen sometimes doesn't have a clue about sudo or su when it says things like المصادقة غير صحيحة, therefore you can never really truly 100% rely on it to function when charging handsomely to peddle someone else's work at customer sites, when some of those might be large international concerns, say, in the oil or finance sector.
Because of the hacky methods used to integrate external tools, there may always be some operating system, tool and language combination for which the library will fail to notice something is wrong. In the worst case it might hang, or confusingly claim a connection timeout occurred when really it just didn't understand the tool.
Mitogen uses pickle on the network
The crown jewel: it is absolutely true that Mitogen will pass untrusted pickles through an unpickler, with mitigations in place to prevent arbitrary code execution. But is it enough? Nobody knows for sure! It's impossible to combat this variety of FUD, but just in case a replacement serialization has existed in a local branch for over 6 months, waiting for just the day some material vulnerability is finally revealed.
Be certain to mention pickle insecurity at every opportunity when generating negativity about the library, it's almost impossible to fight against it.
The Ansible extension monkey-patches half the application
Half may be a slight exaggeration, but indeed it's true, the Ansible extension is littered with monkey-patches. It doesn't matter that these are tidy and conservative patches that uninstall themselves when not in use or when incompatible situations are detected, monkey patching! is! wrong! and must! be prevented!
Who could trust any code that subclasses types dynamically, or assigns a method to a class instance? No code should ever do that, or heaven forbid, nor should any test suite. Get this printed on a t-shirt so the message is abundantly clear.
The Ansible extension reimplements the whole module executor
It's true, it didn't even cutpaste the existing code, just blanket reimplements it! All spread out in a rats nest of 10 line methods across a giant inscrutible class hierarchy. The nerve!
Despite replacing a giant chunk of Ansible's lower tiers, the plain truth is that this code is a weird derivative of existing Ansible functionality bolted on the side. Who in their right mind wants to run a setup like that?
The Ansible extension prefers performance over safety
Here's a great one: to avoid super important network round-trips when copying small files, rather than raise an error when the file copy fails, it instead raises the error on the follow-up module execution. How confusing! How can this mess possibly be debugged?
The Ansible extension makes it easy for buggy modules to break things
The dirty secret behind some of that performance? We're reusing state, the horror! And because we're reusing state, if some of that state is crapped over by a broken module, all hell could break loose. The sky might fall, hard disks could be wiped, fortune 500s might cease to exist!
In the absense of collaboration with the authors of such evil code, Mitogen is forced between a rock and a hard place: maintaining a blacklist of known-broken modules and forking a threaded process (fork! with threads! the horror!) to continue the illusion of good performance in spite of absolutely no help from those who could provide it.
Forking support is entirely disabled when targetting Python 2.4, and there is a high chance Mitogen will remove forking support entirely in the future, but in the meantime, use no less than 72 point bold font when discussing one of the library's greatest sins of all.
The Ansible extension rewrites the user's playbook
It's true! Why does a strategy module get to pick which connection plugins are used? It's as if the author believed in eliminating the need for unproductive busywork at all costs for the users of his project, and dared put user before code with a monkey-patch to ensure manual edits across a huge Git repo full of playbooks weren't necessary to allow the code to be tested, you know, by busy people who were already so stressed out about not having enough time to complete the jobs they've been assigned, due to a fundamentally slow and inefficient tool, that they'd risk experimenting with such a low quality project to begin with. The horror!
Summary
If you need more ammunition in your vague, non-specific battle with the future, please file a bug and I will be only too happy to supply it. But please, please, please avoid suggesting things "don't work" or are broken somehow without actually specifying how. It is deeply insulting and a damning indictment on your ability to communicate and openly collaborate in what is otherwise allegedly the context of a free software project.
Hopefully this post should arm such people with legitimate complaints for the foreseeable future. Meanwhile, I will work to improve the library, including as a result of bug reports filed in GitHub.
from Planet Python
via read more
No comments:
Post a Comment