Last week Mitogen gained an exciting user in the form of Opsmop, a brand new automation tool undergoing heavy development from Michael DeHaan, the original creator of Ansible.
Opsmop should be conceptually familiar to infrastructure folk, differentiating itself with a clean and more expressive default syntax (plain Python!), a model-driven internal design offering stronger possibilities for pre-run validation, decoupling of resources from their implementation, and now a push mode that exploits unique zero deploy, connection proxying and message routing capabilities available with Mitogen.
Approach
Opsmop uses a Mitogen remote function call to trigger deployments via SSH without having first been installed on remote hosts, transferring Python modules alongside the user's role definitions on demand, without writing temporary files to disk, and gaining a message bus that provides a sound basis for any future functionality or communication pattern. File transfer reuses this bus alongside RPCs, with one network round-trip per file, offering a big performance improvement for small files compared to scp or sftp.
Unlike in Mitogen for Ansible, most role evaluation in Opsmop already runs remotely, including logic like deciding which task to run next, or whether to fetch a file from the controller, eliminating a major cause of latency and bottlenecking. On the controller, Mitogen's scatter/gather functionality is used to schedule evaluations from a single process, streamily processing completions and progress updates sent by tasks as they occur, with file transfer running on demand in the background, and without the need to fork per-task workers or threads.
Opsmop offloads much more processing than the controller-heavy Ansible while maintaining its agentless simplicity. It has rich online communication with a privileged coordinator as in agentful designs like Salt, Chef, Puppet, or mgmt, yet avoids the versioning and operational hassle of preconfigured agents and servers running 24/7, or external dependencies like etcd and ZeroMQ. It avoids any need for target-local configuration as in purely pull-based designs like ansible-pull or chef-solo, while avoiding pull's pitfalls of lacking a coordinator, such as the inability to sequence tasks across machines, share expensive intermediate steps, or selectively grant access to secrets.
From Mitogen's perspective, Opsmop is exciting because it makes heavier use of network import than previous consumers – most of the program really runs on the remote, striking truer to the library's design promise than the Ansible extension. I worried this might reveal lurking nasties, but except for some noisy logs and unintuitive behaviour, so far no showstoppers were hit.
Lessons Learned
This integration provided invaluable perspective, revealing problems and misconceptions fresh eyes encounter with the library, and helpful sanity checks that are currently missing. The experience supplied all the fuel for a good Getting Started guide, a task delayed for over a year mostly due to having no real clue how to approach it.
Opsmop benefits from planned features like asynchronous connect, along with plenty of scope for improvement unique to it, and so there is renewed motivation to get Mitogen's user-reported tickets under control in preparation for another round of development.
Finally it is painfully clear that Mitogen's APIs are fairly nasty on initial approach. As a result, I've begun to think about a better Session API to wrap common tasks in a more intuitive interface.
Conclusion
Despite adopting a complex young library, progress was quick and involved minimal support. From a fresh user with no background to a functionally complete integration only took around a week, an experience that was extraordinarily refreshing to observe, and a candid insight into the origins of the most popular automation tool around.
Opsmop is shaping up to be an excellent option in the automation space, and by delegating connection handling to a library expressly designed for it, enjoys easy access to already-tested functionality like automagic proxying, Docker, Kubernetes and FreeBSD Jail connections, forming a virtuous cycle of free fixes and exciting improvements long into the future.
Until next time!
Just tuning in?
- 2017-09-15: Mitogen, an infrastructure code baseline that sucks less
- 2018-03-06: Quadrupling Ansible performance with Mitogen
- 2018-07-10: Mitogen released!
- 2018-08-30: Transmit optimisations in Mitogen 0.3
from Planet Python
via read more
No comments:
Post a Comment