Monday, November 26, 2018

Sunday, November 25, 2018

Kushal Das: Building wheels and Debian packages for SecureDrop on Qubes OS

For the last couple of months, the SecureDrop team is working on a new set of applications + system for the journalists, which are based on Qubes OS, and desktop application written particularly for Qubes. A major portion of the work is on the Qubes OS part, where we are setting up the right templateVMs and AppVMs on top of those templateVMs, setting up the qrexec services and right configuration to allow/deny services as required.

The other major work was to develop a proxy service (on top of Qubes qrexec service) which will allow our desktop application (written in PyQt) to talk to a SecureDrop server. This part finally gets into two different Debian packages.

  1. The securedrop-proxy package: which contains only the proxy tool
  2. The securedrop-client: which contains the Python SDK (to talk to the server using proxy) and desktop client tool

The way to build SecureDrop server packages

The legacy way of building SecureDrop server side has many steps and also installs wheels into the main Python site-packages. Which is something we plan to remove in future. While discussing about this during PyCon this year, Donald Stufft suggested to use dh-virtualenv. It allows to package a virtualenv for the application along with the actual application code into a Debian pacakge.

The new way of building Debian packages for the SecureDrop on Qubes OS

Creating requirements.txt file for the projects

We use pipenv for the development of the projects. pipenv lock -r can create a requirements.txt, but, it does not content any sha256sums. We also wanted to make sure that doing these steps become much easier. We have added a makefile target in our new packaging repo, which will first create the standard requirements.txt and then it will try to find the corresponding binary wheel sha256sums from a list of wheels+sha256sums, and before anything else, it verifies the list (signed with developers’ gpg keys).

PKG_DIR=~/code/securedrop-proxy make requirements

If it finds any missing wheels (say new dependency or updated package version), it informs the developer, the developer then can use another makefile target to build the new wheels, the new wheels+sources do get synced to our simple index hosted on s3. The hashes of the wheels+sources also get signed and committed into the repository. Then, the developer retries to create the requirements.txt for the project.

Building the package

We also have makefile targets to build the Debian package. It actually creates a directory structure (only in parts) like rpmbuild does in home directory, and then copies over the source tarball, untars, copies the debian directory from the packaging repository, and then reverifies each hashes in the project requirements file with the current signed (and also verified) list of hashes. If everything looks good, then it goes to build the final Debian package. This happens by the following environment variable exported in the above mention script.

DH_PIP_EXTRA_ARGS="--no-cache-dir --require-hashes"

Our debian/rules files make sure that we use our own packaging index for building the Debian package.

#!/usr/bin/make -f

%:
        dh $@ --with python-virtualenv --python /usr/bin/python3.5 --setuptools --index-url https://dev-bin.ops.securedrop.org/simple

For example, the following command will build the package securedrop-proxy version 0.0.1.

PKG_PATH=~/code/securedrop-proxy/dist/securedrop-proxy-0.0.1.tar.gz PKG_VERSION=0.0.1 make securedrop-proxy

The following image describes the whole process.

We would love to get your feedback and any suggestions to improve the whole process. Feel free to comment in this post, or by creating issues in the corresponding Github project.



from Planet Python
via read more

Mike Driscoll: PyDev of the Week: Reimar Bauer

This week we welcome Reimar Bauer (@ReimarBauer) as our PyDev of the Week! Reimar is a core developer of the popular Python wiki package, MoinMoin. He has spoken at PyCON DE, FOSDEM and EuroPython about Python. Let’s take a few minutes to get to know him better!

Can you tell us a little about yourself (hobbies, education, etc):

I am a programmer from Jülich, Germany. That‘s a small town between Aachen and Cologne.

I work at the Forschungszentrum Jülich GmbH. Employees research in the fields of energy and the environment, information and brain research with the aim of providing society with options for action facilitating sustainable development.

My work is related to atmospheric science.

copyright: Forschungszentrum Jülich GmbH

I wrote a few books in the past years about data structures and about IDL, one of my first science related programming languages, developed by Ray Sterner in Boulder, Colorado.

In 2012 I was elected to the board of the German Python Software Verband. I helped to organize several Python Events in Germany, several PyConDE and the famous EuroPython in Berlin. I stepped down from this position and I left in 2017 because I needed and wanted to spend more time with my family.

I have been a fellow of the Python Software Foundation since 2013.

I used to play trombone in a Big Band, but had to take a break because I just couldn’t find the time this hobby needs. I enjoy sightseeing, hiking, sociocracy3.0, meeting a lot of different people. And sometimes I play an augmented reality game named ingress.

In 2019 the 10th PythonCamp Cologne Germany, a classic barcamp, will be the next community project Reimar Bauer is involved in.

Why did you start using Python?

It was because of the MoinMoin Software. I had an idea. I wanted to have a Gantt chart view in MoinMoin.

I got all help needed doing that, and learned to use IRC ;), too, and had a start into the Python Open Source World. In 2003 I used 1.5.2 . Later I wrote many plugins for MoinMoin and a description with examples about the API.

What other programming languages do you know and which is your favorite?

I have only one favorite, and that‘s Python. It also plays a role what my colleagues need for their work. I still maintain a large IDL library, but since development froze about 2013, we moved on to Python. I also know Fortran, but stopped using it.

What projects are you working on now?

Since 2016 I have been maintaining and developing the Mission Support System. This project is completely open source, Apache 2.0 licenses. This software is used to interactively plan aircraft missions for atmospheric science.

Because of the project I also have become a maintainer for an amount of conda-forge feed-stocks. And because of my love for pyfilesystem2, I started a qt5 file picker.

Which Python libraries are your favorite (core or 3rd party)?

Pyfilesystem2:
Work with files and directories in archives, memory, the cloud etc. as easily as your local drive. Write code now, decide later where the data will be stored; unit test without writing real files; upload files to the cloud without learning a new API; sandbox your file writing code; etc.
https://pyfilesystem2.readthedocs.io/en/latest/

requests: http for humans
http://docs.python-requests.org/en/master/

netcdf4:
NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.
http://unidata.github.io/netcdf4-python/

borgbackup:
Borgbackup is a de-duplicating backup program. Optionally, it supports compression and authenticated encryption.
http://borgbackup.readthedocs.io/en/stable/

Which Python communities do you want to mention?

The German Python Software Verband, which is focused on the PyConDE and the EuroSciPy conference series but also supports a lot of smaller communities similar to the PSF.

The Euro Python Society, which is focused on the EuroPython conference

The Python Software Foundation – the organisation which over years tied everything together. Organizes the Pycon US and also helps smaller communities with grants.
Thanks to all of you for your time, and for doing such a great job!

What made you decide to work on the MoinMoin project?

The community: Helpful, warm and welcoming folks. I continued working on it because we were able to use MoinMoin as project wiki on any of our measurement campaigns for our institute. There we have to show model forecast data, manage pages with different access rules, provide a lot information to participants. Another use case is to create sphinx latex documents from reST formatted wiki pages.

Can you tell us some of the things you learned as a dev on MoinMoin?

I learned about a lot of tools, how to write proper unit tests, web server options; I learned from bugs, too. I also enjoyed new contacts to communities. Also the time as mentor for students during google summer of code was a great time. We were at the bleeding edge and solved a lot browser based issues or at least found a way to bypass them.

Would you do anything differently if you were able to start over from scratch?

We started with moin-2. But got stuck. A reason may be the unique selling point of wiki itselfs is wide spread meanwhile. Also many comparable tools exist, and many developers who just want to stabilize some project are not that interested in this kind of software mechanism. There are just too many options for this nowadays. As we have already spent much time replacing a lot of our self made model and views in moin-2, I would prefer to finish this.
We currently reunite devs and also build a new core team.

If you like to join us, feel free to get in contact https://mail.python.org/mailman/listinfo/moin-devel

Thanks for doing the interview, Reimar!



from Planet Python
via read more

Test and Code: 54: Python 1994 - Paul Everitt

Paul talks about the beginning years of Python.
Talking about Python's beginnings is also talking about the Python community beginnings.
Yes, it's reminiscing, but it's fun.

Special Guest: Paul Everitt.

Sponsored By:

Support Test and Code

Links:

<p>Paul talks about the beginning years of Python.<br> Talking about Python&#39;s beginnings is also talking about the Python community beginnings.<br> Yes, it&#39;s reminiscing, but it&#39;s fun.</p><p>Special Guest: Paul Everitt.</p><p>Sponsored By:</p><ul><li><a rel="nofollow" href="https://do.co/testandcode">DigitalOcean</a>: <a rel="nofollow" href="https://do.co/testandcode">Get started with a free $100 credit toward your first project on DigitalOcean and experience everything the platform has to offer, such as: cloud firewalls, real-time monitoring and alerts, global datacenters, object storage, and the best support anywhere. Claim your credit today at: do.co/testandcode</a></li></ul><p><a rel="payment" href="https://www.patreon.com/testpodcast">Support Test and Code</a></p><p>Links:</p><ul><li><a title="Python 1994 Panel Discussion PyCon 2017" rel="nofollow" href="https://www.youtube.com/watch?v=iOUrf7WSSLc">Python 1994 Panel Discussion PyCon 2017</a></li><li><a title="&quot;Python 1994&quot;, PyBay2017" rel="nofollow" href="https://www.youtube.com/watch?v=7NrPCsH0mBU">&quot;Python 1994&quot;, PyBay2017</a></li><li><a title="Spam, Spam, Spam, ..." rel="nofollow" href="https://www.youtube.com/watch?v=anwy2MPT5RE">Spam, Spam, Spam, ...</a></li></ul>

from Planet Python
via read more

Podcast.__init__: Using Calibre To Keep Your Digital Library In Order with Kovid Goyal

Digital books are convenient and useful ways to have easy access to large volumes of information. Unfortunately, keeping track of them all can be difficult as you gain more books from different sources. Keeping your reading device synchronized with the material that you want to read is also challenging. In this episode Kovid Goyal explains how he created the Calibre digital library manager to solve these problems for himself, how it grew to be the most popular application for organizing ebooks, and how it works under the covers. Calibre is an incredibly useful piece of software with a lot of hidden complexity and a great story behind it.

Summary

Digital books are convenient and useful ways to have easy access to large volumes of information. Unfortunately, keeping track of them all can be difficult as you gain more books from different sources. Keeping your reading device synchronized with the material that you want to read is also challenging. In this episode Kovid Goyal explains how he created the Calibre digital library manager to solve these problems for himself, how it grew to be the most popular application for organizing ebooks, and how it works under the covers. Calibre is an incredibly useful piece of software with a lot of hidden complexity and a great story behind it.

Preface

  • Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. Go to podcastinit.com/linode to get a $20 credit and launch a new server in under a minute.
  • Visit the site to subscribe to the show, sign up for the newsletter, and read the show notes. And if you have any questions, comments, or suggestions I would love to hear them. You can reach me on Twitter at @Podcast__init__ or email hosts@podcastinit.com)
  • To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
  • Join the community in the new Zulip chat workspace at podcastinit.com/chat
  • Your host as usual is Tobias Macey and today I’m interviewing Kovid Goyal about Calibre, the powerful and free ebook management tool

Interview

  • Introductions
  • How did you get introduced to Python?
  • Can you start by explaining what Calibre is and how the project got started?
  • How are you able to keep up to date with device support in Calibre, given the continual release of new devices and platforms that a user can read ebooks on?
  • What are the main features of Calibre?
    • What are some of the most interesting and most popular plugins that have been creatd for Calibre?
  • Can you describe the software architecture for the project and how it has evolved since you first started working on it?
  • You have been maintaining and improving Calibre for a long time now. What is your motivation to keep working on it?
    • How has the focus of the project and the primary use cases changed over the years that you have been working on it?
  • In addition to its longevity, Calibre has also become a de-facto standard for ebook management. What is your opinion as to why it has gained and kept its popularity?
    • What are some of the competing options and how does Calibre differentiate from them?
  • In addition to the myriad devices and platforms, there is a significant amount of complexity involved in supporting the different ebook formats. What have been the most challenging or complex aspects of managing and converting between the formats?
  • One of the challenges around maintaining a private library of electronic resources is the prevalence of DRM restricted content available through major publishers and retailers. What are your thoughts on the current state of digital book marketplaces?
  • What was your motivation for implementing Calibre in Python?
    • If you were to start the project over today would you make the same choice?
    • Are there any aspects of the project that you would implement differently if you were starting over?
  • What are your plans for the future of Calibre?

Keep In Touch

Picks

  • Tobias
  • Kovid
    • Into Thin Air by John Krakauer
      About how an expedition to climb Everest went wrong. Wonderful account of the difficulties of high altitude mountaineering and the determination it needs.
    • The Steerswoman’s Road by Rosemary Kirstein
      About the spirit of scientific enquiry in a fallen civilization on an alien planet with partial terraforming that is slowly failing.

Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA



from Planet Python
via read more

Ned Batchelder: Coverage.py 5.0a4: the sys.path to hell

Another alpha of Coverage.py 5.0 is available: 5.0a4. This fixes a few problems with the new SQLite-based storage. Please give it a try, especially to experiment with dynamic contexts.

The challenge with this release was something that started as a seemingly simple fix. Coverage.py tries to emulate how Python runs programs, including how the first element of sys.path is set. A few people run coverage with sys.path fully configured, and coverage’s setting of sys.path[0] was breaking their stuff.

The proposal was simple: delete the one line of code that set sys.path[0]. I tried that, and it seemed to work. Fixed!

Not so fast: the Windows builds failed. This started a multi-week adventure of debugging and refactoring. The Windows builds were failing not because of Windows itself, but because on Windows, I don’t use pytest-xdist, which parallelizes tests into worker processes. With xdist, the tests were all passing. Without xdist, a few sys.path-related tests were failing.

It turns out that xdist manipulates sys.path itself, which was masking the fact that I had removed an important step from coverage.py. First thing to do was to adjust my test code so that even with xdist, my tests didn’t get xdist’s path changes.

Then I had to re-think how to adjust sys.path. That required refactoring how I ran the user’s Python code, so that I could apply the path changes a little earlier than I used to. That made me look at how I was testing that layer of code with mocks, and I changed it from explicit dependency injection to implicit mock patching.

A few more little fixes were needed here and there along the way. All told, the “one line” fix ended up being 14 files changed, 587 insertions, 427 deletions.



from Planet Python
via read more

Saturday, October 20, 2018

Exploring Jupyer Notebook to write a Blog

Exploring Jupyer Notebook to write a Blog - gopython - Medium

Jupyter Notebooks is a great platform for doing a quick code experiment and also recording and sharing your analysis with code snippet with others. But today I am trying to explore how we can write blogs with Jupyter Notebook.


TestDriven.io: Working with Static and Media Files in Django

This article looks at how to work with static and media files in a Django project, locally and in production. from Planet Python via read...