Friday, April 30, 2021

Montreal Python User Group: Translation sprint for the Python documentation

French, please!

From May 16th to May 31st, Montréal-Python is hosting a translation sprint around the official documentation of Python 3.9 to French.

Why should we translate ?

At the moment speaking, the official Python documentation for the 3.9 version is 54% translated. It’s a good ratio considering that many important parts of it are yet to be translated. One example would be the C API pages that are only 16% translated. In short, we are good with being a bit behind on the translation of advanced topics, as long as the translation of fundamentals is well on the way.

That being said, there is work to be done if we want the 3.9 doc to be as accessible as possible to people who are getting started with Python. Here is a summary of topics that are in need of attention:

  • What's New: 9%
  • Howto: 67%
  • Install: 50%
  • Config Parser: 2%
  • Context Vars: 2%
  • Datetime: 79%
  • Decimal: 19%
  • Doctest: 0%
  • Formatter: 0%
  • Functools: 72%
  • Ipaddress: 1%
  • Locale: 22%
  • Math: 86%
  • shutil: 12%
  • Types: 4%
  • xml.etree: 9%

On top of being useful to the community, translating a technical document is one of the best to gain the motivation needed for understanding a module in depth and for becoming an expert on the topic at hands.

This is why we get together to make progress on this topic towards the end of May.

How the sprint will work

Sunday May 16th at 11am, Montreal time, we will meet virtually in a video call to go through the necessary tooling for the translation project. Do not worry if you cannot attend: there will be a write up following the video call with the essential information.

After that, we will meet two times a week (Tuesdays and Thursdays) to discuss our problems and to find together the best translations for particularly niche matters. These meetings are completely optional as we will be staying in touch on Slack throughout the week.

Towards the end of the month we will work on tying some loose ends in order to present the results at Montréal-Python 86 (Jubilant Quadriceps).

I am interested! How do I sign up?

You can get started already by joining us on our Slack in the #hackathon-docs-python-fr channel. Get an invite to the Slack here: https://mtlpy.org/fr/slackin



from Planet Python
via read more

William Minchin: Jinja Filters 1.0.0 for Pelican Released

Jinja Filters is a plugin for Pelican, a static site generator written in Python.

Jinja Filters provides a selection of functions (called filters) for templates to use when building your website. They are packaged for Pelican, but may prove useful for other projects that make use of Jinja2.

Installation

The easiest way to install Jinja Filters is through the use of pip. This will also install the required dependencies automatically.

pip install minchin.pelican.jinja_filters

Then, in your pelicanconf.py file, add Jinja Filters to your list of plugins:

PLUGINS = [
    # ...
    'minchin.pelican.jinja_filters',
    # ...
]

And that’s it! The filters are now available for use in your templates.

Usage

At present, the plugin includes the following filters:

  • datetime — allows you to change to format displayed for a datetime object. Optionally supply a datetime format string to get a custom format.
  • article_date — a specialized version of datetime that returns datetimes as wanted for article dates; specifically Friday, November 4, 2016.
  • breaking_spaces — replaceds non-breaking spaces (HTML code &nbsp) with normal spaces.
  • titlecase — Titlecases the supplied string

For example, within your theme templates, you might have code like:

<span class="published">
    Article Published 
</span>

gives:

Article Published Friday, November 4, 2016

Or with your own dateformat:

<span class="published">
    Article Published 
</span>

gives:

Article Published Nov 04, 2016

Filters can also be chained, or applied in sequence. For example to remove breaking spaces and then titlecase a category name, you might have code like:

<a href="/">
    
</a>

Known Issues

An issue, as such, is that there is no formal test suite. Testing is currently limited to my in-use observations. I also run a basic check upon uploaded the package to PyPI that it can be downloaded and loaded into Python.

The package is tested in Python 3.5; compatibility with other version of Python is unknown.

Code

The code for this project is available on GitHub. Contributions are welcome!



from Planet Python
via read more

William Minchin: Jinja Filters 1.1.0 & 2.1.0 for Pelican Released

Jinja Filters is a plugin for Pelican, a static site generator written in Python.

Jinja Filters provides a selection of functions (called filters) for templates to use when building your website. They are packaged for Pelican, but may prove useful for other projects that make use of Jinja2.

This Release

This post actually covers three releases:

  • v1.1.0 doesn’t add any functionality or bugfixes directly, but is designed to point users to the new v2 releases.
  • v2.0.0 reorganized the project codebase to make this work as a “namespace plugin”. Added by Pelican 4.5 is a feature to automatically activate such plugins. It also transfers the code repo to the Pelican-Plugins organization and moved the PyPI package to pelican-jinja-filters.
  • v2.1.0 adds two filters — merge_date_url and datetime_from_period. It also lowers the minimum Pelican version to 3 (from 4.5). Under the hood, it also updates the local development infrastructure to work better on Windows.

Upgrading

to v1.1.0

To upgrade simply use pip:

pip install minchin.pelican.jinja-filters --upgrade

v1.1.0 actually depends on v2.1.0 or newer, which will automatically be installed.

As a side, this (having one version of a package rely on another version of the “same” package) is generally not desirable or even possible. But PyPI/pip has no real concept of package re-naming, and the two different package names is what makes this work.

If you run v1.1.0, you will get a warning message when you generate your site with Pelican encouraging you to upgrade to v2. This is mostly for those who won’t stumble upon this blog entry! That said, the plugin will continue to work as it has previously without further effort on your part.

to v2.0.0

v2.0.0 has a different package name, so you’ll have to uninstall the old package and install the new one. Again, pip is the simplest way:

pip install pelican-jinja-filters --upgrade
pip uninstall minchin.pelican.jinja-filters

The new package name and file layout is to make the plugin a “namespace plugin”. Namespace plugins are actually a really cool idea that if you create your package in the right way, your “host” program can find the plugins simply by having them installed on your system! For Pelican, they need to be in the pelican.plugins namespace.

Two caveats of this approach is that you’ll need Pelican version 4.5 (or later) to automatically load these namespace plugins, and (at least if my understanding is correct) you have to either rely on namespace plugins alone OR the PLUGINS setting of your pelicanconf.py; i.e. if you specify PLUGINS in your settings, auto-loading of namespace plugins is turned off. Neither of these are deal breakers, but this background may prove useful in debugging your setup. Overall, I think namespace plugins are an awesome idea, and I hope it doesn’t take too long to get everything switched over.

So if you’re using other non-namespace plugins, or a Pelican version before 4.5, you’ll also need to update your pelicanconf.py with the new plugin name:

# pelicanconf.py

PLUGINS = [
    # others...
    # minchin.pelican.jinja_filters  # <-- remove this line
    "pelican.plugins.jinja_filters",
]

Finally, v2.0.0 bumps the minimum Pelican version up to 4.5; if you’re using an older version of Pelican and don’t want to upgrade yet, then use v2.1.0 of the plugin.

to v2.1.0

Assuming you’ve done the steps listed above to upgrade to v2.0.0, pip remains the simplist way to upgrade:

pip install pelican-jinja-filters --upgrade

This version lowers the minimum Pelican version 3 (which is something I needed to incrementally upgrade my site; I’m stuck at v3.7.1 for a bit yet while I upgrade some other plugins).

It also added two new filters: merge_date_url and datetime_from_period. I added these in particular for use on the period archive pages (i.e. yearly, monthly, and daily archives) and will be used by the next version of seafoam (the theme on this site).

Details on the New Features (of v2.1.0)

datetime_from_period

By way of background, Pelican feeds a tempate variable called period to the template when generating period archive pages. However, the variable is a tuple with the month (when present) as a string. If the text is already formatted the way you want (and the default is generally sensible), then you can just display it as is. However, this filter can turn that tuple into a “proper” datetime.datetime object to be further processed.

You might use it like this:


(datetime, as used here, is another filter provided by this plugin.)

As an implementation note, if the month is not supplied (e.g. on a yearly archive page), this filter will assume it is January; if the date is not supplied (i.e. on a yearly or monthly archive page), the 1st is assumed.

merge_date_url

When given a datetime (on the left to operate on), and provided with a period archive url (typically a Pelican setting like YEAR_ARCHIVE_URL), it will “apply” the date to the URL.

So the two might be used together like this (example pulled from the pending seafoam release):

<a href=" /
         ">
    
</a>

while will result in the generated HTML:

<a href="https://blog.minchin.ca/posts/2021/>2021</a>

Thoughts on These Releases and the Future

This part is more of a personal than technical note.

Eagle-eyed among you may notice that the initial version 2 release happened back in August. I was actively involved in the process, but it’s taken some time to decide what to make the changes that involved.

Overall, I still think moving the plugin to a organization repo is a good thing. This should help with visibility, and hopefully this will result in more people contributing to it. But it is also weird to “let go” of this code, in the sense that it is no longer under my username on GitHub.

The biggest technical changes on the backend side involved a complete replacement of the release machinery. Previously I’d been using a home-grown script (minchin.releaser) to do releases; it was mine, and I could punch out a release from a single command. That was replaced by AutoPub, which is a “bot” to automatically publish releases from your project’s continuous integration system; this has the benefit of basically being completely automated. And for a multi-user environment (like the hope is with the new code repo location), this new setup makes a lot of sense.

One oddity of giving up, or a least sharing, ownership of the code, is that I had Justin Mayer (who manages both the Pelican project and Pelican-Plugins) review my proposed changes. I think my code is better for it; I know for many of the changes I ended up writing a whole bunch about what and why I wanted the changes I was proposing, sometimes longer than the actual code in question.

The last big change was a switch from a setup.py/requirements.txt setup (supported by pip-tools) to a poetry/pyproject.toml setup. I don’t know if this is “the way of the future” (as some have made it out to be), but this is the change I’m least sold on. poetry has proved tricky to configure at times, is very opinionated, and sometimes hides it’s logic under the covers. As well, I’m not sure it’s solved any problem I didn’t already have solved. But I’ll use it for projects like this, and just count it among the (slight) costs of working with others.

Moving forward, I’m not sure if every release will get a release post. I suspect the releases I’m involved in will get a post, but hopefully there will be some without my involvement!

Now, only 11 more plugins to go! I want to move all the plugins I use to namespace plugins and then upgrade from Pelican 3.7.1 to 4.6 (or whatever the then-current version is). I’m a little bit closer. :)



from Planet Python
via read more

Talk Python to Me: #314 Ask us about modern Python projects and tools

Here's an episode that I did not see coming! Sebastian Witowski and I put together a live stream ask me anything (AMA) as a follow up to some of the ideas around his recent course, Modern Python Projects. We dove deep in comparisons of Poetry vs. pip vs pyenv and answered questions like do you need to use Docker? When should you? and so on. <br/> <br/> After the AMA was over, I realized it would also make a great podcast too. So here go you! This is our AMA with Sebastian around all the ideas of modern Python workflows. I hope you enjoy it!<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Sebastian on Twitter</b>: <a href="https://twitter.com/SebaWitowski" target="_blank" rel="noopener">@SebaWitowski</a><br/> <b>Modern Python Projects Course</b>: <a href="/modern-python-projects" target="_blank" rel="noopener">talkpython.fm/modern-python-projects</a><br/> <b>YouTube Live Stream edition</b>: <a href="https://www.youtube.com/watch?v=vr1B3Fd3lFg" target="_blank" rel="noopener">youtube.com</a><br/></div><br/> <strong>Sponsors</strong><br/> <br/> <a href='https://ift.tt/3drA2lU> <a href='https://ift.tt/2PVc9qH Python Training</a>

from Planet Python
via read more

Doug Hellmann: sphinxcontrib-spelling 7.2.1

What’s new in 7.2.1? release note for spelling_verbose configuration option add verbose configuration option (contributions by 12rambau)


from Planet Python
via read more

Switching From Spreadsheets to Neptune.ai and How It Pushed My Model Building Process to the Next Level

Many ML projects, including Kaggle competitions, have a similar workflow. You start with a simple pipeline with a benchmark model.  Next, you begin incorporating improvements: adding features, augmenting the data, tuning the model… On each iteration, you evaluate your solution and keep changes that improve the target metric. This workflow involves running a lot of […]

The post Switching From Spreadsheets to Neptune.ai and How It Pushed My Model Building Process to the Next Level appeared first on neptune.ai.



from Planet SciPy
read more

Stack Abuse: Spring Boot and Flask Microservice Discovery with Netflix Eureka

Introduction

In this guide, we'll be utilizing Netflix Eureka, a microservice discovery service to combine a Spring Boot microservice with a Flask microservice, bridging services written in totally different programming languages and frameworks.

We'll be building two services - The End-User Service, which is a Spring Boot service oriented at the end-user, that collects data and sends it to the Data-Aggregation Service - a Python service, using Pandas to perform data aggregation, and return a JSON response to the End-User Service.

Netflix Eureka Serice Discovery

When switching from a monolith codebase to a microservice-oriented architecture - Netflix built a plethora of tools that helped them overhaul their entire architecture. One of the in-house solutions, which was subsequently released to the public is Eureka.

Netflix Eureka a service discovery tool (also known as a lookup server or service registry), that allows us to register multiple microservices, and handles request routing between them.

It's a central hub where each service is registered, and each of them communicates with the rest through the hub. Instead of sending REST calls via hostnames and ports - we delegate this to Eureka, and simply call the name of the service, as registered in the hub.

To achieve this, a typical architecture consists of a few elements:

eureka microservice architecture

You can spin off the Eureka Server in any language that has a Eureka wrapper, though, it's most naturally done in Java, through Spring Boot, since this is the original implementation of the tool, with official support.

Each Eureka Server can register N Eureka Clients, each of which is typically an individual project. These can also be done in any language or framework, so each microservice uses what's most suitable for their task.

We'll have two clients:

  • End-User Service (Java-based Eureka Client)
  • Data-Aggregation Service (Python-based Eureka Client)

Since Eureka is a Java-based project, originally meant for Spring Boot solutions - it doesn't have an official implementation for Python. However, we can use a community-driven Python wrapper for it:

With that in mind, let's create an Eureka Server first.

Creating a Eureka Server

We'll be using Spring Boot to create and maintain our Eureka Server. Let's start off by making a directory to house our three projects, and within it a directory for our server:

$ mkdir eureka-microservices
$ cd eureka-microservices
$ mkdir eureka-server
$ cd eureka-server

The eureka-server directory will be the root directory of our Eureka Server. You can start a Spring Boot project here through the CLI:

$ spring init -d=spring-cloud-starter-eureka-server

Alternatively, you can use Spring Initializr and include the Eureka Server dependency:

spring initializr eureka server

If you already have a project and just wish to include the new depdenency, if you're using Maven, add:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-eureka-server</artifactId>
    <version>${version}</version>
</dependency>

Or if you're using Gradle:

implementation group: 'org.springframework.cloud', name: 'spring-cloud-starter-eureka-server', version: ${version}

Regardless of the initialization type - the Eureka Server requires a single anotation to be marked as a server.

In your EndUserApplication file class, which is our entry-point with the @SpringBootApplication annotation, we'll just add an @EnableEurekaServer:

@SpringBootApplication
@EnableEurekaServer
public class EurekaServerApplication {
    public static void main(String[] args) {
        SpringApplication.run(EurekaServerApplication.class, args);
    }
}

The default port for Eureka Servers is 8761, and it's also recommended by the Spring Team. Though, for good measure, let's set it in the application.properties file as well:

server.port=8761

With that done, our server is ready to run. Running this project will start up the Eureka Server, available at localhost:8761:

eureka server

Note: Without registering any services, Eureka may incorrectly claim an UNKNOWN instance is up.

Creating a Eureka Client - End User Service in Spring Boot

Now, with our server spun up and ready to register services, let's go ahead and make our End-User Service in Spring Boot. It'll have a single endpoint that accepts JSON data regarding a Student. This data is then sent as JSON to our Data Aggregation Service that calculates general statistics of the grades.

In practice, this operation would be replaced with much more labor-intensive operations, which make sense to be done in dedicated data processing libraries and which justify the use of another service, rather than performing them on the same one.

That being said, let's go back and create a directory for our End-User Service:

$ cd..
$ mkdir end-user-service
$ cd end-user-service

Here, let's start a new project via the CLI, and include the spring-cloud-starter-netflix-eureka-client dependency. We'll also add the web dependency since this application will actually be facing the user:

$ spring init -d=web, spring-cloud-starter-netflix-eureka-client

Alternatively, you can use Spring Initializr and include the Eureka Discovery Client dependency:

spring initializr eureka client

If you already have a project and just wish to include the new depdenency, if you're using Maven, add:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
    <version>${version}</version>
</dependency>

Or if you're using Gradle:

implementation group: 'org.springframework.cloud', name: 'spring-cloud-starter-netflix-eureka-client', version: ${version}

Regardless of the initialization type - to mark this application as a Eureka Client, we simply add the @EnableEurekaClient annotation to the main class:

@SpringBootApplication
@EnableEurekaClient
public class EndUserServiceApplication {
    public static void main(String[] args) {
        SpringApplication.run(EndUserServiceApplication.class, args);
    }
    
    @LoadBalanced
    @Bean
    RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

Note: Alternatively, you can use the @EnableDiscoveryClient annotation, which is a more wide-encompassing annotation. It can refer to Eureka, Consul or Zookeper, depending on which tool is being used.

We've also defined a @Bean here, so that we can @Autowire the RestTemplate later on in our controller. This RestTemplate will be used to send a POST request to the Data Aggregation Service. The @LoadBalanced annotation signifies that our RestTeamplate should use a RibbonLoadBalancerClient when sending requests.

Since this application is a Eureka Client, we'll want to give it a name for the registry. Other services will refer to this name when relying on it. The name is defined in the application.properties or application.yml file:

server.port = 8060
spring.application.name = end-user-service
eureka.client.serviceUrl.defaultZone = http://localhost:8761/eureka
server:
    port: 8060
spring:
    application:
        name: end-user-service
eureka:
    client:
      serviceUrl:
        defaultZone: http://localhost:8761/eureka/

Here, we've set the port for our application, which Eureka needs to know to route requests to it. We've also specified the name of the service, which will be referenced by other services.

Running this application will register the service to the Eureka Server:

INFO 3220 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8060 (http) with context path ''
INFO 3220 --- [           main] .s.c.n.e.s.EurekaAutoServiceRegistration : Updating port to 8060
INFO 3220 --- [nfoReplicator-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_END-USER-SERVICE/DESKTOP-8HAKM3G:end-user-service:8060 - registration status: 204
INFO 3220 --- [           main] c.m.e.EndUserServiceApplication          : Started EndUserServiceApplication in 1.978 seconds (JVM running for 2.276)
INFO 3220 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_END-USER-SERVICE/DESKTOP-8HAKM3G:end-user-service:8060 - Re-registering apps/END-USER-SERVICE
INFO 3220 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_END-USER-SERVICE/DESKTOP-8HAKM3G:end-user-service:8060: registering service...
INFO 3220 --- [tbeatExecutor-0] com.netflix.discovery.DiscoveryClient    : DiscoveryClient_END-USER-SERVICE/DESKTOP-8HAKM3G:end-user-service:8060 - registration status: 204

Now, if we visit localhost:8761, we'll be able to see it registered on the server:

eureka server registered service

Now, let's go ahead and define a Student model:

public class Student {
    private String name;
    private double mathGrade;
    private double englishGrade;
    private double historyGrade;
    private double scienceGrade;
    
    // Constructor, getters and setters and toString()
}

For a student, we'll want to calculate some summary statistics of their performance, such as the mean, minimum and maximum of their grades. Since we'll be using Pandas for this - we'll leverage the very handy DataFrame.describe() function. Let's make a GradesResult model as well, that'll hold our data once returned from the Data Aggregation Service:

public class GradesResult {
    private Map<String, Double> mathGrade;
    private Map<String, Double> englishGrade;
    private Map<String, Double> historyGrade;
    private Map<String, Double> scienceGrade;
    
    // Constructor, getters, setters and toString()
}

With the models done, let's make a really simple @RestController that accepts a POST request, deserializes it into a Student and sends it off to the Data Aggregation service, which we haven't made yet:

@RestController
public class HomeController {
    @PostMapping("/student")
    public ResponseEntity<String> student(@RequestBody Student student) {
        RestTemplate restTemplate = new RestTemplate();
        GradesResult grades = restTemplate.getForObject("http://data-aggregation-service/calculateGrades", GradesResult.class);

        return ResponseEntity
            .status(HttpStatus.OK)
            .body(String.format("Sent the Student to the Data Aggregation Service: %s \nAnd got back:\n %s", student.toString(), gradesResult.toString()));
    }
}

This @RestController accepts a POST request, and deserializes its body into a Student object. Then, we're sending a request to our data-aggregation-service, which isn't yet implemented, as it will be registered on Eureka, and we pack the JSON results of that call into our GradesResult object.

Note: If the serializer has issues with constructing the GradesResult object from the given result, you'll want to manually convert it using Jackson's ObjectMapper:

String result = restTemplate.postForObject("http://data-aggregation-service/calculateGrades", student, String.class);
ObjectMapper objectMapper = new ObjectMapper();
GradesResult gradesResult = objectMapper.readValue(result, GradesResult.class);

Finally, we print the student instance we've sent as well as the grades instance we constructed from the result.

Now, let's go ahead and create the Data Aggregation Service.

Creating a Eureka Client - Data Aggregation Service in Flask

The only missing component is the Data Aggregation Service, which accepts a Student, in JSON format and populates a Pandas DataFrame, performs certain operations and returns the result back.

Let's create a directory for our project and start a virtual environment for it:

$ cd..
$ mkdir data-aggregation-service
$ python3 -m venv flask-microservice

Now, to activate the virtual environment, run the activate file. On Windows:

$ flask-microservice/Scripts/activate.bat

On Linux/Mac:

$ source flask-microservice/bin/activate

We'll be spinning up a simple Flask application for this, so let's install the dependencies for both Flask and Eureka via pip in our activated environment:

(flask-microservice) $ pip install flask pandas py-eureka-client requests

And now, we can create our Flask application:

$ touch flask_app.py

Now, open the flask_app.py file and import Flask, Pandas and the Py-Eureka Client libraries:

from flask import Flask, request
import pandas as pd
import py_eureka_client.eureka_client as eureka_client

We'll be using Flask and request to handle our incoming requests and return a response, as well a spin up a server. We'll be using Pandas to aggregate data, and we'll use the py_eureka_client to register our Flask application to the Eureka Server on localhost:8761.

Let's go ahead and set this application up as a Eureka Client and implement a POST request handler for the student data:

rest_port = 8050
eureka_client.init(eureka_server="http://localhost:8761/eureka",
                   app_name="data-aggregation-service",
                   instance_port=rest_port)

app = Flask(__name__)

@app.route("/calculateGrades", methods=['POST'])
def hello():
    data = request.json
    df = pd.DataFrame(data, index=[0])
    response = df.describe().to_json()
    return response

if __name__ == "__main__":
    app.run(host='0.0.0.0', port = rest_port)

Note: We have to set the host to 0.0.0.0 to open it to external services, lest Flask refuse them to connect.

This is a pretty minimal Flask app with a single @app.route(). We've extracted the incoming POST request body into a data dictionary through request.json, after which we've made a DataFrame with that data.

Since this dictionary doesn't have an index at all, we've manually set one.

Finally, we've returned the describe() function's results as JSON. We haven't used jsonify here since it returns a Response object, not a String. A Response object, when sent back would contain extra \ characters:

{\"mathGrade\":...}
vs
{"mathGrade":...}

These would have to be escaped, lest they throw off the deserializer.

In the init() function of eureka_client, we've set the URL to our Eureka Server, as well as set the name of the application/service for discovery, as well as supplied a port on which it'll be accessible. This is the same information we've provided in the Spring Boot application.

Now, let's run this Flask application:

(flask-microservice) $ python flask_app.py

And if we check our Eureka Server on localhost:8761, it's registered and ready to receive requests:

eureka server registered service

Calling Flask Service from Spring Boot Service using Eureka

With both of our services up and running, registered to Eureka and able to communicate with each other, let's send a POST request to our End-User Service, containing some student data, which will in turn send a POST request to the Data Aggregation Service, retrieve the response, and forward it to us:

$ curl -X POST -H "Content-type: application/json" -d "{\"name\" : \"David\", \"mathGrade\" : \"8\", \"englishGrade\" : \"10\", \"historyGrade\" : \"7\", \"scienceGrade\" : \"10\"}" "http://localhost:8060/student"

This results in a response from the server to the end-user:

Sent the Student to the Data Aggregation Service: Student{name='David', mathGrade=8.0, englishGrade=10.0, historyGrade=7.0, scienceGrade=10.0}
And got back:
GradesResult{mathGrade={count=1.0, mean=8.0, std=null, min=8.0, 25%=8.0, 50%=8.0, 75%=8.0, max=8.0}, englishGrade={count=1.0, mean=10.0, std=null, min=10.0, 25%=10.0, 50%=10.0, 75%=10.0, max=10.0}, historyGrade={count=1.0, mean=7.0, std=null, min=7.0, 25%=7.0, 50%=7.0, 75%=7.0, max=7.0}, scienceGrade={count=1.0, mean=10.0, std=null, min=10.0, 25%=10.0, 50%=10.0, 75%=10.0, max=10.0}}

Conclusion

In this guide, we've created a microservice environment, where one service relies on another, and hooked them up using Netflix Eureka.

These services are built using different frameworks, and different programming languages - though, through REST APIs, communicating between them is straightforward and easy.

The source code for these two services, including the Eureka Server is available on Github.



from Planet Python
via read more

Real Python: The Real Python Podcast – Episode #58: Podcast Rewind With Guest Highlights for 2020-2021

This week's show is a bit different. We are taking a well-deserved short break, but we still wanted to share an episode with you. This rewind episode highlights clips from the many interviews over the past year or so of the show.


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]



from Planet Python
via read more

The Real Python Podcast – Episode #58: Podcast Rewind With Guest Highlights for 2020-2021

This week's show is a bit different. We are taking a well-deserved short break, but we still wanted to share an episode with you. This rewind episode highlights clips from the many interviews over the past year or so of the show.


[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]



from Real Python
read more

Thursday, April 29, 2021

Zero to Mastery: Python Monthly 💻🐍 April 2021

17th issue of Python Monthly! Read by 20,000+ Python developers every month. This monthly Python newsletter is focused on keeping you up to date with the industry and keeping your skills sharp, without wasting your valuable time.

from Planet Python
via read more

Python Engineering at Microsoft: Our support for the Python community

Microsoft has been a sponsor of the Python Software Foundation since 2006, and we are excited to announce that this year we have increased our sponsorship of the Python Software Foundation to the new top Visionary level. We wanted to take this opportunity to share why Python is so dear to our hearts and the ways we contribute to the Python community across Microsoft beyond direct financial support.

Python and our core mission of developer productivity

In Microsoft’s Developer Division we have a core mission of helping every developer on the planet achieve more. We have had over 20 years of history of building programming languages and tools for software developers. Along the way we have created open-source programming languages like C# and TypeScript and made our developer tools in Visual Studio Code free for developers working with any programming languages, on any platform.

The rapid rise of Python over the past 5 years is well documented and impressive. Python’s rise has been driven primarily by the growth of data science and aided by its long history of scripting, web development, use in education, and a careful balance of the language’s productivity with ease-of-use. Python, a language with a strong emphasis on developer productivity, has been dear to our hearts and aligns closely with our mission.

Supporting the Python Community

The Python community is what really makes Python special, and many of us feel as Brett Cannon once said so eloquently: “I came for the language, but I stayed for the community”. This community is backed by the countless Python conferences, meetups, online communities, chat rooms, and GitHub projects that give Python developers a place to connect with others. The Python community is a diverse community that prides itself in being inclusive, and at Microsoft diversity and inclusion is a top priority.

That is why we are proud to be long-time sponsors of PyCon US, having been the top-tier Keystone sponsors of the event for four years and continuing this year with our visionary sponsorship of PyCon US. At PyCon US we have sponsored the PyLadies auction for the past 3 years.

Our sponsorship of Python events extends around the world including in emerging regions. We feel privileged to have participated in and sponsored events like: EuroPython, DjangoCon US, PyCon India, PyCon Africa, PyCon JP, PyCon AU, PyParis, and PyData. In the case of PyCascades, we have also helped organize the event with Nina Zakharenko, a Principal Cloud Advocate at Microsoft, on the PyCascades steering council.

Microsoft also has several developers across the company contributing to the development of the Python language. We have at the time of writing, 5 core developers who contribute part time to the development of CPython Brett Cannon, Steve Dower, Guido van Rossum, Eric Snow, and Barry Warsaw.

If you haven’t heard the news yet, Guido van Rossum, the creator of the Python language, recently joined Microsoft as a distinguished engineer and is currently exploring performance improvements to CPython. Brett Cannon has been a Python core developer for more than 18 years and is currently working on direction of the language through his involvement on the steering council, the community conduct through the PSF Conduct Working Group, and improving Python’s packaging story through driving standards and creation of shared libraries.

Eric Snow has been working on multi-threaded performance with his work on Subinterpreters (PEP 554), Steve Dower acts as build manager and CPython expert for Windows since the 3.5 release, and Barry Warsaw continues to bring his unique brand of humor, technical insight, and historical perspective across the entire language through his membership on the Steering Council, Python Security Response Team, and Python (former and current) Release Managers list.

As part of our $150K financial sponsorship of the PSF, we will be focusing our funds to the Packaging Working Group to help with development costs for further improvements to PyPI and the packaging ecosystem. With recently disclosed security vulnerabilities, trusted supply chain is a critical issue for us and the Python community, and we are excited to help contribute to long-term improvements.

Microsoft also employs several core contributors and maintainers of key open-source projects in the Python ecosystem, including pandas, Dask, Jupyter, nteract, scikit-learn and Apache Arrow.

Open sourcing our own tools and services

We believe that we should open source as much of the work we do as possible because it enables developers to have more flexibility when using our products and contributing back to the open-source community advances the state of the art for everyone. While not all the work we do makes sense to operate under open-source governance, we strive to structure our projects such that we can share major components with the open-source community.

As part of our support for Python in Visual Studio Code, we have open sourced the core Python extension, the Jupyter extension, the debugpy debug engine, and the pyright type checker. We have also integrated many open-source components from the community to help support the vibrant set of tools including: Pylint, Flake8, Black, Poetry, and Jedi. We have also made contributions back to many of the open-source projects we use, including packaging, PyDev.Debugger, jedi-language-server, and pygls.

With our Pylance IntelliSense engine, we are contributing to the advancement of type checking by open sourcing pyright, the underlying type-checker. We have made significant improvements in the speed and accuracy of the pyright so that it is fast enough to be used interactively in an editor. We have also contributed to the development of several typing PEPs (e.g. PEP 647), and improved type-stubs and typing implementations in libraries such as PyTorch. Improvements to type checking makes it easier for all editors to have basic auto-complete, and helps developers use type checking to improve quality and reliability of Python code.

In Azure, we have open-sourced our serverless Azure Functions runtime and the Azure Functions Python worker, enabling you to run and debug functions locally and host them in any cloud by using Kubernetes. We have also open-sourced our Azure App Service Oryx build engine and Python runtime image, our Azure CLI and the knack CLI framework, and the Azure SDK for Python. Our Citus extension for PostgreSQL enables anyone to run a horizontally scaled PostgreSQL cluster, and is the extension that powers our managed Azure Database for PostgreSQL – Hyperscale (Citus) offering.

Get started with Python today!

Most of all we want you to be able to build great things with Python, and we are committed to supporting Python as a first-class language across our products and services. To learn how you can start using Python today be sure to check out Python on Windows, Python in Visual Studio Code, and Python in Azure.

To connect with us and stay up to date on all things Python at Microsoft, be sure to check out our Python Blog, follow @PythonVSCode on twitter, and come chat with us on the Python at Microsoft Discord community.

The post Our support for the Python community appeared first on Python.



from Planet Python
via read more

Python⇒Speed: The hidden performance overhead of Python C extensions

Python is slow, and compiled languages like Rust, C, or C++ are fast. So when your application is too slow, rewriting some of your code in a compiled extension can seem like the natural approach to speeding things up.

Unfortunately, compiled extensions are sometimes actually slower than the equivalent Python code. And even when they’re faster, the performance improvement might be far less than you’d imagine, due to hidden overhead caused by two factors:

  1. Function call overhead.
  2. Serialization/deserialization overhead.

Let’s see where these hidden performance overheads comes from, and then see some solutions to get around them.

Problem #1: Call overhead

The first performance overhead we’re going to face is that of function calls. Let’s write a function in Cython, a Python variant language that compiles to C, and see how long it takes to run it.

Here’s the Cython function:

def do_nothing():
    pass

We can use the IPython %timeit magic function to measure performance:

In [1]: from overhead_cythong import do_nothing

In [2]: %timeit do_nothing()
30 ns ± 0.0352 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

In [3]: def py_do_nothing(): pass

In [4]: %timeit py_do_nothing()
62.5 ns ± 0.114 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Calling the Cython function is faster than calling a Python function call, it’s true. But even 30 nanoseconds is rather slow by the standards of compiled languages: for comparison, a C function called by another C function might take only 3 nanoseconds, or much less if it gets inlined.

Problem #2: (De)serialization overhead

Beyond the overhead of calling the extension, there is the overhead of getting arguments into the function, and getting the result back. The way Python represents objects is different than how C or Rust represent it, and so a conversion process is necessary. And the conversion code also needs error handling in case it fails.

The result is more overhead. For example, here’s a Cython function that takes a Python list of integers, sums them, and returns the result:

def sum(values):
    cdef long result = 0
    cdef long v
    for v in values:
        result += v
    return result

We can compare performance to Python:

In [1]: from example_cython import sum as cython_sum

In [2]: l = list(range(1000000))

In [3]: sum(l), cython_sum(l)
Out[3]: (499999500000, 499999500000)

In [4]: %timeit sum(l)
7.64 ms ± 27.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [5]: %timeit cython_sum(l)
6.99 ms ± 29.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The compiled Cython code is no faster than Python’s built-in sum(). And that’s not surprising: sum() is written in C, and the actual math is quite fast as we’ll see below. All the runtime is spent converting Python objects into C integers.

So how do we solve these two forms of overhead?

Solution #1: Manage data inside the extensions

The first solution is to manage all of our data using the Rust or C extension, which means we don’t have all the serialization overhead.

This is what NumPy does with arrays: in theory, it could have used Python lists, but then it would have suffered from (de)serialization. So instead, NumPy arrays internally store numbers not as Python lists of Python integers, but as C arrays of C integers. Operations on NumPy arrays therefore don’t require deserializing every entry in the array.

For example, we can create a NumPy array that is a range of numbers. That will involve creating a C array with C integers, much less work (and much less memory!) than a Python list of Python integers. We can then sum it, and that logic will all be in C, with no need for deserialization except for the final sum:

In [1]: import numpy as np

In [2]: l = list(range(1_000_000))

In [3]: arr = np.arange(1_000_000)

In [4]: type(arr)
Out[4]: numpy.ndarray

In [5]: sum(l), arr.sum()
Out[5]: (499999500000, 499999500000)

In [6]: %timeit sum(l)
7.68 ms ± 26.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [7]: %timeit arr.sum()
620 µs ± 11 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The NumPy code takes less than a millisecond: it is 12× faster than Python. Success!

Solution #2: Move more work into the extension

Another approach we can take is moving more of the calculation into the extension. In all of our examples we’ve been summing a range of numbers, typically 0 to 999,999. If that’s all we need to do, we don’t have to create a whole list of numbers in Python, we can just write a Cython function that sums a range of integers:

def sum_range(long start, long end):
    cdef long i, result
    result = 0
    for i in range(start, end):
        result += i
    return result

We can measure the performance:

In [1]: from example_cython import sum_range

In [2]: sum(list(range(1_000_000))), sum_range(0, 1_000_000)
Out[2]: (499999500000, 499999500000)

In [3]: %timeit sum_range(0, 1_000_000)
306 µs ± 359 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)

That’s pretty good, even faster than NumPy’s implementation, plus we don’t have the overhead of creating a whole array.

But we can do better!

Let’s switch languages, and try out Rust. Unfortunately, the Rust Python bindings via the PyO3 library have a lot higher overhead for calls and (de)serialization than Cython; PyO3 is much newer than Cython, so hopefully it will improve with time. On the other hand, Rust has memory safety, thread safety, and a much higher-level language that still has the same performance as C/C++.

In Rust, we can use a range object that is not that different from Python’s slices:

#[pyfunction]
fn sum_range(start: u64, end: u64) -> u64 {
    assert!(start <= end);
    (start..end).sum()
}

We can then measure the performance:

In [1]: from example_rust import sum_range

In [2]: sum(list(range(1_000_000))), sum_range(0, 1_000_000)
Out[2]: (499999500000, 499999500000)

In [3]: %timeit sum_range(0, 1_000_000)
165 ns ± 0.0381 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Notice the result is in nanoseconds. Rust is 1,800× faster than Cython–how is this possible?

The best solution: do less work

It turns out that when we moved more of the work into a high-level construct like Rust’s Ranges, we ended up with a much more clever solution. Summing an arbitrary array of integers requires looping over all the values, and so it’s O(N): the more values in the array, the longer it will take. And summing a consecutive range of integers can be done the same way.

Or we can take advantage of the fact it’s consecutive, in which case it can be done in a fixed amount of time, using for example (start + end)(N / 2). And either the Rust standard library or possibly the Rust compiler–I’m not sure which–are smart enough to use a (slightly different) fixed-time calculation. You can see the resulting assembly here, if you’re interested.

That means that unlike Cython’s implementation, the size of the range doesn’t matter much:

In [4]: %timeit sum_range(0, 100)
188 ns ± 0.616 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [5]: %timeit sum_range(0, 100_000_000)
189 ns ± 0.132 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

We can implement the faster algorithm in Python too:

In [15]: def py_sum_range(start, end):
    ...:     return (start + end - 1) * (end - start) // 2
    ...: 

In [16]: py_sum_range(0, 1_000_000)
Out[16]: 499999500000

In [17]: %timeit py_sum_range(0, 1_000_000)
252 ns ± 1.33 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Our Rust code is faster than this Python implementation, but only by a little. And our Python implementation is still vastly faster than the Cython or NumPy solutions we used above. By constraining the problem, we’ve come up with a fundamentally superior solution, and quite possibly it’s fast enough not to need a compiled extension at all.

Reducing extension overhead

What have we learned about optimizing Python code?

Start by trying to redefine the problem in a way that requires less work. You might be able to get a massive speedup using plain old Python.

If you do need to write an extension:

  1. Try to move as much of the computation as possible into the extension, to reduce Python prep overhead, serialization costs, and function call overhead.
  2. If you’re dealing with large numbers of objects, switch to an extension type that is managed by the extension, the way NumPy does with its array objects.
Read more...

from Planet Python
via read more

Python Software Foundation: Welcoming Microsoft as a Visionary Sponsor

Microsoft helps millions of Python developers achieve more by enabling Python support across products and services like Windows, Visual Studio Code, GitHub, and Microsoft Azure.

Microsoft has been a long-time supporter of the Python Software Foundation, starting in 2006. Today, Microsoft employs several core developers working part-time on CPython and the Python Steering Council, making significant contributions as PEP authors. We are also excited that Guido van Rossum has recently joined Microsoft as a Distinguished Engineer and is exploring performance improvements to CPython.

Aside from helping advance the Python tooling and ecosystem, Microsoft has shown strong involvement with the Python community and has been a long-time sponsor of PyCon US, including four years at the top tier Keystone level. Microsoft’s support for the community extends to sponsorships of regional and international Python events including in emerging regions.

This year, we are excited that Microsoft is increasing their contributions to the PSF even further as a Visionary Sponsor. Microsoft’s sponsorship funds will be used to support the PSF with a focus on working with the Packaging Working Group on improving PyPI and the packaging ecosystem. 

You can read more about how Microsoft is supporting and contributing to the Python ecosystem on here.





from Planet Python
via read more

Python Software Foundation: Announcing six scientific Python grants

The Scientific Python Working Group considers funding proposals that advance the scientific use of Python. In the past these have most often been workshops and conferences, but we also fund project improvements and outreach. Recently, we broadened our scope to include developer sprints.

Our most recent call for proposals has closed, and we are excited to announce six grants to:

  • Spyder for the development of online learning content for the Spyder IDE,
  • Nicolas McKibben to implement MRI reconstruction algorithms in PyGrappa,
  • The Carpentries to improve and maintain their Python curriculum,
  • Qu4nt for translating the scikit-learn documentation to Spanish,
  • Rosangela Mesquita for documenting common Continuous Integration practices for PyOceans, and
  • Blosc for the development of a Python wrapper for their new C-Blosc2 library.

We look forward to reporting back on these projects in a few months.


Written by the PSF Scientific Working Group



from Planet Python
via read more

Python for Beginners: Stack in Python

In python, there are inbuilt data structures like list,tuple, set and dictionary but we may need some additional  functionalities in python programs. For example, If we need a last in first out (LIFO) functionality in our program then none of the inbuilt data structures provide that functionality. We can use stack to implement last in first out functionality in python. In this article, we will study the concept behind stack data structure and implement it in python.

What is a stack?

A stack is a linear data structure in which we can only access or remove the element which was added last to it. In real life, we can take the example of a stack of plates where we can only take out the uppermost plate because it was added last. We can perform the following operations on a stack data structure in python.

  1. Insert an element(Push an element)
  2. Remove an element (Pop an element)
  3. Check if the stack is empty
  4. Check the topmost element in the stack
  5. Check the size of the stack

In the following subsections, we will implement stack and all its and will implement them in python. 

How to implement stack in python?

In this tutorial,we will define a class named Stack and will implement the Stack using python lists. In the Stack class, we will have a list for containing the data added to the Stack and a variable for storing the size of the Stack. All the operations like push, pop, checking the size of the stack,checking the topmost element of stack and checking the stack if it is empty will be executed in constant time and thus will have O(1) time complexity. The Stack class will be defined as follows.The stackList is initialized as an empty list and stackSize is initialized to 0.

class Stack:
    def __init__(self):
        self.stackList=[]
        self.stackSize=0

Push items to a stack in python

To insert an item into the stack i.e. to push an element into the stack, we will simply append the element in the list and then we will increment the stackSize variable by 1. To implement the operation, we define a method which takes the element as an argument and appends the element to the list in the stack. Then it increments the stackSize variable. Following is the implementation for the push() method.

def push(self,item):
        self.stackList.append(item)
        self.stackSize+=1

Pop items from a stack in python

To remove an item from the stack i.e. to pop an element from the stack, we have to remove the element from the stack which was added last to it. As we append the element to the list in stack while push operation, the last element in the list will be the most recent element and will be removed from the stack. So we simply delete the last element from the list. 

To implement the pop operation, we will implement the pop() method which first checks if the number of elements in the stack is greater than 0. If yes, then it will remove the last element from the list and decrease the stackSize by 1. If the number of elements in stack is 0, it will show an error message saying that the list is empty. For this task, we will use exception handling using python try except to raise an exception when the size of stack is 0. The pop() method can be implemented as follows.

def pop(self):
        try:
            if self.stackSize==0:
                raise Exception("Stack is Empty, returning None")
            temp=self.stackList.pop()
            self.stackSize-=1
            return temp
        except Exception as e:
            print(str(e))

Check size of the stack

To check the size of the stack, we simply have to check the value of the stackSize variable. For this operation, we will implement the size() method which returns the value of stackSize variable as follows.

def size(self):
        return self.stackSize

Check if stack is empty

To check if the stack has no elements i.e it is empty, we will have to check if the stackSize variable is 0 or not. For this operation, we will implement the isEmpty() method which returns True if the stackSize variable is 0 and returns false otherwise. Following is the implementation for isEmpty() method.

def isEmpty(self):
        if self.stackSize==0:
            return True
        else:
            return False

Check topmost element of an stack

The topmost element of the stack will be the element most recently added to it. To check the topmost element of the stack, we just have to return the last element in the list in the stack. For this operation, we will implement the top() method which first checks if the stack is empty i.e. stackSize is 0 or not, if yes then it will print a message saying that the stack is empty. Otherwise it will return the last element of the list. This can be implemented as follows.

def top(self):
        try:
            if self.stackSize==0:
                raise Exception("Stack is Empty, returning None")
            return self.stackList[-1]
        except Exception as e:
            print(str(e))

The complete working code for implementing stack in python is follows.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Fri Apr 23 19:38:55 2021

@author: aditya1117
"""

class Stack:
    def __init__(self):
        self.stackList=[]
        self.stackSize=0
    def push(self,item):
        self.stackList.append(item)
        self.stackSize+=1
    def pop(self):
        try:
            if self.stackSize==0:
                raise Exception("Stack is Empty, returning None")
            temp=self.stackList.pop()
            self.stackSize-=1
            return temp
        except Exception as e:
            print(str(e))
    def size(self):
        return self.stackSize
    def isEmpty(self):
        if self.stackSize==0:
            return True
        else:
            return False
    def top(self):
        try:
            if self.stackSize==0:
                raise Exception("Stack is Empty, returning None")
            return self.stackList[-1]
        except Exception as e:
            print(str(e))
            
#Execution
s=Stack()
#push element
s.push(1)
#push element
s.push(2)
#push element
s.push(3)
print("popped element is:")
print(s.pop())
#push an element
s.push(4)
print("topmost element is:")
print(s.top())

Output:

popped element is:
3
topmost element is:
4

Conclusion

In this article, we have studied and implemented stack data structure in python. We have seen the operation for the stack and you may find it very different from python dictionary, list, set e.t.c. Stacks are used extensively in real world applications like expression handling, backtracking, function calls and many other operations and you should have a working knowledge about it. Stay tuned for more informative articles.   

The post Stack in Python appeared first on PythonForBeginners.com.



from Planet Python
via read more

Stack Abuse: Creating a REST API in Python with Django

Introduction

Django is a powerful Python Web Framework used to build secure, scalable web applications rapidly with fewer efforts. It became popular because of its low barrier to entry, and strong community that uses and develops the framework.

In this guide, we are going to build a RESTful API using Django without any external libraries. We will cover the basics of Django and implement a JSON-based API to perform CRUD operations for a shopping cart application.

What is a REST API?

REST (Representational State Transfer) is a standard architecture for building and communicating with web services. It typically mandates resources on the web are represented in a text format (like JSON, HTML, or XML) and can be accessed or modified by a predetermined set of operations. Given that we typically build REST APIs to leverage with HTTP instead of other protocols, these operations correspond to HTTP methods like GET, POST, or PUT.

An API (Application Programming Interface), as the name suggests, is an interface that defines the interaction between different software components. Web APIs define what requests can be made to a component (for example, an endpoint to get a list of shopping cart items), how to make them (for example, a GET request), and their expected responses.

We combine these two concepts to build a REST(ful) API, an API that conforms to the constraints of the REST architectural style. Let's go ahead and make one, using Python and Django.

Setting up Django and Our Application

As mentioned earlier, Django is a Web Framework that promotes the rapid development of secure and scalable web services.

Note: We'll be using Django version 3.1, as that's the latest version as of writing.

Before installing Django, for good measure and in the name of isolating dependencies - let's make a virtual environment:

$ python3 -m venv env

On some code editors, you will find it already activated. If not, you can go to the scripts directory inside the environment and run activate.

On Windows:

$ env\scripts\activate

On Mac or Linux:

$ . env/bin/activate

Now, let's go ahead and install Django via pip:

$ pip install django

Once installed, we can create our project. While you can o it manually, it's much more convenient to start off with a skeleton project through Django itself.

The django-admin tool allows us to spin off a blank, skeleton project that we can start working on immediately. It comes bundled with Django itself, so no further installation is necessary.

Let's start the project by invoking the tool, as well as the startproject command, followed by the project's name:

$ django-admin startproject shopping_cart

This creates a simple skeleton project in the working directory. Each Django project can contain multiple apps - though, we'll be making one. Let's call on the manage.py file, created via the startproject command to spin up an application:

$ cd shopping_cart
$ python manage.py startapp api_app

Once created, our project structure will look something along the lines of:

> env
> shopping_cart
  > api_app
    > migrations
    __init__.py
    admin.py
    apps.py
    models.py
    tests.py
    views.py
  > shopping_cart
    __init__.py
    asgi.py
    settings.py
    urls.py
    wsgi.py
  manage.py

The top-level shopping_cart is the Django root directory and shares the name with the project name. The root directory is the bridge between the framework and the project itself and contains several setup classes such as manage.py which is used to spin up applications.

The api_app is the application we're spinning up, and there can be many here. Each has a few default scripts that we'll be modifying to accomodate for the CRUD functionality.

The low-level shopping-cart is the project directory, which contains settings-related files such as settings.py that will house all of our application's properties.

Django is an Model-View-Controller (MVC). It's a design pattern that separates an application into three components: the model which defines the data being stored and interacted with, the view which describes how the data is presented to the user, and the controller which acts as an intermediary between the model and the view. However, Django's interpretation of this pattern is slightly different from the standard interpretation. For example, in a standard MVC framework, the logic that processes HTTP requests to manage shopping cart items would live in the controller.

In Django, that logic resides in the file containing views. You can read more about their interpretation here. Understanding the core MVC concept as well as Django's interpretation makes the structure of this application easier to understand.

Each Django project comes pre-installed with a few Django applications (modules). These are used for authentication, authorization, sessions, etc. To let Django know that we'd also like to include our own application, api_app, we'll have to list it in the INSTALLED_APPS list.

Let's list it by going to the settings.py file and modifying the list to include our own app:

...
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'api_app',
]

Once listed, we're done with the project setup. Though, we're not done with the foundations on which our app will be built. Before developing the CRUD functionality, we'll need a model to work with as our basic data structure.

Note: Starting a Django project will, by default, also prepare an SQLite database for that project. You don't need to set it up at all - just defining models and calling the relevant CRUD functions will kick off a process under the hood that does everything for you.

Defining a Model

Let's start off with a simple, basic model - the CartItem, that represents an item listed on a fictional eCommerce website. To define models that Django can pick up - we modify the api_app/models.py file:

from django.db import models

class CartItem(models.Model):
    product_name = models.CharField(max_length=200)
    product_price = models.FloatField()
    product_quantity = models.PositiveIntegerField()

Here, we're leveraging the built-in db module, which has a models package within it. The Model class represents, well, a model. It has various fields such as CharField, IntegerField, etc. that are used to define the schema of the model in the database. These are mapped, under the hood, by Django's ORM when you want to save an instance of a model into the database.

Various fields exist, and they're designed to work well with relational databases, which we'll be doing here as well. For non-relational databases, though, it doesn't work very well due to an inherent difference in how data is stored.

If you'd like to work with a non-relational database, such as MongoDB - check out our Guide to Using The Django MongoDB Engine.

To make changes to model schemas, like what we just did, we have to call on Django Migrations. Migrations are pretty effortless to do, though, you'll have to run them each time you want to persist a change in the schema.

To do that, we'll call on the manage.py file, and pass in makemigrations and migrate arguments:

$ python manage.py makemigrations  # Pack model changes into a file
$ python manage.py migrate  # Apply those changes to the database

The migrate operation should result in something like this:

Operations to perform:
  Apply all migrations: admin, api_app, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying api_app.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying sessions.0001_initial... OK

If your migrations did not run successfully, please review the previous steps to ensure that you're set up correctly before continuing.

The Django Admin Site

When creating applications, Django automatically creates an admin-site, intended to be used by the developer to test things out, and give them access to forms generated for registered models. It's only meant to be used as a handy dashboard during development - not as the actual administration dashboard, which you'd create separately if you want to have one.

The admin module of django.contrib is the package that allows us to customize the admin-site.

To leverage the automatic form-creation and model management of Django, we'll have to register our model in the admin.site.

Let's go to api_app/admin.py and register our model:

from django.contrib import admin
from .models import CartItem

admin.site.register(CartItem)

Now, we'll want to create a user that's able to access this dashboard and use it. Let's create a superadmin account and confirm that all these changes were made successfully:

$ python manage.py createsuperuser

To create an account, you'll have to provide a username, email, and password. You can leave the email blank. Password is not reflected when you type. Just type and hit enter:

Username (leave blank to use 'xxx'): naazneen
Email address:
Password:
Password (again):
Superuser created successfully.

Finally, let's run our application to see if things are working as intended:

$ python manage.py runserver

The application is started on our localhost (127.0.0.1) on port 8000 by default. Let's naviate a browser to http://127.0.0.1:8000/admin:

Admin page after successful Django setup

Now that our application and database models are set up, let's focus on developing the REST API.

Creating a REST API in Django

The Django application is all set - the settings are defined, our application is prepared, the model is in place and we've created an administrator user to verify that the model is registered to the admin dashboard.

Now, let's implement the CRUD functionality for our model.

Creating Entities - The POST Request Handler

POST requests are used to send data to the server. Typically, they contain data in their body that's supposed to be stored. When filling out forms, uploading images or submitting a message - POST requests are sent with that data, which is then accordingly handled and saved.

Let's create a Django view to accept data from the client, populate a model instance with it, and add it to the database. Essentially, we'll be able to add an item to our shopping cart with our API. Views in Django can be written purely as functions or as methods of a class. We are going to use Class-Based Views.

To add a view, we'll modify the api_app_views.py file, and add a post() method that receives a POST request. It'll write the incoming request body into a dictionary and create a CartItem object, persisting it in the database:

from django.views import View
from django.http import JsonResponse
import json
from .models import CartItem

class ShoppingCart(View):
    def post(self, request):

        data = json.loads(request.body.decode("utf-8"))
        p_name = data.get('product_name')
        p_price = data.get('product_price')
        p_quantity = data.get('product_quantity')

        product_data = {
            'product_name': p_name,
            'product_price': p_price,
            'product_quantity': p_quantity,
        }

        cart_item = CartItem.objects.create(**product_data)

        data = {
            "message": f"New item added to Cart with id: {cart_item.id}"
        }
        return JsonResponse(data, status=201)

Using the json module, we've decoded and parsed the incoming request's body into an object we can work with, and then extracted that data into variables p_name, p_price and p_quantity.

Finally, we've created a product_data dictionary to hold our fields and their values, and persisted a CartItem to our database, via the create() method of the Model class, filling it with our product_data.

Note the use of the JsonResponse class at the end. We use this class to convert our Python dictionary to a valid JSON object that is sent over HTTP back to the client. We set the status code to 201 to signify resource creation on the server-end.

If we run our application and tried to hit this endpoint, Django would reject the request with a security error. By default, Django adds a layer of protection for Cross-site request forgery (CSRF) attacks. In practice, This token is stored in our browser's cookies and is sent with every request made to the server. As this API will be used without a browser or cookies, the requests will never have a CSRF token. Therefore, we have to tell Django that this POST method does not need a CSRF token.

We can achieve this by adding a decorator to the dispatch method of our class which will set the csrf_exempt to True:

...
from django.utils.decorators import method_decorator
from django.views.decorators.csrf import csrf_exempt

@method_decorator(csrf_exempt, name='dispatch')
class ShoppingCart(View):

    def post(self, request):
        data = json.loads(request.body.decode("utf-8"))
        ...

Now, we have the models that store our data, and a view that can create a new cart item, via an HTTP request. The only thing left to do is tell Django how to treat URLs and their respective handlers. For each accessed URL, we'll have an adequate view mapping that handles it.

It's considered good practice to write respective URLs in each app, and then include them into the project's urls.py file, rather than having them all on the top-level from the get-go.

Let's begin by amending the project's urls.py, in the shopping_cart directory:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('api_app.urls')),
]

Be sure to import the include library from django.urls, it's not imported by default.

The first argument is the string path, and the second is where we're getting the URLs from. With our path being '', or empty, it means that our API's URLs will be the root path of the web app.

We now need to add the endpoints for our API app's urls.py. In the api_app folder, we'll create a new file called urls.py:

from django.urls import path
from .views import ShoppingCart

urlpatterns = [
    path('cart-items/', ShoppingCart.as_view()),
]

Similar to the project's own urls.py, the first argument is the subpath where our views would be accessible, and the second argument is the views themselves.

Finally, we can run the application. Like before, we'll be utilizing the manage.py file, and pass in the runserver argument:

$ python manage.py runserver

Let's open a terminal and send a POST request to our endpoint. You can use any tool here - from more advanced tools like Postman, to simple CLI-based tools like curl:

$ curl -X POST -H "Content-Type: application/json" http://127.0.0.1:8000/car
t-items/ -d "{\"product_name\":\"name\",\"product_price\":\"41\",\"product_quantity\":\"1\"}"

If all works well, you'll be greeted with a message:

{
    "message": "New item added to Cart with id: 1"
}

If you visit http://127.0.0.1:8000/admin/api_app/cartitem/, the cart item we've added will also be listed.

Retrieving Entities - The GET Request Handler

Let's make a handler for GET requests, which are typically sent by clients when they'd like to receive some information. Since we have a CartItem saved to the database, it makes sense that someone would want to retrieve information about it.

Assuming the possibility of more than one item, we'll iterate over all the CartItem entries and add their attributes into a dictionary - which is easily converted into a JSON response for the client.

Let's modify the ShoppingCart view:

...
@method_decorator(csrf_exempt, name='dispatch')
class ShoppingCart(View):

    def post(self, request):
        ...

    def get(self, request):
        items_count = CartItem.objects.count()
        items = CartItem.objects.all()

        items_data = []
        for item in items:
            items_data.append({
                'product_name': item.product_name,
                'product_price': item.product_price,
                'product_quantity': item.product_quantity,
            })

        data = {
            'items': items_data,
            'count': items_count,
        }

        return JsonResponse(data)

The count() method counts the number of occurrences in the database, while the all() method retrieves them into a list of entities. Here, we extract their data and return them as a JSON response.

Let's send a GET request to our the endpoint:

$ curl -X GET http://127.0.0.1:8000/cart-items/

This results in a JSON response to the client:

{
    "items": [
        {
            "product_name": "name",
            "product_price": 41.0,
            "product_quantity": 1
        },
    ],
    "count": 1
}

Updating Entities - The PATCH Requesst Handler

We can persist and retrieve data via our API, though, it's equally as important to be able to update already persisted entities. The PATCH and PUT requests come into play here.

A PUT request entirely replaces the given resource. Whereas, a PATCH request modifies a part of the given resource.

In the case of PUT, if the given resource context does not exist, it will create one. To perform a PATCH request, the resource must already exist. For this application, we only want to update a resource if one exists already, so we'll be using a PATCH request.

The post() and get() methods are both located in the same ShoppingCart view class. This is because they affect more than one shopping cart item.

A PATCH request only affects one cart item. So we'll create a new class that will contain this view, as well as the future delete() method. Let's add a PATCH request handler to api_app/views.py:

...
@method_decorator(csrf_exempt, name='dispatch')
class ShoppingCartUpdate(View):

    def patch(self, request, item_id):
        data = json.loads(request.body.decode("utf-8"))
        item = CartItem.objects.get(id=item_id)
        item.product_quantity = data['product_quantity']
        item.save()

        data = {
            'message': f'Item {item_id} has been updated'
        }

        return JsonResponse(data)

We'll be retrieving the item with a given ID, and modifying it before saving it again.

Since we don't want the customer to be able to change the product price or name, we're only making the quantity of the item in the shopping cart variable in size. Then, we're calling the save() method to update the already existing entity in the database.

Now, we need to register an endpoint for this as well, just like we did for the cart-items/ endpoint:

api_app/urls.py:

from django.urls import path
from .views import ShoppingCart, ShoppingCartUpdate


urlpatterns = [
    path('cart-items/', ShoppingCart.as_view()),
    path('update-item/<int:item_id>', ShoppingCartUpdate.as_view()),
]

This time around, we're not only relying on the HTTP verb. Last time, sending a POST request to /cart-items resulted in the post() method being called, while sending a GET request resulted in the get() method being run.

Here, we're adding a URL-variable - /<int:item_id>. This is a dynamic component in the URL, which is mapped to the item_id variable from the view. Based on the provided value, the appropriate item is retrieved from the database.

Let's send a PATCH request to http:127.0.0.1:8000/update-item/1 with the appropriate data:

$ curl -X PATCH http://127.0.0.1:8000/update-item/1 -d "{\"product_quantity\":\"3\"}"

This results in a JSON response:

{
    "message": "Item 1 has been updated"
}

Let's also verify this via the Admin Panel at: http://127.0.0.1:8000/admin/api_app/cartitem/1/change/.

Deleting Entities - The DELETE Request Handler

Finally, the last piece of the CRUD functionality - removing entities.

To remove the item from the cart, we'll use the same ShoppingCartUpdate class as it only affects one item. We will add the delete() method to it which takes the ID of the item we'd like to remove.

Similar to how we save() the item when updating it with new values, we can use delete() to remove it. Let's add the delete() method to the api_app/views.py:

...
@method_decorator(csrf_exempt, name='dispatch')
class ShoppingCartUpdate(View):

    def patch(self, request, item_id):
        ...

    def delete(self, request, item_id):
        item = CartItem.objects.get(id=item_id)
        item.delete()

        data = {
            'message': f'Item {item_id} has been deleted'
        }

        return JsonResponse(data)

And now, let's send the DELETE request, and provide the ID of the item we'd like to remove:

curl -X "DELETE" http://127.0.0.1:8000/update-item/1

And we will get the following response:

{
    "message": "Item 1 has been deleted"
}

Visiting http://127.0.0.1:8000/admin/api_app/cartitem/ verifies that the item is no longer there.

Conclusion

In this short guide, we've gone over how to create a REST API in Python with Django. We've gone over some of the fundamentals of Django, started a new project and an app within it, defined the requisite models and implemented CRUD functionality.

The complete code for this application can be found here.



from Planet Python
via read more

TestDriven.io: Working with Static and Media Files in Django

This article looks at how to work with static and media files in a Django project, locally and in production. from Planet Python via read...