Python Pool: 4 Easy Ways to Find String in List in Python

We got to know about many topics in python. But have you ever tried to find the string in a list in Python? In this tutorial, we will be focusing on finding the string in a list. There are multiple ways through which we can find the string in a list. We will be discussing all of them in this article.

We will find the string in a python list with multiple ways like for loop, in operator, using count, any() function.

4 Ways With Examples to Find The String In List in Python

We will discuss how we can find the string in a list with examples explained in detail.

1. Using Simple For Loop

In this example, we will be making a list with elements of a string. We will then take an input string from the user, which we want to find that the string contains in a list. By using for loop, we will find the string in a list. Let us look in detail with the help of the example explained below:

#input list
lst = ['abc', 'def', 'ghi','python', 'pool']

#input string from user
str = input("Enter the input string : ")
c = 0
#applying loop
for i in lst:
    if i == str:
        c = 1
        break
if c == 1:
    print("String is present")
else:
    print("String is not present")

Output:

Enter the input string : python
String is present

Explanation:

• Firstly, we will make a list that will contain some element in the form of a string.
• Then, we will take the input from the user in the form of a string.
• We will take a variable c which is set to 0.
• After that, we will be applying for loop.
• Inside for loop, we will apply if condition in which we will check if the element of the list is equal to the string. If they are equal c value becomes 1, and the loop gets breaks.
• At last, we will check if c is equal to 1 or 0. if c is equal to 1, we will print string is present and if c == 0, we will print string is not present.
• Hence, you can see the output.

2. Using in operator to Find The String List in Python

It is the membership operator in python. It is used to test whether a value or variable is found in a sequence (string, list, tuple, set, and dictionary).

In this example, we will be making a list with elements of a string. We will apply in operator if we give the string to be searched and check if it is in the list. Let us look in detail with the help of the example explained below:

#input list
lst = ['abc', 'def', 'ghi','python', 'pool']

#applying if condition
if 'pool' in lst:
    print("string is present")
else:
    print("string is not present")

#checking false condition
if 'pools' in lst:
    print("string is present")
else:
    print("string is not present")

Output:

string is present
string is not present

Explanation:

• Firstly, we will make a list that will contain some element in the form of a string.
• Then, we will apply the ‘in’ operator inside the if condition and check if the string we want to check is present in the given list or not.
• If the string is present, the output is printed as the string is present, and if the string is not present, the output is printed as the string is not present.
• We have taken both examples. One of the lists is present, and one of the lists is not present in the program.
• Hence, you can see the output.

3. Using count() function

The count() function is used to count the occurrences of a string in a list. If we get the output as 0, then it says that the string does not contain the list, and if the output is 1, then the string is present in the list.

In this example, we will be making a list with elements of a string. We will then take an input string from the user, which we want to find that the string contains in a list. Then, we will apply the count function passing the input string and see the count value through the if condition, and print the output.

#input list
lst = ['abc', 'def', 'ghi','python', 'pool']

#input string from user
str = input("Enter the input string : ")

#count function
count = lst.count(str)

if count > 0:
    print("String is present")
else:
    print("String is not present")

Output:

Enter the input string : ghi
String is present

Explanation:

• Firstly, we will make a list that will contain some element in the form of a string.
• Then, we will take the input from the user in the form of a string.
• Then, we will apply the count() function, which will count the string’s occurrence in a list.
• If the count’s value is equal to 0, the output will be printed as the string is not present, and if the count is greater than 0, the output is printed as the string is present in the list.
• Hence, you can see the output.

4. Using any() function to Find String In List in Python

The any() function returns True if the string is present in a list. Otherwise, it returns False. If the list is empty, it will return False.

In this example, we will be making a list with elements of a string. We will apply if condition, and in that, we will apply any() function through which we can find the string in a list is present or not.

#input list
lst = ['abc', 'def', 'ghi','python', 'pool']
 
if any("pool" in word for word in lst):
    print("string is present")
else:
    print("string is not present")

Output:

string is present

Explanation:

• Firstly, we will make a list that will contain some element in the form of a string.
• Then, we will apply any() function inside the condition where we will give the string we have to find and apply for loop.
• If the function finds any same string as given, the output gets printed as the string is present, and if it does not find any such string, the output gets printed as a string is not present.
• Hence, you can finally see the output as the string is present as the pool is present in the list.

Conclusion

In this tutorial, we have learned about how to find the string in a list. We have seen all the multiple methods through which we can find the string present in the list. We have also discussed all the methods with the examples explained in detail. You can use any of the methods according to your need in the program.

However, if you have any doubts or questions, do let me know in the comment section below. I will try to help you as soon as possible.

The post 4 Easy Ways to Find String in List in Python appeared first on Python Pool.

from Planet Python
via read more

Real Python: Python Community Interview With Will McGugan

Today I’m joined by Will McGugan. Will is a freelance Python developer based in Scotland and the creator of the popular Python library Rich.

In this interview, we discuss the origins of Rich, what it’s like to maintain a popular Python package, building chess engines, and Will’s love of photography.

Ricky: Thanks for joining me for this interview, Will. I’d like to start in the same manner we do with all our guests: how did you get into programming, and when did you start using Python?

Will: Thanks for having me on!

I discovered programming as a kid in the ’80s. For younger readers, it was exactly like Stranger Things, minus the CGI monsters. We had a ZX Spectrum+ computer back then. The graphics were flickery, and the sound was tinny, but I was hooked. At some point I discovered that you could type things on the keyboard and—if you did it just right—the computer would do as you told it.

Fast-forward a few years, and I’m working in the video games industry. I’m building a game engine and looking for a scripting language to handle the game mechanics while C++ does the graphics and heavy lifting. That’s when I came across Python, version 2.1, I think.

I didn’t consider it a proper programming language at the time, probably because it was too easy to use, and programming should be hard, right? Even so, I found myself coming back to Python for scripts and tools, which gradually became more sophisticated.

Read the full article at https://realpython.com/interview-will-mcgugan/ »

---

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

from Planet Python
via read more

Python for Beginners: Extract a specific word from a string in Python

While handling text data, sometimes we have to search for occurrences of specific words in the text and extract specific words. In this tutorial, we will learn about different methods to extract a specific word from a string in python using inbuilt string methods and regular expressions.So, let’s dive into it.

Extract a specific word from a string using string slicing in python

If we know the exact position of the word to be extracted from the string, we can perform slicing operation on the string to extract the desired word from the string as shown below.

search_string= "I am a python programmer and I am writing this code for pythonforbeginners.com"
print("String from which word has to be searched is:")
print(search_string)
print("word to be extracted from string:")
word="writing"
print(word)
#calculate length of the word
lword=len(word)
#suppose we already know the starting index of word writing i.e. 34
extracted_string= search_string[34:34+lword]
print("Extracted word is:")
print(extracted_string)

Output:

String from which word has to be searched is:
I am a python programmer and I am writing this code for pythonforbeginners.com
word to be extracted from string:
writing
Extracted word is:
writing

Extract a specific word from a string using find() method.

If we want to extract a specific word from the string and we do not know the exact position of the word, we can first find the position of the word using find() method and then we can extract the word using string slicing.

The find() method when invoked on any string, takes the string to be searched as a parameter and gives as output the position of first occurrence of the input string which was to be searched. If the string to be searched is not present, the find() method returns -1.

After finding the position of the word to be extracted with find() method, we can simply extract it using slice operation as follows.

search_string= "I am a python programmer and I am writing this code for pythonforbeginners.com"
print("String from which word has to be searched is:")
print(search_string)
print("word to be extracted from string:")
word="writing"
print(word)
#calculate length of the word
lword=len(word)
start_index=search_string.find(word)
print("start index of the word in string is:")
print(start_index)
extracted_string= search_string[start_index:start_index+lword]
print("Extracted word is:")
print(extracted_string)

Output:

String from which word has to be searched is:
I am a python programmer and I am writing this code for pythonforbeginners.com
word to be extracted from string:
writing
start index of the word in string is:
34
Extracted word is:
writing

Using index() method.

If we don’t know the exact position of the word to be extracted,we can also use string index() method to find out the exact position of the word and then we can use slicing to extract the word.

The index() method when invoked on any string, takes the string to be searched as a parameter and gives as output the position of first occurrence of the input string which was to be searched. If the string to be searched is not present, index() throws an exception. For this reason we will have to use python try except to handle the exceptions while using index() method.

We can extract a specific word from a string in python using index() method and string slicing as follows.

search_string= "I am a python programmer and I am writing this code for pythonforbeginners.com"
print("String from which word has to be searched is:")
print(search_string)
print("word to be extracted from string:")
word="writing"
print(word)
#calculate length of the word
lword=len(word)
try:
    
    start_index=search_string.index(word)
    print("start index of the word in string is:")
    print(start_index)
    extracted_string= search_string[start_index:start_index+lword]
    print("Extracted word is:")
    print(extracted_string)
except:
    print("word not found")

Output:

String from which word has to be searched is:
I am a python programmer and I am writing this code for pythonforbeginners.com
word to be extracted from string:
writing
start index of the word in string is:
34
Extracted word is:
writing

Using regular expressions to extract any specific word

We can use regular expressions in python to extract specific words from a string. We can use search() method from re module to find the first occurrence of the word and then we can obtain the word using slicing.

re.search() method  will take the word to be extracted in regular expression form and the string as input and and returns a re.MatchObject which contains the starting and ending index of the word.If the given word is not found, re.search() will return None. After getting the indices of the word to be extracted, we can extract it using string slicing as shown below.

import re
search_string= "I am a python programmer and I am writing this code for pythonforbeginners.com"
print("String from which word has to be searched is:")
print(search_string)
print("word to be extracted from string:")
word=r"writing"
print(word)
#calculate length of the word
lword=len(word)
start_index=re.search(word,search_string).start()
print("start index of the word in string is:")
print(start_index)
extracted_string= search_string[start_index:start_index+lword]
print("Extracted word is:")
print(extracted_string)

Output:

String from which word has to be searched is:
I am a python programmer and I am writing this code for pythonforbeginners.com
word to be extracted from string:
writing
start index of the word in string is:
34
Extracted word is:
writing

Conclusion

In this article, we have seen how to find any specific word in a string using different string methods and regular expressions and then print the word using string slicing in python. We can also use python string split operation when we just have to search if the word is present or not, given that words are space separated. Stay tuned for more informative articles.

The post Extract a specific word from a string in Python appeared first on PythonForBeginners.com.

from Planet Python
via read more

Python Engineering at Microsoft: Jupyter in Visual Studio Code – April 2021 Release

We are pleased to announce that the April 2021 release of the Jupyter Extension for Visual Studio Code is now available. If working with Python, we recommend installing the Python extension directly from the extension gallery in Visual Studio Code. If you already have the Python or Jupyter extensions installed, you can also get the latest update by restarting Visual Studio Code. You can learn more about Python support in Visual Studio Code in the documentation.  

This release of the Jupyter Extension includes an overhaul and new functionality of the Data Viewer, as well as some UI enhancements and multiselect support for Native Notebooks.  

If you’re interested, you can check the full list of improvements in our changelog. 

Data Viewer Enhancements 

We’ve added many improvements to our Data Viewer in the latest release. Firstly, we have added the ability to refresh the Data Viewer. If you’ve made some changes or transformations to your data, rather than having to close and reopen the Data Viewer to view the changes, you can now click the refresh button in the top left to grab the most up-to-date data. 

Secondly, the Data Viewer now supports viewing both PyTorch and TensorFlow Tensor data types! 

Finally, we’ve given the entire Data Viewer UI a visual update to make it more aesthetically pleasing. You can now find the filter box at the heading of each column. You can also continue to click on any column heading to sort its data ascending/descending. Lastly, you can now click into individual cells in the Data Viewer to copy out their contents. 

Data Slicing 

The Data Viewer now has support for slicing data. This will allow you to view any 2D slice of your higher dimensional data. If you have 3-dimensional or greater data (numpy ndarray, PyTorch Tensor or TensorFlow EagerTensor types), you will now be able to view that multi-dimensional data in the Data Viewer and a new data slicing panel will open in the Data Viewer by default. In this panel, you will be able to either use an input box to programmatically specify your slice using Python slice syntax or use the interactive Axis and Index dropdowns to slice as well. Both will be in sync.  

VS Code Insiders – Native Notebooks

To try out Native Notebooks you need to download VS Code Insiders and make sure you have the Jupyter extension! Reminder that the Python extension comes with the Jupyter extension and when working with Python we recommend having both installed for the best Python experience.  

Multiple Cell Selection is Here!  

Jupyter Notebooks now have support for selecting multiple consecutive cells using the mouse (Shift + Click) or the keyboard shortcut (Shift + Arrow) when in cell selection mode.  

You can also select individual cells using the mouse (Ctrl + Click).  

Once the cells are selected, you can copy/cut/paste/duplicate and even move selected cells by dragging and dropping them to your preferred location.  

Quick Collapse 

You’ll no longer need to open the overflow menu in the cell toolbar to collapse cell inputs or outputs! Simply hover over to the left of each cell and double-click in order to collapse the elements of your cell.  To reveal collapsed content, just click the collapsed icon as shown below. 

Customizable Diffing Experience 

Users can now customize the type of information that is surfaced when diffing notebooks and have the option to hide metadata and output differences!  To customize your filters, simply open the overflow menu located in the top right of the rich diff text editor and select the sections on the notebook you would like included!  

Other Changes and Enhancements 

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Notebooks in Visual Studio Code. Some notable changes include: 

• Variable view will stay as long as you have a notebook open (not necessarily active). (#4562) 
• Ensure extensions depending on Jupyter do not fail to load if Jupyter extension fails to load. (#5145)  
• Fix data viewer display of non-numeric index columns in DataFrames. (#5253) 
• When reopening a newly created Notebook with a Julia kernel, the cells should be detected as Julia. (#5148) 
• Support switching kernels in Native Notebooks when connecting to Jupyter. (#1215) 
• Refactor how Kernels are searched and selected. (#4995) 
• Be able to provide string argument to jupyter.execSelectionInteractive for extensibility. (Thanks Andrew Craig) (#1689) 

Be sure to have the latest version of the Python and Jupyter Extensions for Visual Studio Code now to try out the above improvements. If you run into any problems or have suggestions, please file an issue on the Jupyter VS Code GitHub repository. 

 

The post Jupyter in Visual Studio Code – April 2021 Release appeared first on Python.

from Planet Python
via read more

Python Pool: Working With Carriage Return (\r) in Python

Introduction

Sometimes, we occur in a situation where we want to go back to the starting point of the same line. In this article will help you understand the concept of carriage return in python or \r in python.

What is a carriage return (\r) in Python?

It helps us move the cursor at the beginning of the line without moving the cursor to the new line.

Ways to use carriage return

We will showing all the types by which we can use the ‘\r’ in python.

1. Using only carriage return in Python

In this example, we will be using only the carriage return in the program in between the string.

string = 'My website is Latracal \rSolution'

print(string)

Output:

Solutionte is Latracal

Explanation:

• Firstly, we have taken an input string as a string.
• we have applied \r in between the string.
• \r has shifted the cursor to the start, and ‘solution’ contains 8 letters. From the beginning, 8 letters will be erased, and in place of that solution gets printed.
• You can see the output for better understanding.

2. Using carriage return in Python with a newline character

In this example, we will be using ‘\r’ with the new line character(\n) in the string program.

string = 'My website is Latracal \r\nSolution'
print(string)

string = 'My website is Latracal \n\rSolution'
print(string)
string = 'My web\nsite is Latracal \rSolution'

print(string)

Output:

My website is Latracal 
Solution
My website is Latracal 
Solution
My web
SolutionLatracal

Explanation:

• Firstly, we have taken an input string as a string.
• Then, we have applied \n and \r in multiple places of the string.
• \n is for newline.
• In the first two strings, we have put \n before and after \r. so the output gets printed in a newline.
• In the last string, \n is first after the ‘site is, ‘which contains 8 letters as solution, so they get replaced.
• Hence, you can see the output.

3. Using Carriage return in python with tab space

In this example, we will be using the carriage or \r with the combination of tab space or \t in the program between the string.

str = ('\tLatracal \rsolution')
print(str)

Output:

solutionLatracal 

Explanation:

• Firstly, we have taken an input as str.
• Then, we have applied tab space at the beginning of the string, which will give 8 spaces at the beginning.
• Then, we have applied \r. After \r, there are 8 letters of solution.
• In the output, the letter of solution will fill the tab spaces as they are equal.
• You can see the output for better understanding.

4. Using carriage return in python, tab space and newline character

In this example, we will be mixing all the characters such as carriage return(\r), tab space(\t), and newline character(\n) in the given string and see the output so that we can understand the use to \r more clearly.

str = ('\tlatracal\rsolution\n\tis a\rwebsite\n')
print(str)

Output:

solutionlatracal
website is a

Explanation:

• Firstly, we have taken an input string as str.
• Then, we have applied all the characters like \t for tab space, \n for the new-line, and \r for carriage return.
• Hence you can see the output.

How \r and \n is handled on Linux and windows

As we all know, we are using \r for carriage return and \n for newline in windows. But, for different operating systems, there are different conventions. The difference is simple, i.e., OS designers had to choose how we should represent a new line in text in computer files. For some reason, in Unix/Linux world, a single LF(Line feed) was chosen as the new line marker. MS-DOS chose CR+LF, and windows inherited \n as a new line. Thus, we got to know that different platforms have different conventions.

In practice, this is becoming a shorter of a problem. The newline marker is basically relevant for the programs that process “plain text,” and there are not that many. This mostly only affects program source code, configuration files, and some simple text files with documentation. As in today’s world, most programs handling these kinds of files (editors, compilers, etc.) can handle both newline conventions, so it does not matter which one you choose.

Must Read

• Python Print Without Newline [How to, Examples, Tutorial]
• Multiple Ways To Print Blank Line in Python
• Python is Not Recognized as an Internal or External Command
• 5 Ways to Remove the Last Character From String in Python

Conclusion

In this tutorial, we have learned about the concept of carriage return (‘\r’) with its definition. Also understood all the ways through which we can use ‘\r’ by different- different ways in detail with the help of an example. All the examples are explained in detail.

However, if you have any doubts or questions, do let me know in the comment section below. I will try to help you as soon as possible.

The post Working With Carriage Return (\r) in Python appeared first on Python Pool.

from Planet Python
via read more

Python Pool: 3 Ways to Convert String to Variable Name in Python

Introduction

We got to know about many topics in python. But, have you ever use the input string as the variable name in your program. In this tutorial, we will be focusing on this topic only to convert a Python string to a variable name. In other words, we will be creating dynamic variable names and assigning a value to them. There are multiple ways to convert a string to a variable name. We will discuss all of them in this article.

We will convert a string into a variable name with multiple ways like exec() function, locals() function and globals() function.

Multiple Ways With Examples to Convert a Python string to a Variable Name

We will then discuss how we can convert a string to a variable name with examples explained in detail.

1. Using exec() Method in Python to Convert a Python string to a Variable Name

The Exec() methods helps us to execute the python program dynamically.

In this example, we will have a variable named by us and an input string. We will then be applying the exec() method with some modifiers and trying to convert a string into a variable name. Let us look in detail with the help of the example explained below:

#input String
str = "Pythonpool"

#apply exec() method
exec("%s = %d" % (str,5000))

#print string
print("output : ",Pythonpool) 

Output:

output :  5000

Explanation:

• Firstly, we have taken an input string in str as Pythonpool.
• Then, we have applied the exec() function.
• Inside the exec() function, we have taken %s and %d, which are used as a placeholder for string value and decimal value, respectively. It means that we have assigned an integer value to a string with the help of the assignment operator =. Both %s and %d are enclosed inside the quotations ” “.
• Then, we have parenthesis inside which we have passed 2 things, i.e., string and integer.
• At last, we have printed the input string and see if the string contains a variable name or not.
• If the program runs successfully and prints the output, we have converted a string into a variable name.
• Hence, you can see the output.

2. Using locals() function to Convert a Python string to a Variable Name

The locals() function in python is used to return the dictionary of the current local symbol table. The local symbol table is accessed with the help of the locals() function. This function works the same as the globals() function. The only difference between them is the locals() function access local symbol table, and globals() access the global symbol table, and both return the dictionary.

In this example, we will be taking the input as a string. Then, we will apply the locals as a dictionary by putting some value to it. At last, we will print the input string and convert a string into a variable name.

#taking input as a string
str = "pythonpool"

locals()[str] = 5000
print(pythonpool)

Output:

5000

Explanation:

• Firstly, we have taken an input in str as pythonpool.
• Then, we have modified the value of the given string through the local’s dictionary through the locals() function.
• At last, we have printed the input string and see if the string contains a variable name or not.
• If the program runs successfully and prints the output, we have converted a string into a variable name.
• Hence, you can see the output.

3. Using globals() function to Convert a Python string to a Variable Name

The globals() function is used to return the dictionary of the current symbol table. A global symbol table is used to store all the information related to the program’s global scope, accessed using the globals() function.

In this example, we will be taking the input as a string. Then, we will apply the globals as a dictionary by putting some value to it. At last, we will print the input string and convert a string into a variable name.

#taking input as a string
str = "pythonpool"

globals()[str] = 5000
print(pythonpool)

Output:

5000

Explanation:

• Firstly, we have taken an input in str as pythonpool.
• Then, we have modified the value of the given string through the globals dictionary through the globals() function.
• At last, we have printed the input string and see if the string contains a variable name or not.
• If the program runs successfully and prints the output, we have converted a string into a variable name.
• Hence, you can see the output.

Pros and cons of creating Global variables in python

Pros

• When we create dynamic variables, they add another level of indirection.
• It avoids more code duplication.

Cons

• We cannot create dynamic variables for functions.
• It isn’t easy to keep all the tracks of lexical references:
  • if we create arbitrary variable names, conflicts can occur.
  • It is difficult to predict the behavior and find inputs and outputs in the code.

Must Read

• 【How to】 Check if the Variable Exists in Python
• 5 Best Ways to Find Python String Length

Conclusion

In this tutorial, we have learned about the concept of converting a string to a variable name. We have discussed all the ways through which we can convert a string into a variable name. All the ways are explained in detail with the help of examples. You can use any of the methods which you find suitable for you and your program.

However, if you have any doubts or questions, do let me know in the comment section below. I will try to help you as soon as possible.

The post 3 Ways to Convert String to Variable Name in Python appeared first on Python Pool.

from Planet Python
via read more

Python Pool: Numpy Permutation() | How to use np.random.permutation()

Introduction

In this tutorial, We will learn how to find the permutation of the numpy array. We can find the permutation of the numpy array using np.random.permutation() function. we will learn the calculation of this function in deep. We will give you a thorough explanation of every part of the code with examples.

What is Numpy Permutation?

Numpy is a toolkit that helps us in working with numeric data. It contains a set of tools for creating a data structure called a Numpy array. It is basically a row and column grid of numbers.

Numpy. random. permutation() function gives us the random samples of a sequence of permutation and returns sequence by using this method. If x is a multi-dimensional array, it is only shuffled along with its first index.

Syntax

numpy.random.permutation(x)

Parameters of np.random.permutation

x: It is an array. If the input in x is an integer value, then it randomly permutes np. arange(x). If the input in x is an array, then it makes a copy and shuffles the elements randomly.

Return Value of np.random.permutation

The np.random.permutation() function returns the permuted sequence or an array range.

Example of np.random.permutation

Let us understand the permutation function of the numpy module in detail with the help of different examples:

1. Taking x parameter as a integer

In this example, we will be importing a numpy library. Then, we will be taking input x(parameter) as an integer. After that, we will apply the permutation function and see the output.

#importing numpy library
import numpy as np

result = np.random.permutation(8)
print("Output : ",result)

Output:

Output :  [1 4 0 7 3 5 6 2]

Explanation:

• Firstly, we will import a numpy module with an alias name as np.
• Then, we will take the variable result in which we have applied the permutation() function.
• At last, we have printed the output stored in the result variable.

2. Taking x parameter as a array on np.random.permutation

In this example, we will be importing a numpy library. Then, we will be taking input x(parameter) as an array. After that, we will apply the permutation function and see the output.

#importing numpy library
import numpy as np

arr = [1,2,3,4,5,6]
result = np.random.permutation(arr)
print("Output : ",result)

Output:

Output :  [1 5 3 2 6 4]

Explanation:

• Firstly, we will import a numpy module with an alias name as np.
• Then, we will take an input array in the variable named arr.
• Then, we will take the variable result in which we have applied the permutation() function passing the input array in it.
• At last, we have printed the output stored in the result variable.

3. Using arange and reshape function on np.random.permutation

In this example, we will be importing the numpy library. Then, we will be using the np. arange and reshape function from the numpy library. After that, we will apply the permutation function and see the output.

#importing numpy library
import numpy as np

arr = np.arange(16).reshape((4, 4))
result = np.random.permutation(arr)
print("output : ",result)

Output:

output :  [[ 4  5  6  7]
 [ 0  1  2  3]
 [ 8  9 10 11]
 [12 13 14 15]]

Explanation:

• Firstly, we will import a numpy module with an alias name as np.
• Then, we will take an input variable ‘arr’ in which we have applied numpy arrange() function with reshape().
• Then, we will take the variable result in which we have applied the permutation() function passing the input array in it.
• At last, we have printed the output stored in the result variable.

4. Implementing graph of permutation on np.random.permutation

In this example, we will be importing the numpy module and matplotlib module. Then, we will apply the permutation function with the value as an integer. After that, we will apply the graph function and print the output.

# import numpy
import numpy as np
import matplotlib.pyplot as plt
  
# Using permutation() method
arr = np.random.permutation(100)
  
c, bin, ig = plt.hist(arr, 15, density = True)
plt.show()

Output:

Explanation:

• Firstly, we will be importing the numpy module with an alias name as np and matplotlib. pyplot as plt.
• Then, we will use the permutation function and store the output in the arr variable.
• After that, we will be applying the plt library with hist function to make the histogram with their parameters.
• At last, we have printed the output.
• Hence, you can see the output.

Difference between permutation and shuffle function

Np.random.permutation()

Numpy.random.permutation() function gives us the random samples of a sequence of permutation and returns sequence by using this method.

Numpy.random.shuffle()

Numpy.random.shuffle() function gives us the modified sequence in-place by shuffling its contents.

Example

In this example, we will be importing the numpy library. Then, we will take an input array as input. After that, we will be applying both the function permutation and shuffle. Then, see the necessary changes by seeding the output.

#importing numpy library
import numpy as np

#taking input for permutaion and shuffle function
arr = [2,4,6,8,10]

#applying permutation function
result = np.random.permutation(arr)

#applyimg shuffle function
results = np.random.shuffle(arr)

#printing both the result
print("Permutation output : ",result)
print("shuffle output : ",results) 

Output:

Permutation output :  [ 4  8  6  2 10]
shuffle output :  [4, 6, 2, 8, 10]

Explanation:

• Firstly, we have imported the numpy library with an alias name as np.
• Then, we have taken the input array in the variable as arr.
• After that, we have applied the numpy permutation() with passing the input array parameter.
• Then, we have applied the shuffle() function and stored the output in the results variable.
• At last, we have printed both the output and see the changes in the output.
• Hence, you can see the output.

Conclusion

In this tutorial, we have learned about the concept of numpy random permutation. We have discussed all the ways through which we can use the permutation concept and also discussed the example in detail. We have also discussed the difference between the numpy permutation() and numpy shuffle() with the example explained in detail. You can use the permutation function as required by you in the program.

The post Numpy Permutation() | How to use np.random.permutation() appeared first on Python Pool.

from Planet Python
via read more

neptune.new

First of all – we’re sorry! We’ve been awfully quiet in the last months in terms of product updates. You’ve seen here and there an update in the web interface or a bug fix in the Python client library, but nothing larger than that.  What was going on? Over time we got a tremendous amount […]

The post neptune.new appeared first on neptune.ai.

from Planet SciPy
read more

Michał Bultrowicz: Some shell functions for working with Python virtualenvs

Finally doing some scripting today. I noticed that there are some python-virtualenv-related commands I run often (in my tmux-based “IDE”), so I automated them away to shave off some keystrokes.

from Planet Python
via read more

Python Community Interview With Will McGugan

Today I’m joined by Will McGugan. Will is a freelance Python developer based in Scotland and the creator of the popular Python library Rich.

In this interview, we discuss the origins of Rich, what it’s like to maintain a popular Python package, building chess engines, and Will’s love of photography.

Ricky: Thanks for joining me for this interview, Will. I’d like to start in the same manner we do with all our guests: how did you get into programming, and when did you start using Python?

Will: Thanks for having me on!

I discovered programming as a kid in the ’80s. For younger readers, it was exactly like Stranger Things, minus the CGI monsters. We had a ZX Spectrum+ computer back then. The graphics were flickery, and the sound was tinny, but I was hooked. At some point I discovered that you could type things on the keyboard and—if you did it just right—the computer would do as you told it.

Fast-forward a few years, and I’m working in the video games industry. I’m building a game engine and looking for a scripting language to handle the game mechanics while C++ does the graphics and heavy lifting. That’s when I came across Python, version 2.1, I think.

I didn’t consider it a proper programming language at the time, probably because it was too easy to use, and programming should be hard, right? Even so, I found myself coming back to Python for scripts and tools, which gradually became more sophisticated.

Read the full article at https://realpython.com/interview-will-mcgugan/ »

---

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

from Real Python
read more

Stack Abuse: Matplotlib: Plot Multiple Line Plots On Same and Different Scales

Introduction

Matplotlib is one of the most widely used data visualization libraries in Python. From simple to complex visualizations, it's the go-to library for most.

In this tutorial, we'll take a look at how to plot multiple line plots in Matplotlib - on the same Axes or Figure.

If you'd like to read more about plotting line plots in general, as well as customizing them, make sure to read our guide on Plotting Lines Plots with Matplotlib.

Plot Multiple Line Plots in Matplotlib

Depending on the style you're using, OOP or MATLAB-style, you'll either use the plt instance, or the ax instance to plot, with the same approach.

To plot multiple line plots in Matplotlib, you simply repeatedly call the plot() function, which will apply the changes to the same Figure object:

import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5, 6]
y = [2, 4, 6, 5, 6, 8]
y2 = [5, 3, 7, 8, 9, 6]

fig, ax = plt.subplots()

ax.plot(x, y)
ax.plot(x, y2)
plt.show()

Without setting any customization flags, the default colormap will apply, drawing both line plots on the same Figure object, and adjusting the color to differentiate between them:

Now, let's generate some random sequences using Numpy, and customize the line plots a tiny bit by setting a specific color for each, and labeling them:

import matplotlib.pyplot as plt
import numpy as np

line_1 = np.random.randint(low = 0, high = 50, size = 50)
line_2 = np.random.randint(low = -15, high = 100, size = 50)

fig, ax = plt.subplots()

ax.plot(line_1, color = 'green', label = 'Line 1')
ax.plot(line_2, color = 'red', label = 'Line 2')
ax.legend(loc = 'upper left')
plt.show()

We don't have to supply the X-axis values to a line plot, in which case, the values from 0..n will be applied, where n is the last element in the data you're plotting. In our case, we've got two sequences of data - line_1 and line_2, which will both be plotted on the same X-axis.

While plotting, we've assigned colors to them, using the color argument, and labels for the legend, using the label argument. This results in:

Plot Multiple Line Plots with Different Scales

Sometimes, you might have two datasets, fit for line plots, but their values are significantly different, making it hard to compare both lines. For example, if line_1 had an exponentially increasing sequence of numbers, while line_2 had a linearly increasing sequence - surely and quickly enough, line_1 would have values so much larger than line_2, that the latter fades out of view.

Let's use Numpy to make an exponentially increasing sequence of numbers, and plot it next to another line on the same Axes, linearly:

import matplotlib.pyplot as plt
import numpy as np

linear_sequence = np.linspace(0, 10, 10)
# [0, 1.1, 2.2, 3.3, 4.4, 5.5, 6.6, 7.7, 8.8, 10]
exponential_sequence = np.exp(linear_sequence)
# [1.00e+00, 3.03e+00, 9.22e+00, 2.80e+01, 8.51e+01, 2.58e+02, 7.85e+02, 2.38e+03, 7.25e+03, 2.20e+04]

fig, ax = plt.subplots()

ax.plot(linear_sequence)
ax.plot(exponential_sequence)
plt.show()

Running this code results in:

The exponential growth in the exponential_sequence goes out of proportion very fast, and it looks like there's absolutely no difference in the linear_sequence, since it's so minuscule relative to the exponential trend of the other sequence.

Now, let's plot the exponential_sequence on a logarithmic scale, which will produce a visually straight line, since the Y-scale will exponentially increase. If we plot it on a logarithmic scale, and the linear_sequence just increases by the same constant, we'll have two overlapping lines and we will only be able to see the one plotted after the first.

Let's change up the linear_sequence a bit to make it observable once we plot both:

import matplotlib.pyplot as plt
import numpy as np

# Sequences
linear_sequence = [1, 2, 3, 4, 5, 6, 7, 10, 15, 20]
exponential_sequence = np.exp(np.linspace(0, 10, 10))

fig, ax = plt.subplots()

# Plot linear sequence, and set tick labels to the same color
ax.plot(linear_sequence, color='red')
ax.tick_params(axis='y', labelcolor='red')

# Generate a new Axes instance, on the twin-X axes (same position)
ax2 = ax.twinx()

# Plot exponential sequence, set scale to logarithmic and change tick color
ax2.plot(exponential_sequence, color='green')
ax2.set_yscale('log')
ax2.tick_params(axis='y', labelcolor='green')

plt.show()

This time around, we'll have to use the OOP interface, since we're creating a new Axes instance. One Axes has one scale, so we create a new one, in the same position as the first one, and set its scale to a logarithmic one, and plot the exponential sequence.

This results in:

We've also changed the tick label colors to match the color of the line plots themselves, otherwise, it'd be hard to distinguish which line is on which scale.

Plot Multiple Line Plots with Multiple Y-Axis

Finally, we can apply the same scale (linear, logarithmic, etc), but have different values on the Y-axis of each line plot. This is achieved through having multiple Y-axis, on different Axes objects, in the same position.

For example, the linear_sequence won't go above 20 on the Y-axis, while the exponential_sequence will go up to 20000. We can plot them both linearly, simply by plotting them on different Axes objects, in the same position, each of which set the Y-axis ticks automatically to accommodate for the data we're feeding in:

import matplotlib.pyplot as plt
import numpy as np

# Sequences
linear_sequence = [1, 2, 3, 4, 5, 6, 7, 10, 15, 20]
exponential_sequence = np.exp(np.linspace(0, 10, 10))

fig, ax = plt.subplots()

# Plot linear sequence, and set tick labels to the same color
ax.plot(linear_sequence, color='red')
ax.tick_params(axis='y', labelcolor='red')

# Generate a new Axes instance, on the twin-X axes (same position)
ax2 = ax.twinx()

# Plot exponential sequence, set scale to logarithmic and change tick color
ax2.plot(exponential_sequence, color='green')
ax2.tick_params(axis='y', labelcolor='green')

plt.show()

We've again, created another Axes in the same position as the first one, so we can plot on the same place in the Figure but different Axes objects, which allows us to set values for each Y-axis individually.

Without setting the Y-scale to logarithmic this time, both will be plotted linearly:

Conclusion

In this tutorial, we've gone over how to plot multiple Line Plots on the same Figure or Axes in Matplotlib and Python. We've covered how to plot on the same Axes with the same scale and Y-axis, as well as how to plot on the same Figure with different and identical Y-axis scales.

If you're interested in Data Visualization and don't know where to start, make sure to check out our book on Data Visualization in Python.

Data Visualization in Python, a book for beginner to intermediate Python developers, will guide you through simple data manipulation with Pandas, cover core plotting libraries like Matplotlib and Seaborn, and show you how to take advantage of declarative and experimental libraries like Altair.

Data Visualization in Python

Want to actually understand your data?

✅ 30-day no-question money-back guarantee

✅ Beginner to Advanced

✅ Updated regularly (latest update in January 2021)

✅ Contains FREE Supplemental Guide

from Planet Python
via read more

MLOps at GreenSteam: Shipping Machine Learning [Case Study]

GreenSteam is a company that provides software solutions for the marine industry that help reduce fuel usage. Excess fuel usage is both costly and bad for the environment, and vessel operators are obliged to get more green by the International Marine Organization and reduce the CO2 emissions by 50 percent by 2050. Even though we […]

The post MLOps at GreenSteam: Shipping Machine Learning [Case Study] appeared first on neptune.ai.

from Planet SciPy
read more

Test and Code: 149: I don't test my code, "crappy Python" is all I write - Corey Quinn

Corey Quinn is the Chief Cloud Economist at The Duckbill Group. He's also a podcaster and writes a newsletter. And he also automates things with Python. But he doesn't write tests. Let's find out why.

Reason for the interview. Rough summary of a twitter conversation:

• Corey: What podcasts should I try to get an invite onto?
• ToonArmyCaptain: Python Bytes, Test & Code, Talk Python
• Corey: But... I don't test my code, "crappy Python" is all I write, and I'd feel like a giant imposter. So yeah, I'd be game. link
• So here we are.

This diagram is referenced in the show, the Last Week In AWS Newsletter Production Pipeline.

Special Guest: Corey Quinn.

Sponsored By:

• PyCharm Professional: Try PyCharm Pro for 4 months and learn how PyCharm will save you time. Promo Code: TESTANDCODE21

Support Test & Code : Python Testing

Links:

• Last Week in AWS

<p>Corey Quinn is the Chief Cloud Economist at The Duckbill Group. He&#39;s also a podcaster and writes a newsletter. And he also automates things with Python. But he doesn&#39;t write tests. Let&#39;s find out why.</p> <p>Reason for the interview. Rough summary of a twitter conversation:</p> <ul> <li>Corey: What podcasts should I try to get an invite onto?</li> <li>ToonArmyCaptain: Python Bytes, Test &amp; Code, Talk Python</li> <li>Corey: But... I don&#39;t test my code, &quot;crappy Python&quot; is all I write, and I&#39;d feel like a giant imposter. So yeah, I&#39;d be game. <a href="https://twitter.com/QuinnyPig/status/1354093298890141697?s=20" rel="nofollow">link</a></li> <li>So here we are. </li> </ul> <p>This diagram is referenced in the show, the <a href="https://ift.tt/3m5HIgt" rel="nofollow">Last Week In AWS Newsletter Production Pipeline</a>.</p><p>Special Guest: Corey Quinn.</p><p>Sponsored By:</p><ul><li><a href="https://ift.tt/2JDHRTz" rel="nofollow">PyCharm Professional</a>: <a href="https://ift.tt/2JDHRTz" rel="nofollow">Try PyCharm Pro for 4 months and learn how PyCharm will save you time.</a> Promo Code: TESTANDCODE21</li></ul><p><a href="https://ift.tt/2tzXV5e" rel="payment">Support Test & Code : Python Testing</a></p><p>Links:</p><ul><li><a href="https://ift.tt/2ItmbeU" title="Last Week in AWS" rel="nofollow">Last Week in AWS</a></li></ul>

from Planet Python
via read more

PyCoder’s Weekly: Issue #466 (March 30, 2021)

#466 – MARCH 30, 2021
View in Browser »

---

Many Models Workflows in Python

Learn how to organize models into dataframes for exploratory data analysis, and why you might want to consider doing this.
ALEX HAYES

Building a Full-Text Search Engine in 150 Lines of Python Code

Go from data preparation to search engine in just a few lines of Python.
BART DE GOEDE

Get Granular Visibility into your Python Apps with Detailed Metrics and End-To-End Traces in Datadog

Datadog’s distributed tracing and APM generates flame graphs from real requests, enabling you to visualize app performance in real-time. Trace requests across service boundaries and optimize bottlenecks by drilling into individual traces end-to-end. Try Datadog APM free →
DATADOG sponsor

OrderedDict vs dict in Python: The Right Tool for the Job

In this step-by-step tutorial, you’ll learn what Python’s OrderedDict is and how to use it in your code. You’ll also learn about the main differences between regular dictionaries and ordered dictionaries.
REAL PYTHON

Announcing OpenTelemetry Python 1.0

OpenTelemetry Python distributed tracing API/SDK released its 1.0 version.
MATTHEW MCCLEARY • Shared by Alex Mitelman

PyCharm 2021.1 EAP 5 Is Out

JETBRAINS.COM

Announcing Mu Version 1.1.0-Beta.3

MADEWITH.MU

Python Basics Book: Paperback Copies Now Available

“Python Basics: A Practical Introduction to Python 3” is now available as a paperback book. Go from beginner to intermediate Python knowledge with a clear, concise, and practical curriculum. Print copies available on Amazon and your favorite book store →
REAL PYTHON sponsor

Discussions

Python Made Me Look Like a Wizard

A design team needs to process 745 headshots to fit 150x150px for an external platform. The team: Ugh, this will take over 2 weeks. A Python developer: Hang on, give me 5 minutes.
REDDIT

Python Jobs

Intermediate Python Developer (Boulder, CO, USA)

Uplight

Data Engineer (Seattle, WA, USA)

doxo

Software Development Engineer (Indianapolis, IN, USA)

TOC Logistics International, Inc.

Full-Stack Django Developer (Oslo, Norway)

unifai

More Python Jobs >>>

Articles & Tutorials

Understanding Python List Comprehensions

Python list comprehensions make it easy to create lists while performing sophisticated filtering, mapping, and conditional logic on their members. In this course, you’ll learn when to use a list comprehension in Python and how to create them effectively.
REAL PYTHON course

Using Python and Twilio Messaging to Find Out if the Suez Canal Is Still Blocked

The boat has finally been unstuck, but just in case you want to check on whether or not anything like this happens again, here you go!
SAM AGNEW

$100,000 Build on Redis Hackathon

Join us April 15 – May 15 to show the world what you can build on Redis for a chance to win big prizes totaling US$100,000 →
REDIS LABS sponsor

HPy: A Better C API for Python

The HPy project aims to create a simpler and faster C API for Python.
ANTONIO CUNI

Python Community Interview With Dane Hillard

Dane Hillard, the author of the book Practices of the Python Pro, discusses a variety of topics, including code complexity, Python package maintenance, and popcorn.
REAL PYTHON

Records and Sets: Selecting the Ideal Data Structure

Learn about two of Python’s data structures: records and sets. You’ll look at multiple types and classes for both of these and learn which implementations are best for your specific use cases.
REAL PYTHON course

Understand the Architecture Around your Python Apps in Containers and Serverless Environments

Epsagon lets dev teams see and understand dependencies and API integrations in microservices architecture. It’s a Microservices Observability SaaS with monitoring and investigative tools to ‘drill-down’ and explore modern workloads. Setup Epsagon and see your python services in minutes
EPSAGON sponsor

Executing Encrypted Python with No Performance Penalty

If you need to secure Python’s bytecode to minimize attack surfaces in your production deployment, then you’ll definitely want to check this out!
WOLFGANG RICHTER

All the Important Features and Changes in Python 3.10

Python 3.10 is slated to be released in October of this year. Read about some of the features and changes you can expect to see.
MARTIN HEINZ • Shared by Martin Heinz

Improving the Learning Experience on Real Python

Dan Bader talks about Real Python’s completely refreshed layout designed to improve users’ learning experience.
REAL PYTHON podcast

Projects & Code

hpy: A Better C API for Python

GITHUB.COM/HPYPROJECT

pyce: Encrypted Python Execution

GITHUB.COM/SOROCO

lux: Python API for Intelligent Visual Data Discovery

GITHUB.COM/LUX-ORG

mcpyrate: Advanced Macro Expander and Language Lab for Python

GITHUB.COM/TECHNOLOGICAT

flenser: A Simple, Minimal, Automated Exploratory Data Analysis Tool

GITHUB.COM/JOHNMCCAMBRIDGE

opentelemetry-python: OpenTelemetry Python API and SDK

GITHUB.COM/OPEN-TELEMETRY

icecream: Never Use print() to Debug Again

GITHUB.COM/GRUNS

Polars: Blazingly Fast DataFrames in Rust & Python

GITHUB.COM/RITCHIE46 • Shared by Robin van den Brink

Events

Real Python Office Hours (Virtual)

March 31, 2020
REALPYTHON.COM

Python for ML and AI Global Summit ‘21

April 8 – 10, 2021
GEEKLE.US

PyCon Israel 2021 (Virtual)

May 2 – 3, 2021
PYCON.ORG.IL

PyCon 2021 (Virtual)

May 12 – 18, 2021
PYCON.ORG

DjangoCon Europe 2021 (Virtual)

June 2 – 6, 2021
DJANGOCON.EU

PyCon India 2021

September 17 – 20, 2021
PYCON.ORG • Shared by Gajendra Deshpande

---

Happy Pythoning!
This was PyCoder’s Weekly Issue #466.
View in Browser »

---

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

from Planet Python
via read more

Juri Pakaste: Swift networking with AsyncHTTPClient

When you need to access resources over HTTP in Swift, in most cases the answer is URLSession from Foundation. On server side that's most probably not the right choice; there you are most likely running on SwiftNIO and you'll want something that integrates with it. On command line it's a toss up; on macOS URLSession is great, on other platforms… well, hope you don't run into any unimplemented corners.

I needed a command line tool that fetches a JSON file, parses it, downloads the files listed in the JSON file, and saves the JSON file too. I'm mostly on macOS so URLSession would have been fine, but I wanted to explore my options. SwiftNIO ships with a low-level HTTP client implementation, but that's not the right choice for a quick utility. The good news is there's also a higher-level implementation: AsyncHTTPClient. It's a lovely future based (while we wait for async/await in Swift) asynchronous HTTP client that makes this task a breeze.

The format of the JSON manifest file looks like this:

{
    "files": [
        {
            "file": "filename1",
            "bytes": 42,
            "sha256": "8e079926d7340822e6b4c501811af5d1edc47d796b97f56c1cbe3177b47d588b"
        },

        {
            "file": "filename2",
            "bytes": 10,
            "sha256": "4998ab8c155e03ebf843d4adf51d702b9560dc0fbafe35405c826d9a76460289"
        }
    ]
}

And so on. That's just a couple of structs:

struct ManifestFile: Codable {
    let file: String
}

struct Manifest: Codable {
    let files: [ManifestFile]
}

I'll just ignore bytes and sha256. They are relevant in other contexts, but here they don't matter.

Command line

Let's start by defining a command line interface. We'll use Swift Argument Parser:

import ArgumentParser
import Foundation

struct DownloadManifestFiles: ParsableCommand {
    static var configuration = CommandConfiguration(
        commandName: "manifestdl",
        abstract: "Download manifest files"
    )
    
    @Argument(help: "URL to the manifest")
    var url: String

    @Option(help: "Target directory, default to working directory")
    var directory: String = "."

    mutating func validate() throws {
        guard URL(string: self.url) != nil else {
            throw ValidationError("url \(self.url) was not valid")
        }
    }

    mutating func run() throws {
        guard let reallyURL = URL(string: self.url) else {
            throw ValidationError("URL \(self.url) was not valid")
        }
        let cwd = URL(fileURLWithPath: FileManager.default.currentDirectoryPath, isDirectory: true)
        let directory = URL(fileURLWithPath: self.directory, isDirectory: true, relativeTo: cwd)

        fatalError("TODO")
    }
}

DownloadManifestFiles.main()

Nice and easy. An interesting wrinkle is that ArgumentParser doesn't support URLs, probably because they're in Foundation and ArgumentParser uses only stdlib. And NSURL, which backs URL, is a class with far more responsibilities than is comfortable in a simple data wrapper.

Downloading…

Next we need the actual networking. Let's wrap it in a helper class:

import AsyncHTTPClient
import Foundation
import NIO
import NIOHTTP1

class Downloader {
    let httpClient = HTTPClient(eventLoopGroupProvider: .createNew)

    func syncShutdown() throws {
        try self.httpClient.syncShutdown()
    }

    func downloadListedFiles(url: URL, directory: URL) -> EventLoopFuture<Void> {
        fatalError("TODO")
    }
}

We don't really care about any details about the download, we just want to know if it succeeded or not, hence EventLoopFuture<Void>. EventLoopFuture uses untyped errors, so we don't need to include an error in the type signature. It makes a bit odd next to Result and Combine's Publisher, but it does help when integrating with Swift's exceptions.

Next let's implement the downloadListedFiles method.

class Downloader {
    /* … */
    func downloadListedFiles(url: URL, directory: URL) -> EventLoopFuture<Void> {
        let future = self.downloadManifest(url: manifestURL)
            .flatMap { manifest, data in
                self.downloadManifestContent(
                    manifest: manifest,
                    manifestURL: manifestURL,
                    directory: directory
                ).map { data }
            }
            .flatMap { data in
                self.saveManifest(directory: directory, data: data)
            }
        return future
    }

    func downloadManifest(url: URL) -> EventLoopFuture<(manifest: Manifest, data: Data)> {
        fatalError("TODO")
    }

    func downloadManifestContent(
        manifest: Manifest,
        manifestURL: URL,
        directory: URL
    ) -> EventLoopFuture<Void> {
        fatalError("TODO")
    }

    func saveManifest(directory: URL, data: Data) -> EventLoopFuture<Void> {
        fatalError("TODO")
    }
}

That looks like an acceptable outline for it. Give downloadListedFiles an URL to a listing file and a directory to download to and it'll download the file manifest, parse it, and then download the listed files, and finally save the manifest too. I'll fill in the blanks one by one.

Next let's look how downloadManifest should look.

func downloadManifest(url: URL) -> EventLoopFuture<(manifest: Manifest, data: Data)> {
    self.httpClient.get(url: url.absoluteString)
        .flatMapThrowing { response in
            guard var body = response.body,
                    let data = body.readData(length: body.readableBytes)
            else {
                throw MissingBodyError()
            }
            return (manifest: try JSONDecoder().decode(Manifest.self, from: data), data: data)
        }
}

As you can see, just like ArgumentParser, HTTPClient eschews URL as a Foundation type. Other than that, HTTPClient gives us a really easy interface. Just .get a String containing an URL, and then do whatever you need to do with a chaining method like .map, .flatMap, or as in this case, .flatMapThrowing.

Next we can tackle downloadManifestContent. It's the function that's responsible for downloading all the listed files.

func downloadManifestContent(
    manifest: Manifest,
    manifestURL: URL,
    directory: URL
) -> EventLoopFuture<Void> {
    let baseURL = manifestURL.deletingLastPathComponent()
    let requestFutures: [EventLoopFuture<Void>]
    do {
        requestFutures = try manifest.files.map { manifestFile in
            let localURL = directory.appendingPathComponent(manifestFile.file)
            try FileManager.default.createDirectory(
                at: localURL.deletingLastPathComponent(),
                withIntermediateDirectories: true,
                attributes: nil
            )
            let localPath = localURL.path
            let delegate = try FileDownloadDelegate(path: localPath, pool: self.threadPool)
            let request = try HTTPClient.Request(
                url: baseURL.appendingPathComponent(manifestFile.file).absoluteString
            )
            return self.httpClient.execute(request: request, delegate: delegate)
                .futureResult
                .map { _ in () }
        }
    } catch {
        return self.httpClient.eventLoopGroup.next().makeFailedFuture(error)
    }
    return EventLoopFuture.andAllSucceed(requestFutures, on: self.eventLoopGroup.next())
}

var eventLoopGroup: EventLoopGroup {
    self.httpClient.eventLoopGroup
}

This one's not quite as simple, but it's not too bad. For each listed file we create a future for downloading it. The download to disk, as opposed to memory, happens with the help of FileDownloadDelegate, a delegate included in AsyncHTTPClient that can write downloads to disk and report progress. Then once we have a list of futures, we smash them all together with andAllSucceeded. Again we don't care about anything else other than success, so Void is a perfectly fine value type.

One detail I need to point out here is the eventLoopGroup property. SwiftNIO works with EventLoops, and EventLoops are apparently usually threads. While we're working only with with networking code it's probably not a problem to ask HTTPClient for its EventLoopGroup instance.

File I/O

We have read the manifest and written the files listed in it to disk. One thing left to do: saving the manifest. Writing the file with SwiftNIO isn't quite as friendly as AsyncHTTPClient is, and if you were doing more of this you'd want to put a nicer façade on it, but here we just need it this once.

To prepare for this, lets first set up a bit of scaffolding. It feels cleaner to move the management of the EventLoopGroup to our own code now that we're using it for not just the HTTP client, and we'll also need a thread pool for the file I/O.

class Downloader {
    let eventLoopGroup: EventLoopGroup // replaces the computed property
    let httpClient: HTTPClient
    let threadPool: NIOThreadPool

    init() {
        self.eventLoopGroup = NIOTSEventLoopGroup()
        self.httpClient = HTTPClient(eventLoopGroupProvider: .shared(self.eventLoopGroup))
        self.threadPool = NIOThreadPool(numberOfThreads: 1)

        self.threadPool.start()
    }

    func syncShutdown() throws {
        try self.httpClient.syncShutdown()
        try self.threadPool.syncShutdownGracefully()
        try self.eventLoopGroup.syncShutdownGracefully()
    }

    /* … */
}

Forcing NIOTSEventLoopGroup here probably ties this code to macOS. For portability, there are other implementations. Here's what AsyncHTTPClient does when you ask it to create the event loop group itself:

#if canImport(Network)
    if #available(OSX 10.14, iOS 12.0, tvOS 12.0, watchOS 6.0, *) {
        self.eventLoopGroup = NIOTSEventLoopGroup()
    } else {
        self.eventLoopGroup = MultiThreadedEventLoopGroup(numberOfThreads: 1)
    }
#else
    self.eventLoopGroup = MultiThreadedEventLoopGroup(numberOfThreads: 1)
#endif

Doing something similar in your own code should help make this more cross platform.

With that setup done, we can dive into the file writing itself.

func saveManifest(directory: URL, data: Data) -> EventLoopFuture<Void> {
    let io = NonBlockingFileIO(threadPool: self.threadPool)
    let eventLoop = self.eventLoopGroup.next()
    let buffer = ByteBuffer(data: data)
    return io
        .openFile(
            path: directory.appendingPathComponent("manifest.json").path,
            mode: .write,
            flags: .allowFileCreation(),
            eventLoop: eventLoop
        ).flatMap { handle in
            io.write(fileHandle: handle, buffer: buffer, eventLoop: eventLoop)
                .map { handle }
        }.flatMapThrowing { handle in
            try handle.close()
        }
}

It's two async operations and one synchronous one in a pipeline. Open file asynchronously, write file asynchronously, close it synchronously. The flatMaps can feel a little daunting if you're not used to them, as always with future libraries. But once you get used to them, it's pretty OK. Async/await will hopefully help.

After all that work we're ready to loop back to our run method. We left it calling fatalError after processing the arguments. Now we can finish it up:

mutating func run() throws {
    guard let reallyURL = URL(string: self.url) else {
        throw ValidationError("URL \(self.url) was not valid")
    }
    let cwd = URL(fileURLWithPath: FileManager.default.currentDirectoryPath, isDirectory: true)
    let directory = URL(fileURLWithPath: self.directory, isDirectory: true, relativeTo: cwd)

    let downloader = Downloader()
    let dlFuture = downloader.downloadListedFiles(url: reallyURL, directory: directory)

    defer {
        do {
            try downloader.syncShutdown()
        } catch {
            print("Error shutting down: \(error)")
        }
    }
    try dlFuture.wait()
}

And that's it! Create a Downloader, call it to get a future, set up cleanup, then wait until the future is done.

Conclusion

SwiftNIO is a fantastic library that powers the Swift on the server ecosystem. It's also great with command line tooling. It can occasionally be a bit more involved than Foundation, but especially with HTTP requests the difference is negligible. You'd have had to bring in Combine too to make URLSessions composable.

The Foundation/standard library split is a bit awkward here, as it often is when working with Swift command line tools. It's not that Foundation doesn't work, but it's clear that often there's the Swift way and then there's the Foundation way. And Foundation's cross platform story has been a bit rocky.

As Swift's async story progresses a lot of this code can be simplified, I hope. In the ideal case the structure would stay pretty much as is, but those nested maps and flatMaps could be replaced with more straightforward code. However, I don't think you need to wait for async/await and all the related enhancements to arrive. This is already pretty great.

from Planet Python
via read more