Stack Abuse: Convert Bytes to String in Python

Introduction

In this article, we'll take a look at how to convert Bytes to a String in Python. By the end of this article you will have a clear idea of what these types are and how to effectively handle data using them.

Depending on the version of Python you're using, this task will differ. Although Python 2 has reached its end of life, many projects still use it, so we'll include both the Python 2 and Python 3 approaches.

Convert Bytes to String in Python 3

Since Python 3, the old ASCII way of doing things had to go, and Python became completely Unicode.

This means that we lost the explicit unicode type: u"string" - every string is a u"string"!

To differentiate these strings from good old bytestrings, we're introduced to a new specifier for them - the b"string".

This was added in Python 2.6, but it served no real purpose other than to prepare for Python 3 as all strings were bytestrings in 2.6.

Bytestrings in Python 3 are officially called bytes, an immutable sequence of integers in the range 0 <= x < 256. Another bytes-like object added in 2.6 is the bytearray - similar to bytes, but mutable.

Convert Bytes to String with decode()

Let's take a look at how we can convert bytes to a String, using the built-in decode() method for the bytes class:

>>> b = b"Lets grab a \xf0\x9f\x8d\x95!"
# Let's check the type
>>> type(b)
<class 'bytes'>

# Now, let's decode/convert them into a string
>>> s = b.decode('UTF-8')
>>> s
"Let's grab a 🍕!"

Passing the encoding format, we've decoded the bytes object into a string and printed it.

Convert Bytes to String with codecs

Alternatively, we can use the built-in codecs module for this purpose as well:

>>> import codecs
>>> b = b'Lets grab a \xf0\x9f\x8d\x95!'

>>> codecs.decode(b, 'UTF-8')
"Let's grab a 🍕!"

You don't really need to pass in the encoding parameter, though, it is advised to pass it in:

>>> codecs.decode(b)
"Let's grab a 🍕!"

Convert Bytes to String with str()

Finally, you can use the str() function, which accepts various values and converts them into strings:

>>> b = b'Lets grab a \xf0\x9f\x8d\x95!'
>>> str(b, 'UTF-8')
"Let's grab a 🍕!"

Make sure to provide the encoding argument to str() though, otherwise you might get some unexpected results:

>>> str(b)
b'Lets grab a \xf0\x9f\x8d\x95!'

This brings us to encodings once again. If you specify the wrong encoding, the best case is your program crashing because it can't decode the data. For example, if we tried using the str() function with UTF-16, we'd be greeted with:

>>> str(b, 'UTF-16')
'敌❴\u2073牧扡愠\uf020趟↕'

This is even more important given that Python 3 likes to assume Unicode - so if you are working with files or data sources that use an obscure encoding, make sure to pay extra attention.

Convert Bytes to String in Python 2

In Python 2, a bundle of bytes and a string are practically the same thing - strings are objects consisting of 1-byte long characters, meaning that each character can store 256 values. That's why they are sometimes called bytestrings.

This is great when working with byte data - we just load it into a variable and we are ready to print:

>>> s = "Hello world!"

>>> s
'Hello world!'

>>> len(s)
12

Using Unicode characters in bytestrings does change this behavior a bit though:

>>> s = "Let's grab a 🍕!"

>>> s
'Lets grab a \xf0\x9f\x8d\x95!'
# Where has the pizza gone to?

>>> len(s)
17
# Shouldn't that be 15?

Convert Bytes to Unicode (Python 2)

Here, we'll have to use Python 2's Unicode type, which is assumed and automatically used in Python 3. This stores strings as a series of code points, rather than bytes.

The \xf0\x9f\x8d\x95 represents bytes as two-digit hex numbers as Python doesn't know how to represent them as ASCII characters:

>>> u = u"Let's grab a 🍕!"
u"Let's grab a \U0001f355!""

>>> u
"Let's grab a 🍕!"
# Yum.

>>> len(u)
15

As you can see above, the Unicode string contains \U0001f355 - a Unicode escaped character which our terminal now knows how to print out as a slice of pizza! Setting this was as easy as using the u specifier before the value of the bytestring.

So, how do I switch between the two?

You can get the Unicode string by decoding your bytestring. This can be done by constructing a Unicode object, providing the bytestring and a string containing the encoding name as arguments or by calling .decode(encoding) on a bytestring.

Convert Bytes to String Using decode() (Python 2)

You can also use the codecs.encode(s, encoding) from the codecs module.

>>> s = "Let's grab a \xf0\x9f\x8d\x95!"
>>> u = unicode(s, 'UTF-8')

>>> u
"Let's grab a 🍕!"

>>> s.decode('UTF-8')
"Let's grab a 🍕!"

Convert Bytes to String Using codecs (Python 2)

Or, using the codecs module:

import codecs

>>> codecs.decode(s, 'UTF-8')
"Let's grab a 🍕!"

Be Mindful of your Encoding

A word of caution here - bytes can be interpreted differently in different encodings. With around 80 different encodings available out of the box, it might not be easy to know if you've got the right one!

s = '\xf8\xe7'

# This one will let us know we used the wrong encoding

>>> s.decode('UTF-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf8 in position 0:
invalid start byte

# These two overlaps and this is a valid string in both

>>> s.decode('latin1')
øç

s.decode('iso8859_5')
јч

The original message was either øç or јч, and both appear to be valid conversions.

Conclusion

As programmers, there are some things we must constantly think about and actively prepare for in order to avoid pitfalls. This holds especially true on the lower levels, where we seldom go when we use a high-level language like Python as our daily driver.

Things like charsets, encodings and binary are there to remind us that our job is to code - to encode our thoughts into working solutions. Thankfully, a lot of this thinking becomes part of our routine after a few rounds at the keyboard.

In this article, we've gone over how to convert bytes to Strings in Python.

from Planet Python
via read more