What's new in Python 3.11? - DeepSource (2023)

The first beta release of Python 3.11 is out, bringing some fascinating featuresfor us to tinker with. This is what you can expect to see in 2022's release ofPython later this year.

Even better error messages

Python 3.10 gave us better error messages in various regards, but Python 3.11aims to improve them even more. Some of the most important things that are addedto error messages in Python 3.11 are:

Exact error locations in tracebacks

Until now, in a traceback, the only information you got about where anexception got raised was the line. The issue could have been anywhere on theline though, so sometimes this information was not enough.

Here's an example:

def get_margin(data): margin = data['profits']['monthly'] / 10 + data['profits']['yearly'] / 2 return margindata = { 'profits': { 'monthly': 0.82, 'yearly': None, }, 'losses': { 'monthly': 0.23, 'yearly': 1.38, },}print(get_margin(data))

This code results in an error, because one of these fields in the dictionary isNone. This is what we get:

Traceback (most recent call last): File "/Users/tusharsadhwani/code/marvin-python/mytest.py", line 15, in <module> print(get_margin(data)) File "/Users/tusharsadhwani/code/marvin-python/mytest.py", line 2, in print_margin margin = data['profits']['monthly'] / 10 + data['profits']['yearly'] / 2TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

But it is impossible to tell by the traceback itself, which part of thecalculation caused the error.

On 3.11 however:

Traceback (most recent call last): File "asd.py", line 15, in <module> print(get_margin(data)) ^^^^^^^^^^^^^^^^ File "asd.py", line 2, in print_margin margin = data['profits']['monthly'] / 10 + data['profits']['yearly'] / 2 ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

It is crystal clear, that data['profits']['yearly'] was None.

To be able to render this information, the end_line and end_col data wasadded to Python code objects. You can also access this information directlythrough the obj.__code__.co_positions() method.

Notes for exceptions

To make tracebacks even more context rich, Python 3.11 allows you to add notesto exception objects, which get stored in the exceptions, and displayed whenthe exception is raised.

Take this code for example, where we add important information about some APIdata conversion logic:

def get_seconds(data): try: milliseconds = float(data['milliseconds']) except ValueError as exc: exc.add_note( "The time field should always be a number, this is a critial bug. " "Please report this to the backend team immediately." ) raise seconds = milliseconds / 1000 return secondsget_seconds({'milliseconds': 'foo'}) 

This added note gets printed just below the Exception message:

(Video) CPython: Async Task Groups in Python 3.11 - Python Bytes Live Stream Episode 271

Traceback (most recent call last): File "asd.py", line 14, in <module> get_seconds({"milliseconds": "foo"}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "asd.py", line 3, in get_seconds milliseconds = float(data["milliseconds"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError: could not convert string to float: 'foo'The time field should always be a number, this is a critial bug. Please report this to the backend team immediately.

Built in toml support

The standard library now has built-in support for reading TOML files, using thetomllib module:

import tomllibwith open('.deepsource.toml', 'rb') as file: data = tomllib.load(file)

tomllib is actually based on an open source TOML parsing library calledtomli. And currently, only reading TOML files is supported. If you need towrite data to a TOML file instead, consider using the tomli-w package.

asyncio Task Groups

When doing asynchronous programming, you often run into situations where youhave to trigger many tasks to run concurrently, and then take some action whenthey are completed. For example, downloading a bunch of images in parallel, andthen bundling them to a zip file at the end.

To do that, you need to collect tasks and pass them to asyncio.gather.Here's asimple example of parallelly running tasks with the gather function:

import asyncioasync def simulate_flight(city, departure_time, duration): await asyncio.sleep(departure_time) print(f"Flight for {city} departing at {departure_time}PM") await asyncio.sleep(duration) print(f"Flight for {city} arrived.")flight_schedule = { 'Boston': [3, 2], 'Detroit': [7, 4], 'New York': [1, 9],}async def main(): tasks = [] for city, (departure_time, duration) in flight_schedule.items(): tasks.append(simulate_flight(city, departure_time, duration)) await asyncio.gather(*tasks) print("Simulations done.")asyncio.run(main())
$ python asd.pyFlight for New York departing at 1PMFlight for Boston departing at 3PMFlight for Boston arrived.Flight for Detroit departing at 7PMFlight for New York arrived.Flight for Detroit arrived.Simulations done.

But having to maintain a list of the tasks yourself to be able to await themis a bit clunky. So now a new API is added to asyncio called Task Groups:

import asyncioasync def simulate_flight(city, departure_time, duration): await asyncio.sleep(departure_time) print(f"Flight for {city} departing at {departure_time}PM") await asyncio.sleep(duration) print(f"Flight for {city} arrived.")flight_schedule = { 'Boston': [3, 2], 'Detroit': [7, 4], 'New York': [1, 9],}async def main(): async with asyncio.TaskGroup() as tg: for city, (departure_time, duration) in flight_schedule.items(): tg.create_task(simulate_flight(city, departure_time, duration)) print("Simulations done.")asyncio.run(main())

When the asyncio.TaskGroup() context manager exits, it ensures that all thetasks created inside it have finished running.

Bonus: Exception groups

A similar feature was also added for exception handling inside async tasks,called exception groups.

Say you have many async tasks running together, and some of them raised errors.Currently Python's exception handling system doesn't work well in this scenario.

Here's a short demo of what it looks like with 3 concurrent crashing tasks:

import asynciodef bad_task(): raise ValueError("oops")async def main(): tasks = [] for _ in range(3): tasks.append(asyncio.create_task(bad_task())) await asyncio.gather(*tasks)asyncio.run(main())

When you run this code:

$ python asd.pyTraceback (most recent call last): File "asd.py", line 13, in <module> asyncio.run(main()) File "/usr/bin/python3.8/lib/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/bin/python3.8/lib/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "asd.py", line 9, in main tasks.append(asyncio.create_task(bad_task())) File "asd.py", line 4, in bad_task raise ValueError("oops")ValueError: oops

There's no indication that 3 of these tasks were running together. As soon asthe first one fails, it crashes the whole program.

But in Python 3.11, the behaviour is a bit better:

import asyncioasync def bad_task(): raise ValueError("oops")async def main(): async with asyncio.TaskGroup() as tg: for _ in range(3): tg.create_task(bad_task())asyncio.run(main())
$ python asd.py + Exception Group Traceback (most recent call last): | File "<stdin>", line 1, in <module> | File "/usr/local/lib/python3.11/asyncio/runners.py", line 181, in run | return runner.run(main) | ^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.11/asyncio/runners.py", line 115, in run | return self._loop.run_until_complete(task) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/usr/local/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete | return future.result() | ^^^^^^^^^^^^^^^ | File "<stdin>", line 2, in main | File "/usr/local/lib/python3.11/asyncio/taskgroups.py", line 139, in __aexit__ | raise me from None | ^^^^^^^^^^^^^^^^^^ | ExceptionGroup: unhandled errors in a TaskGroup (3 sub-exceptions) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "<stdin>", line 2, in bad_task | ValueError: oops +---------------- 2 ---------------- | Traceback (most recent call last): | File "<stdin>", line 2, in bad_task | ValueError: oops +---------------- 3 ---------------- | Traceback (most recent call last): | File "<stdin>", line 2, in bad_task | ValueError: oops +------------------------------------

The exception now tells us that we had 3 errors thrown, in a structure known asan ExceptionGroup.

Exception handling with these exception groups is also interesting, you caneither do except ExceptionGroup to catch all the exceptions in one go:

$ python asd.pyCaught exceptions: unhandled errors in a TaskGroup (3 sub-exceptions)

Or you can catch them based on the exception type, using the new except*syntax:

try: asyncio.run(main())except* ValueError as eg: print(f"Caught ValueErrors: {eg}")
$ python asd.pyCaught ValueErrors: unhandled errors in a TaskGroup (3 sub-exceptions)

Typing improvements

The typing module saw a lot of interesting updates this release. Here are someof the most exciting ones:

Variadic generics

Support for variadic generics has been added to the typing module in Python3.11.

What that means is that, now you can define generic types that can take anarbitrary number of types in them. It is useful for defining generic methodsfor multi-dimensional data.

For example:

from typing import Genericfrom typing_extensions import TypeVarTupleShape = TypeVarTuple('Shape')class Array(Generic[*Shape]): ...items: Array[int] = Array()market_prices: Array[int, int, float] = Array()def double(array: Array[Unpack[Shape]]) -> Array[Unpack[Shape]]: ...def get_values(array: Array[int, int, *Shape]) -> Array[*Shape]: ...vector_space: Array[int, int, complex] = Array()reveal_type(get_values(vector_space)) 

Variadic generics can be really useful for defining functions that map overN-dimensional data. This feature can help a lot in type checking codebases thatrely on data science libraries such as numpy or tensorflow.

The new Generic[*Shape] syntax is only supported in Python 3.11. To use thisfeature in Python 3.10 and below, you can use the typing.Unpack builtininstead: Generic[Unpack[Shape]].

singledispatch now supports unions

functools.singledispatch is a neat way to do function overloading in Python,based on type hints. It works by defining a generic function, and decorating itwith @singledispatch. Then you can define specialized variants of thatfunction, based on the type of the function arguments:

from functools import singledispatch@singledispatchdef half(x): """Returns the half of a number""" return x / 2@half.registerdef _(x: int): """For integers, return an integer""" return x // 2@half.registerdef _(x: list): """For a list of items, get the first half of it.""" list_length = len(x) return x[: list_length // 2] print(half(3.6)) print(half(15)) print(half([1, 2, 3, 4])) 

By inspecting the type given to the function's arguments, singledispatchcan create generic functions, providing a non object-oriented way to do functionoverloading.

But this is all old news. What Python 3.11 brings is that now, you canpass union types for these arguments. For example, to register a function forall the number types, previously you would have to do it separately for eachtype, such as float, complex or Decimal:

@half.registerdef _(x: float): return x / 2@half.registerdef _(x: complex): return x / 2@half.registerdef _(x: decimal.Decimal): return x / 2

But now, you can specify all of them in a Union:

@half.registerdef _(x: float | complex | decimal.Decimal): return x / 2

And the code will work exactly as expected.

Self type

Previously, if you had to define a class method that returned an object of theclass itself, adding types for it was a bit weird, it would look something likethis:

from typing import TypeVarT = TypeVar('T', bound=type)class Circle: def __init__(self, radius: int) -> None: self.radius = radius @classmethod def from_diameter(cls: T, diameter) -> T: circle = cls(radius=diameter/2) return circle

To be able to say that a method returns the same type as the class itself, youhad to define a TypeVar, and say that the method returns the same type T asthe current class itself.

But with the Self type, none of that is needed:

from typing import Selfclass Circle: def __init__(self, radius: int) -> None: self.radius = radius @classmethod def from_diameter(cls, diameter) -> Self: circle = cls(radius=diameter/2) return circle

Required[] and NotRequired[]

TypedDict is really useful to add type information to a codebase that usesdictionaries heavily to store data. Here's how you can use them:

from typing import TypedDictclass User(TypedDict): name: str age: intuser : User = {'name': "Alice", 'age': 31}reveal_type(user['age']) 

However, TypedDicts had a limitation, where you could not have optionalparameters inside a dictionary, kind of like default parameters inside functiondefinitions.

For example, you can do this with a NamedTuple:

from typing import NamedTupleclass User(NamedTuple): name: str age: int married: bool = Falsemarie = User(name='Marie', age=29, married=True)fredrick = User(name='Fredrick', age=17) 

This was not possible with a TypedDict (at least without defining multiple ofthese TypedDict types). But now, you can mark any field as NotRequired, tosignal that it is okay for the dictionary to not have that field:

from typing import TypedDict, NotRequiredclass User(TypedDict): name: str age: int married: NotRequired[bool]marie: User = {'name': 'Marie', 'age': 29, 'married': True}fredrick : User = {'name': 'Fredrick', 'age': 17} 

NotRequired is most useful when most fields in your dictionary are required,having a few not required fields. But, for the opposite case, you can tellTypedDict to treat every single field as not required by default, and then useRequired to mark actually required fields.

For example, this is the same as the previous code:

from typing import TypedDict, Requiredclass User(TypedDict, total=False): name: Required[str] age: Required[int] married: bool marie: User = {'name': 'Marie', 'age': 29, 'married': True}fredrick : User = {'name': 'Fredrick', 'age': 17} 

contextlib.chdir

contextlib has a small addition to it, which is a context manager calledchdir. All it does is change the current working directory to the specifieddirectory inside the context manager, and set it back to what it was before whenit exits.

One potential usecase can be to redirect where you write the logs to:

import osdef write_logs(logs): with open('output.log', 'w') as file: file.write(logs)def foo(): print("Scanning files...") files = os.listdir(os.curdir) logs = do_scan(files) print("Writing logs to /tmp...") with contextlib.chdir('/tmp'): write_logs(logs) print("Deleting files...") files = os.listdir(os.curdir) do_delete(files)

This way, you don't have to worry about changing and reverting the currentdirectory manually, the context manager will do it for you.

Summary

In addition to all of these, there's a cherry on top: Python also got 22% fasteron average in this release. It might even get faster by the time the finalrelease is out around October!

Also, work on Python 3.12 has already started. If you want to stay up to datewith all the latest developments with the language, you can check out thepull requests page on Python's GitHubrepository.

FAQs

Which is Python 3.11 release date? ›

This release, 3.11.

The second candidate and the last planned release preview is currently planned for Monday, 2022-09-05 while the official release is planned for Monday, 2022-10-03.

Which Python version is best in 2022? ›

The 7th version 3.11. 0a7 is currently the latest alpha version which was released on 17th April, 2022.
...
  • 5 New features in Python 3.11 that makes it the coolest new release in 2022. ...
  • Better Error Messages. ...
  • CPython Opimizations. ...
  • Adding the new typing feature: Self.

Is Python 3.10 The latest? ›

Python 3.10.6 is the newest major release of the Python programming language, and it contains many new features and optimizations.

What is the latest Python version? ›

Stable Releases
  • Python 3.10.6 - Aug. 2, 2022. ...
  • Python 3.10.5 - June 6, 2022. Note that Python 3.10.5 cannot be used on Windows 7 or earlier. ...
  • Python 3.9.13 - May 17, 2022. ...
  • Python 3.10.4 - March 24, 2022. ...
  • Python 3.9.12 - March 23, 2022. ...
  • Python 3.10.3 - March 16, 2022. ...
  • Python 3.9.11 - March 16, 2022. ...
  • Python 3.8.13 - March 16, 2022.

Will there be a Python 4? ›

At the time of writing this post, there is no release date for Python 4 yet. The next version is going to be 3.9. 0 which is scheduled to be released on October 5, 2020, it is planned to have support approximately until October 2025, so the next release after 3.9 should come out somewhere between 2020 and 2025.

What is __ new __ in Python? ›

The __new__() is a static method of the object class. When you create a new object by calling the class, Python calls the __new__() method to create the object first and then calls the __init__() method to initialize the object's attributes.

Which Python 3 version is best? ›

For the sake of compatibility with third-party modules, it is always safest to choose a Python version that is one major point revision behind the current one. At the time of this writing, Python 3.8. 1 is the most current version. The safe bet, then, is to use the latest update of Python 3.7 (in this case, Python 3.7.

Is Python 3.10 backwards compatible? ›

The Python language does not provide backward compatibility. Changes which are not clearly incompatible are not covered by this PEP. For example, Python 3.9 changed the default protocol in the pickle module to Protocol 4 which was first introduced in Python 3.4.

Which is the most stable Python version? ›

We are currently living in the stable age of Python 3.8 and the latest stable version of Python, 3.8.

Does Python 3.10 have pip? ›

The current version of pip works on: Windows, Linux and MacOS. CPython 3.7, 3.8, 3.9, 3.10 and latest PyPy3.

Is Python easy to learn? ›

Python is widely considered among the easiest programming languages for beginners to learn. If you're interested in learning a programming language, Python is a good place to start. It's also one of the most widely used.

How do I use Python 3.10 4? ›

How to Install Python 3.10.4 on Windows 10/11 [ 2022 ... - YouTube

What changed in Python 3? ›

Python 3.0 uses the concepts of text and (binary) data instead of Unicode strings and 8-bit strings. All text is Unicode; however encoded Unicode is represented as binary data. The type used to hold text is str , the type used to hold data is bytes . The biggest difference with the 2.

Can I use Python 3.10 with anaconda? ›

Anaconda supports Python 3.7, 3.8, 3.9 and 3.10. The current default is Python 3.9.

How many GB is Python? ›

1. The Python download requires about 18 MB of disk space; keep it on your machine, in case you need to re-install Python. When installed, Python requires about an additional 90 MB of disk space.

Does PyPy support Python 3? ›

PyPy supports Python 2.7. PyPy3, released in beta, targets Python 3.

How do I uninstall Python 3.11 on Mac? ›

Mac
  1. Go to the Finder.
  2. Click on Applications in the menu on the left.
  3. Find the Python folder with the version number you want to uninstall, right-click it, and select “Move to Trash”.

What is the Python version? ›

Python 3.9. 6, documentation released on 28 June 2021.

What is the meaning of != In Python? ›

The != operator in Python stands for "not equal to." If the operands on either side are not equal, it returns True; if they are, it returns False. The is not operator, on the other hand, checks whether the id() of two objects is. the same or not.

Top Articles
Latest Posts
Article information

Author: Dan Stracke

Last Updated: 02/11/2023

Views: 6198

Rating: 4.2 / 5 (43 voted)

Reviews: 90% of readers found this page helpful

Author information

Name: Dan Stracke

Birthday: 1992-08-25

Address: 2253 Brown Springs, East Alla, OH 38634-0309

Phone: +398735162064

Job: Investor Government Associate

Hobby: Shopping, LARPing, Scrapbooking, Surfing, Slacklining, Dance, Glassblowing

Introduction: My name is Dan Stracke, I am a homely, gleaming, glamorous, inquisitive, homely, gorgeous, light person who loves writing and wants to share my knowledge and understanding with you.