Poetry Usage Guide
Python users are certainly familiar with conda, this popular package manager. However, after using it for a long time, people often become confused by its aggressive environment-management approach and bloated package set. Anaconda is too heavy: it installs many scientific-computing packages by default and takes about 5-6 GB after installation. You can choose Anaconda's lightweight edition, miniconda. It does not include those default scientific libraries, but miniconda can still be too aggressive in what it installs. Given the same packages, conda may install more dependencies than other package managers, which keeps inflating project environments. So I decided to start using Poetry to manage virtual environments.
What is Poetry
Poetry is roughly a combination of pip + venv. Like pip, it can be used to manage third-party modules, but it is much more capable than pip, and it also includes virtual environment management similar to venv. Its main capabilities include:
- Managing installation and uninstallation of third-party modules
- Managing virtual environments
- Managing dependencies in virtual environments
- Managing packaging and publishing
Terminology: virtual environment management, module management, and module dependency management
Before we begin, let's clarify how they relate to each other.
Virtual environments
Virtual environments refer to built-in tools like venv or virtualenv, conda, and other tools used to create and manage Python virtual environments. Different virtual environments are isolated from each other, and their storage locations and installed modules differ as well.
Module management and module dependency management
Modules refer to the third-party modules installed in a virtual environment and their versions. Most projects have specific version requirements for third-party libraries. If you use newer dependencies for an old project, you may run into strange errors.
When installing a third-party module, that module may install its own dependencies. Once two or more modules are installed, dependency conflicts may occur among those third-party modules. This usually means version conflicts among dependencies, which is what we call dependency relationships.
Limitations of pip
pip is Python's built-in dependency manager, and its biggest limitation is weak dependency-resolution capability for third-party modules. This is especially evident when removing third-party modules: in terms of dependency analysis, it essentially does none.
Let's look at an example:
- Create a virtual environment
~/coding> conda create -n myenv python=3.13
(myenv) ~/coding>
- Install flask and view installed third-party modules (dependencies)
(myenv) ~/coding>pip install flask
(myenv) ~/coding>pip list
Package Version
------------ -------
blinker 1.6.2
click 8.1.3
colorama 0.4.6
Flask 2.3.2
itsdangerous 2.1.2
Jinja2 3.1.2
MarkupSafe 2.1.2
pip 22.3.1
setuptools 65.5.0
Werkzeug 2.3.6
- Then remove flask
(myenv) ~/coding>pip uninstall flask
Found existing installation: Flask 2.3.2
Uninstalling Flask-2.3.2:
Would remove:
/Users/username/anaconda3/envs/myvenv/lib/python3.13/site-packages/flask-2.3.2.dist-info\*
/Users/username/anaconda3/envs/myvenv/lib/python3.13/site-package/flask\*
Proceed (Y/n)? y
Successfully uninstalled Flask-2.3.2
(myvenv) ~/codingo>pip list
Package Version
------------ -------
blinker 1.6.2[pyproject.toml](..%2F..%2F..%2Fpoetry-demo%2Fpyproject.toml)
click 8.1.3
colorama 0.4.6
itsdangerous 2.1.2
Jinja2 3.1.2
MarkupSafe 2.1.2
pip 22.3.1
setuptools 65.5.0
Werkzeug 2.3.6
You'll notice that only flask itself is gone, while all dependencies installed alongside flask remain. In other words, when pip installs a module, related dependencies are downloaded and installed too. But during uninstall, pip does not manage dependencies; it simply removes the specified module and leaves a pile of dependencies behind.
Using Poetry from scratch
Installation
Poetry is a command-line tool. After installing it, you can use Poetry commands. You can install it globally or inside a virtual environment.
pip install poetry
After installation, poetry.exe appears in the Scripts directory under the Python interpreter's installation path. Since environment variables are configured when Python is installed, you can use it globally right away.
Initialize a Poetry project
To make explanation easier, let's create a new project named poetry_test.
All settings are simple. I suggest following along and trying each step yourself.
- Initialize the project
~/coding>mkdir poetry_test
~/coding> cd poetry_test
~/coding/poetry_test> poetry init
Then an interactive prompt sequence appears to create your project config file. I just pressed Enter all the way through. Here's the generated pyproject.toml:
[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.13"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
At this point, the project directory structure is:
poetry_test
└── pyproject.toml
0 directories, 1 file
Managing virtual environments
Poetry has many default virtual environment settings, which you can modify via poetry config.
On macOS, Poetry defaults to creating virtual environments in /Users/<username>/Library/Caches/pypoetry/virtualenvs. When you run commands such as poetry add, Poetry automatically checks whether a virtual environment is currently active:
- If yes, it installs modules directly into the current virtual environment.
- If no, it automatically creates a new virtual environment and then installs modules.
Creating a virtual environment
Use the command poetry env use python:
─ ~/coding/poetry_test 82% 21:21 11.11
╰ poetry env use python
Creating virtualenv poetry-test-ynfwcHPh-py3.13 in /Users/username/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/username/Library/Caches/pypoetry/virtualenvs/poetry-test-ynfwcHPh-py3.13
You can see Poetry created a virtual environment named poetry-test-ynfwcHPh-py3.13.
Key note
poetry env use python creates a virtual environment using the currently active Python interpreter in your shell.
You can also replace the final python with python3, python3.8, and so on. If you specifically need 3.8, just ensure the corresponding interpreter can be found in your environment variables.
For more configuration details, see the official docs.
By default, Poetry stores virtual environments in a unified location. For example, the project just created lives in /Users/username/Library/Caches/pypoetry/virtualenvs/.
The virtual environment naming format is project-name-random-string-python-version.
Create a virtual environment inside the current project
You can run poetry config --list to view several major Poetry settings:
╭─ ~/coding/poetry_test 82% 21:21 11.11
╰ poetry config --list
cache-dir = "/Users/username/Library/Caches/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs" # /Users/username/Library/Caches/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true
If virtualenvs.create = true is changed to false, Poetry stops automatically creating a virtual environment when one is not detected, but I recommend not changing that.
Our target is virtualenvs.in-project = false. Change it with:
poetry config virtualenvs.in-project true
First, remove the previously created virtual environment:
╭─ ~/coding/poetry_test 82% 21:28 11.11
╰ poetry env remove python
Deleted virtualenv: /Users/username/Library/Caches/pypoetry/virtualenvs/poetry-test-ynfwcHPh-py3.13
Recreate the virtual environment and check the difference:
╭─ ~/coding/poetry_test 80% 21:44 11.11
╰ poetry env use python3.12
Creating virtualenv poetry-test in /Users/username/coding/poetry_test/.venv
Using virtualenv: /Users/username/coding/poetry_test/.venv
You can see:
- The virtual environment path is now inside the project root.
- The name is fixed as
.venv.
Entering and exiting a virtual environment
From the project root, run poetry shell to enter the virtual environment.
╭─ ~/coding/poetry_test/.venv 80% 21:45 11.11
╰ poetry shell
Spawning shell within /Users/username/coding/poetry_test/.venv
╭─ ~/coding/poetry_test/.venv 21:46 11.11
╰ emulate bash -c '. /Users/username/coding/poetry_test/.venv/bin/activate'
╭─ ~/coding/poetry_test 80% 21:49 11.11
╰ python --version
Python 3.12.3
poetry shell checks the current or parent directory for pyproject.toml to determine which virtual environment to activate. So if you are not in the project directory, errors will occur.
Exiting the virtual environment is even simpler: just type exit.
─ ~/coding/poetry_test 79% 21:49 11.11
╰ exit
╭─ ~/coding/poetry_test/.venv 02:59 79% 21:49 11.11
╰ python --version
Python 3.11.5
Poetry commands
Poetry is an independent CLI tool with its own command system. It takes additional time and effort to learn and is more complex than pip, which can be a hurdle when adopting Poetry. Fortunately, the commonly used commands are fewer than 10. Let's go through them one by one.
Install modules
Use:
poetry add
Compared with pip install, let's try installing flask and see what changes.
─ ~/coding/poetry_test 3.15s 79% 21:51 11.11
╰ poetry add flask
Using version ^3.0.3 for flask
Updating dependencies
Resolving dependencies... (23.1s)
Package operations: 7 installs, 0 updates, 0 removals
- Installing markupsafe (3.0.2)
- Installing blinker (1.9.0)
- Installing click (8.1.7): Failed
TimeoutError
The read operation timed out
at ~/anaconda3/lib/python3.11/ssl.py:1167 in read
1163│ if self._sslobj is None:
1164│ raise ValueError("Read on closed or unwrapped SSL socket.")
1165│ try:
1166│ if buffer is not None:
→ 1167│ return self._sslobj.read(len, buffer)
1168│ else:
1169│ return self._sslobj.read(len)
1170│ except SSLError as x:
1171│ if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:
The following error occurred when trying to handle this error:
ReadTimeoutError
HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
at ~/anaconda3/lib/python3.11/site-packages/urllib3/response.py:753 in _error_catcher
749│
750│ except SocketTimeout as e:
751│ # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
752│ # there is yet no clean way to get at it from this context.
→ 753│ raise ReadTimeoutError(self._pool, None, "Read timed out.") from e # type: ignore[arg-type]
754│
755│ except BaseSSLError as e:
756│ # FIXME: Is there a better way to differentiate between SSLErrors?
757│ if "read operation timed out" not in str(e):
The following error occurred when trying to handle this error:
ConnectionError
HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
at ~/anaconda3/lib/python3.11/site-packages/requests/models.py:826 in generate
822│ raise ChunkedEncodingError(e)
823│ except DecodeError as e:
824│ raise ContentDecodingError(e)
825│ except ReadTimeoutError as e:
→ 826│ raise ConnectionError(e)
827│ except SSLError as e:
828│ raise RequestsSSLError(e)
829│ else:
830│ # Standard file-like object.
Cannot install click.
- Installing itsdangerous (2.2.0)
- Installing jinja2 (3.1.4)
- Installing werkzeug (3.1.3)
You can see Poetry lists all information and clearly tells you which third-party modules were newly added.
(Since the author is currently writing this on a train sleeper berth, network speed is quite poor, so please ignore the timeout error.)
At this point, pyproject.toml in the project has also changed:
[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.11"
flask = "^3.0.3"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
To clarify: after installing flask, pyproject.toml only adds the direct dependency field flask = "^3.0.3". Other transitive dependencies do not appear in the toml file.
This is a major advantage because it clearly distinguishes user-installed dependencies from dependencies installed as transitive requirements.
poetry.lock and update order
Besides updating pyproject.toml, the project will also gain a new file called poetry.lock, which is effectively Poetry's equivalent of pip's requirements.txt, recording all installed modules and versions in detail.
When you run poetry add, Poetry automatically completes these three steps in order:
Update pyproject.toml.
Update poetry.lock according to pyproject.toml.
Update the virtual environment according to poetry.lock.
From this, we can see that poetry.lock depends on pyproject.toml. But the two do not auto-sync by themselves; synchronization and updates happen only through specific commands, and poetry add is a typical example.
At this moment, the project structure is:
poetry-demo
├── poetry.lock
└── pyproject.toml
0 directories, 2 files
poetry lock: update poetry.lock
If you manually modify pyproject.toml (for example, changing a specific dependency version, which can happen, especially when resolving version conflicts by hand), then poetry.lock and pyproject.toml become out of sync. You must update and synchronize poetry.lock from the new pyproject.toml content using:
poetry lock
This ensures your manual changes are also reflected in poetry.lock. After all, if the virtual environment is rebuilt, dependencies are installed based on poetry.lock, not pyproject.toml.
Again:
poetry.lockis Poetry's equivalent ofrequirements.txt.
But pay special attention: poetry lock only updates poetry.lock; it does not install modules into the virtual environment.
Therefore, after running poetry lock, you must run poetry install to install modules. Otherwise, poetry.lock and the virtual environment can become inconsistent.
Add modules to dev-dependencies
Some modules, such as pytest and black, are only used in development and are not needed in production deployment.
Poetry lets you distinguish these two categories by installing such modules into the dev-dependencies section, making it easy to create an installation set that excludes development dependencies.
Using Black as an example, install it like this:
poetry add black --group dev
The difference appears in pyproject.toml:
[tool.poetry]
name = "poetry-test"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.11"
flask = "^3.0.3"
[tool.poetry.group.dev.dependencies]
black = "^24.10.0"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
You can see black is listed in a different section: tool.poetry.dev-dependencies.
Update modules with Poetry
This one is simple: use poetry update.
poetry update
The command above updates all modules that can be updated. You can also specify particular modules, for example:
poetry update requests toml
List all modules
Similar to pip list, here you should use poetry show.
This list does not come from the virtual environment (unlike pip), but from poetry.lock.
You might wonder: does it matter whether it comes from poetry.lock or the virtual environment? Shouldn't they be identical?
Yes, in theory. But sometimes they differ. For example, if you install modules with pip install, those installs will not be recorded in poetry.lock, so poetry show will not display them.
Display dependency hierarchy as a tree
poetry show --tree
This makes the relationships and hierarchy between primary modules and their dependencies clear at a glance.
And nicely, you can also show the dependency tree for a specific module only, taking flask as an example:
╭─ ~/coding/poetry_test 1 ↵ 77% 22:11 11.11
╰ poetry show flask --tree
flask 3.0.3 A simple framework for building complex web applications.
├── blinker >=1.6.2
├── click >=8.1.3
│ └── colorama *
├── itsdangerous >=2.1.2
├── jinja2 >=3.1.2
│ └── markupsafe >=2.0
└── werkzeug >=3.0.0
└── markupsafe >=2.1.1
Remove modules with Poetry
Use poetry remove. Like poetry add, you can add the -D flag to remove modules from the development group.
Dependency resolution during removal is one of the major places where Poetry is far superior to pip, because pip simply does not provide this capability. This is also a key reason I suggested switching to Poetry: to remove modules and dependencies cleanly.
As mentioned earlier, pip uninstall only removes the module you specify and does not remove dependency modules together with it.
This is based on safety concerns: since pip has no dependency-resolution capability, blindly removing all dependencies installed alongside a module could cause serious breakage for other modules.
So in pip workflows, we rarely remove modules that are no longer needed. Dependency relationships are complex, and removing modules can create many side effects, which is too troublesome.
poetry remove flask
Export requirements.txt from a Poetry virtual environment
In theory, after fully switching to Poetry, a project no longer needs requirements.txt, because its role is fully replaced by poetry.lock.
But in reality, you may still need it, and even want it to update along with poetry.lock.
Inside a Poetry virtual environment, you can use the familiar command pip freeze > requirements.txt.
pip freeze > requirements.txt
Or:
poetry export -f requirements.txt -o requirements.txt --without-hashes
If you want to export modules from [tool.poetry.dev-dependencies], add the --dev parameter:
poetry export -f requirements.txt -o requirements.txt --without-hashes --dev
Common Poetry command list
Commonly used Poetry commands mainly include:
- poetry add
- poetry remove
- poetry export
- poetry env use
- poetry shell
- poetry show
- poetry init
- poetry install
Change Poetry mirror source
Switch to the Tsinghua mirror:
poetry source add tsinghua https://pypi.tuna.tsinghua.edu.cn/simple
Or append the following at the end of pyproject.toml:
[[tool.poetry.source]]
name = "tsinghua"
default = true
url = "https://pypi.tuna.tsinghua.edu.cn/simple"