Manage Python Environment on macOS/Linux

Since Python has been deeply integrated into systems, it is often the case that the version of Python in your system is not the one you want, or it is managed by the system package manager. If this is the case, you cannot install packages using pip directly.

For example, when setting up Coqtail with Neovim , you will encounter the following error:

Coqtail requires Python 3.6 or later.
See https://github.com/whonore/Coqtail/blob/main/README.md#python-2-support.
Press ENTER or type command to continue

Run the following command to check the issue:

$ nvim +checkhealth

...
Python 3 provider (optional) ~
- Disabled (loaded_python3_provider=0).
...

To resolve the issue, you need to install pynvim. However, if you try to install it using pip directly, you will receive the following error:

$ pip install pynvim
× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-brew-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.

    If you wish to install a non-brew packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

This post provides a brief overview of how to manage Python environments on macOS and Linux using pyenv and pipenv. pyenv allows you to easily switch between multiple versions of Python, while pipenv helps you create isolated environments for each Python project and keeps track of dependencies in a Pipfile and Pipfile.lock.

0 - Install Homebrew (macOS)

For macOS users, follow the instruction to install Homebrew. Homebrew is a package manager for macOS that makes it easy to install and manage software packages on macOS.

1 - Setup pyenv and pipenv

Use your package manager to install pyenv and pipenv, as well as some dependencies required to build Python versions.

• Homebrew (macOS)
• APT (Debian-based Linux)
• Pacman (Arch-based Linux)

• Homebrew
• apt
• pacman

• brew install pyenv pipenv openssl readline sqlite3 xz zlib tcl-tk@8 libb2
  

• sudo apt update; sudo apt install -y pyenv pipenv make build-essential libssl-dev zlib1g-dev \
  libbz2-dev libreadline-dev libsqlite3-dev curl git \
  libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
  

• sudo pacman -Syu --needed pyenv pipenv base-devel openssl zlib xz tk
  

Then, to set up your shell environment for pyenv, use the following command to find the shell you are using:

echo $SHELL

Based on the output, run the following command to add them to your shell configuration file:

• zsh
• bash

• PYENV_SETUP='export PYENV_ROOT="$HOME/.pyenv"
  [[ -d $PYENV_ROOT/bin ]] && export PATH=$PYENV_ROOT/bin:$HOME/.local/bin:$PATH"
  eval "$(pyenv init - bash)"'
  
  [ -f ~/.zshrc ] || touch ~/.zshrc
  echo "$PYENV_SETUP" >> ~/.zshrc
  

• PYENV_SETUP='export PYENV_ROOT="$HOME/.pyenv"
  [[ -d $PYENV_ROOT/bin ]] && export PATH=$PYENV_ROOT/bin:$HOME/.local/bin:$PATH"
  eval "$(pyenv init - bash)"'
  
  echo "$PYENV_SETUP" >> ~/.bashrc
  
  [ -f ~/.profile ] || touch ~/.profile
  echo "$PYENV_SETUP" >> ~/.profile
  
  [ -f ~/.bash_profile ] && echo "$PYENV_SETUP" >> ~/.bash_profile
  
  [ -f ~/.bash_login ] && echo "$PYENV_SETUP" >> ~/.bash_login
  

Then, either restart your terminal or run the following command to apply the changes:

• zsh
• bash

• source ~/.zshrc
  

• source ~/.bashrc
  

2 - Install Python versions using pyenv

Run the following command to check the available Python versions:

pyenv install --list

Pick the version you want to install, for example, 3.13.4, and run the following command:

pyenv install 3.12.9

You should be able to see the installed version when you run the following command:

$ pyenv versions
* system (set by /Users/vc/.pyenv/version)
  3.13.4
  3.12.9

To ensure that the entire system uses the correct version of Python, managed by the system package manager, set the global Python version to system:

pyenv global system

3 - Create a virtual environment

Change the current directory to Neovim’s configuration directory.

cd ~/.config/nvim

Next, use pyenv to specify the Python version you want to use for this project, and use pipenv to create a virtual environment with the Python version you just installed:

pyenv local 3.12.9
pipenv --install --python 3.12.9

You should now see the virtual environment that was created in the current directory:

$ ls -la
-rw-r--r-- 1  Jun 01 13:18 Pipfile
-rw-r--r-- 1  Jun 01 13:18 Pipfile.lock
-rw-r--r-- 1  Jun 01 13:17 .python-version

Pipfile and Pipfile.lock are used to keep track of the dependencies of your project, while .python-version is used by pyenv to specify the Python version to use for this directory. You can now install the required packages for Coqtail:

pipenv install pynvim
# Install specific version: pipenv install pynvim==0.5.0
# Install dev version: pipenv install pynvim --dev

After the installation, you should be able to see pynvim in Pipfile and Pipfile.lock:

$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pynvim = "*"

[dev-packages]

[requires]
python_version = "3.12"
python_full_version = "3.12.9"

To test if everything is working properly, enter the project’s virtual environment by running the following command:

pipenv shell

You should see the prompt changed to (nvim) indicate that you are in the virtual environment:

$ pipenv shell
Launching subshell in virtual environment...
 source /Users/vc/.local/share/virtualenvs/nvim-nBOHLvVP/bin/activate
nvim %  source /Users/vc/.local/share/virtualenvs/test_py-nBOHLvVP/bin/activate
(nvim) nvim %

Run the following command to check if pynvim is installed correctly:

(nvim) nvim % pip show pynvim
Name: pynvim
Version: 0.5.2
Summary: Python client for Neovim
Home-page: http://github.com/neovim/pynvim
Author: Neovim Authors
Author-email:
License: Apache
Location: /Users/vc/.local/share/virtualenvs/nvim-nBOHLvVP/lib/python3.12/site-packages
Requires: greenlet, msgpack
Required-by:

Run the command exit to exit the virtual environment.

4 - Configure Neovim to find the path of the virtual environment

If you run nvim +checkhealth again, you will see that the Python 3 provider is still disabled. This is because Neovim cannot find the path of the Python exectuable and virtual environment you created with pipenv. The location of virtual environment is usually located at ~/.local/share/virtualenvs/<project_name>/bin/python, or you can enter the virtual environment and use which python to find out. Run the following to add the path to the Neovim init file, usually located at ~/.config/nvim/init.vim:

echo "let g:python3_host_prog = expand('~/.local/share/virtualenvs/nvim-nBOHLvVP/bin/python')" >> ~/.config/nvim/init.vim

Then, run nvim +checkhealth again, you should see the Python 3 provider is enabled:

$ nvim +checkhealth
...
Python 3 provider (optional) ~
- pyenv: Path: /opt/homebrew/Cellar/pyenv/2.6.1/libexec/pyenv
- pyenv: Root: /Users/vc/.pyenv
- Using: g:python3_host_prog = "/Users/vc/.local/share/virtualenvs/nvim-nBOHLvVP/bin/python3"
- Executable: /Users/vc/.local/share/virtualenvs/nvim-nBOHLvVP/bin/python3
- Python version: 3.12.9
- pynvim version: 0.5.2
- ✅ OK Latest pynvim is installed.
...

5 - Additional tips

Although this post is focused on Neovim, the same method can be applied to other Python projects. Suppose you save your Python project, including the .python-version, Pipfile, and Pipfile.lock files, to a remote repository. If you want to duplicate the environment on another machine, install pyenv and pipenv, and simply run the following commands:

git clone <repository>
cd <repository>
pipenv sync # Sync the virtual environment with the Pipfile.lock

This will generate a virtual environment that mirrors the original project’s Python version and dependencies. Some useful commands for managing Python environments with pipenv:

pipenv uninstall <package> # Uninstall a package
pipenv update # Update all packages
pipenv update <package> # Update a specific package
pipenv graph # Show the dependency graph of the project
pipenv sync # Sync the virtual environment with the Pipfile.lock