Python Projects

Purpose: Notes on good ideas for organizing and developing a Python project.

Project layout

Use a src-layout package layout.

Virtual environment

Use venv for virtual environment.

conda can be used but are best for scientific computing.

Editable install

Install the package in development mode / editable mode. This enables us to edit the package on the fly without the need to reinstall it.

Jupyter magic

Use the autoreload magic command when making prototypes in a notebook. And use Cython magic commands when prototyping Cython code.

Testing

Use pytest and put the tests in a tests/ folder in the same directory.

Often used tools for testing in Python are pytest (most common and the best) and unittest.

Documentation

Use sphinx for documentation. mkdocs seems like the new kid on the block but sphinx is older and more used.

Write doctests when writing documentation. Doctest can be run with pytest.

Type Checking

Use pyright for type checking.

Code style formatter

Often used are flake8 or for automatic code formatting people often use black.

Pre-commit hooks

pre-commit is often used to run a series of checks before making commits to git e.g. black code formatting, pyright, isort etc.

Logging

The logging module is a nice debugging tool in Python, use it.

Cookiecutter

cookiecutter can be used to create a template Python project.

PEP 8

Use PEP 8. And think about the public and non-public API. Default to non-public attributes, API etc.

Branching Model

Use Github flow. See this discussion.

CI/CD Pipelines

Use Githubs workflows for CI/CD pipelines.

Publish Package

Use the pypi package index.

You can create your own package index on premise (conda is pretty easy). Other solutions for an on premise package index exists like fury.

Versioning

Use semantic versioning e.g. MAJOR.MINOR.PATCH and git tag each version and push to the remote.

versioneer seems like a cool project to automate parts of the versioning process.