diff --git a/10_manage_python_project.ipynb b/10_manage_python_project.ipynb index 8a426bbc..75fc6173 100644 --- a/10_manage_python_project.ipynb +++ b/10_manage_python_project.ipynb @@ -27,8 +27,7 @@ " - [Installation and running](#Installation-and-running)\n", " - [Configuration](#Configuration)\n", " - [Supported hooks](#Supported-hooks)\n", - " - [Auto-formatters](#Auto-formatters)\n", - " - [Linters](#Linters)\n", + " - [Auto-formatters and Linters](#Auto-formatters-and-Linters)\n", " - [Type checks](#Type-checks)\n", " - [pre-commit.ci](#pre-commit.ci)\n", " - [Exercise on pre-commit](#Exercise-on-pre-commit)\n", @@ -38,7 +37,8 @@ " - [Write your first unit test](#Write-your-first-unit-test)\n", " - [Run the tests](#Run-the-tests)\n", " - [Test driven development (TDD)](#Test-driven-development-(TDD))\n", - " - [Code coverage](#Code-coverage)\n", + " - [Test coverage](#Test-coverage)\n", + " - [Exercise on testing](#Exercise-on-testing)\n", " - [Documentation](#Documentation)\n", " - [Why document?](#Why-document?)\n", " - [How document?](#How-document?)\n", @@ -50,25 +50,25 @@ " - [Exercise on README.md](#Exercise-on-README.md)\n", " - [GitHub wiki](#GitHub-wiki)\n", " - [Read the Docs](#Read-the-Docs)\n", - " - [Configuration](#Configuration)\n", + " - [Configuring Read the Docs](#Configuring-Read-the-Docs)\n", " - [Writing the documentation](#Writing-the-documentation)\n", " - [Building the documentation](#Building-the-documentation)\n", " - [Deploying on Read the Docs](#Deploying-on-Read-the-Docs)\n", " - [Releasing package](#Releasing-package)\n", " - [Why release?](#Why-release?)\n", - " - [Update the version](#Update-the-version)\n", + " - [Update the version on GitHub](#Update-the-version-on-GitHub)\n", " - [Python Package Index (PyPI)](#Python-Package-Index-(PyPI))\n", " - [Changelog](#Changelog)\n", " - [CHANGELOG.md](#CHANGELOG.md)\n", " - [GitHub releases](#GitHub-releases)\n", " - [Package managers](#Package-managers)\n", " - [Automate things](#Automate-things)\n", - " - [Testing](#Testing)\n", + " - [Automate testing](#Automate-testing)\n", " - [Exercise on testing automation](#Exercise-on-testing-automation)\n", - " - [Version update](#Version-update)\n", + " - [Automate version update](#Automate-version-update)\n", " - [Exercise on version update automation](#Exercise-on-version-update-automation)\n", - " - [Releasing](#Releasing)\n", - " - [Exercises on releasing automation](#Exercises-on-releasing-automation)" + " - [Automate releasing](#Automate-releasing)\n", + " - [Exercises on release automation](#Exercises-on-release-automation)" ] }, { @@ -78,10 +78,10 @@ "source": [ "# References\n", "\n", - "- [Python Packaging User Guide](https://packaging.python.org/)\n", - "- [PEP 518](https://www.python.org/dev/peps/pep-0518/) – Specifying Minimum Build System Requirements for Python Projects\n", + "- [Python Packaging User Guide](https://packaging.python.org/).\n", + "- [PEP 518](https://www.python.org/dev/peps/pep-0518/) – Specifying Minimum Build System Requirements for Python Projects.\n", "- [Pre-commit](https://pre-commit.com/) and its documentation.\n", - "- [Pytest documentation.](https://docs.pytest.org/en/latest/)\n", + "- [Pytest documentation.](https://docs.pytest.org/en/latest/).\n", "- [Python Packages book](https://py-pkgs.org/) by Tomas Beuzen and Tiffany Timbers." ] }, @@ -93,7 +93,6 @@ "# Introduction\n", "\n", "In the basic tutorial we have discussed the package structure.\n", - "\n", "We came up with the following setup:\n", "\n", "```bash\n", @@ -136,8 +135,12 @@ "## Preparatory exercise\n", "\n", "1. Decide which package you want to use for the following exercises, yours or the [mypackage](https://github.com/empa-scientific-it/mypackage).\n", - "2. If you use the `mypackage` fork it to your own GitHub account by clicking on the `Fork` button in the top right corner.\n", - "3. Clone the package to your local machine: `git clone `.\n", + "1. If you use the `mypackage` fork it to your own GitHub account by clicking on the `Fork` button in the top right corner.\n", + "1. Clone the fork to your local machine: `git clone `.\n", + " The URL can be found by clicking on the `` dropdown and selecting __HTTPS__ tab.\n", + "1. [Create an access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token#creating-a-personal-access-token-classic) for your account that will be used instead of your password.\n", + " You should enable `workflow`, `write:packages`, `delete:packages`.\n", + " Make sure to store the token in some secure location.\n", "\n", "
\n", "In the following, we will always assume that mypackage is the package you are working on (either yours or the one you have forked).\n", @@ -167,7 +170,7 @@ "The `project` section contains information about the Python package.\n", "The `name` and `version` are required fields, the other fields are optional.\n", "However, it is recommended to provide as much information as possible about your package.\n", - "Let's take a look at the [`pyproject.toml` file](https://github.com/numpy/numpy/blob/main/pyproject.toml) of the `numpy` package:\n", + "Let's take a look at certain parts of the [`pyproject.toml` file](https://github.com/numpy/numpy/blob/main/pyproject.toml) of the `numpy` package:\n", "\n", "```toml\n", "[project]\n", @@ -271,14 +274,13 @@ "```toml\n", "[project.optional-dependencies]\n", "dev = [\n", - " \"bumpver==2023.1129\",\n", - " \"pre-commit==3.5.0\",\n", - " \"pytest==7.4.3\",\n", - " \"pytest-cov==2.6.1\",\n", + " \"bumpver\",\n", + " \"pytest\",\n", + " \"pytest-cov\",\n", "]\n", "docs = [\n", - " \"sphinx==4.2.0\",\n", - " \"sphinx-rtd-theme==1.0.0\",\n", + " \"sphinx\",\n", + " \"sphinx-rtd-theme\",\n", "]\n", "```" ] @@ -292,8 +294,9 @@ "\n", "1. Add an MIT license to `mypackage` and refer to it in the `pyproject.toml` file.\n", "2. Specify the dependencies of `mypackage` in the `pyproject.toml` file.\n", - "3. Specify optional development dependencies in the `pyproject.toml` file.\n", - "4. Try to re-install `mypackage` with `pip` and see if it works and all the optional dependencies are installed." + " For instance, you can make your package depend on `numpy`.\n", + "4. Specify optional development dependencies in the `pyproject.toml` file.\n", + "5. Try to re-install `mypackage` with `pip` and see if it works and all the optional dependencies are installed." ] }, { @@ -304,7 +307,6 @@ "# Pre-commit\n", "\n", "[Pre-commit](https://pre-commit.com/) is a framework for managing and maintaining multi-language pre-commit hooks.\n", - "\n", "Pre-commit hooks are small scripts that are executed before a commit is created.\n", "These scripts can be used to check the code for formatting errors, syntax errors, and other issues.\n", "If a pre-commit hook fails, the commit is aborted." @@ -359,7 +361,7 @@ "---\n", "repos:\n", " - repo: https://github.com/pre-commit/pre-commit-hooks\n", - " rev: v4.4.0\n", + " rev: v5.0.0\n", " hooks:\n", " - id: end-of-file-fixer\n", " - id: trailing-whitespace\n", @@ -388,101 +390,31 @@ "id": "14", "metadata": {}, "source": [ - "### Auto-formatters\n", + "### Auto-formatters and Linters\n", "\n", "Auto-formatters are tools that automatically format the code.\n", "They take away from you the burden of ensuring the consistency of the code style.\n", - "We recommend to use the following formatters:\n", - "\n", - "- [isort](https://pycqa.github.io/isort/) for sorting imports.\n", - "- [autoflake](https://github.com/PyCQA/autoflake) for removing unused imports and variables.\n", - "- [black](https://black.readthedocs.io/en/stable/) for an uncompromising code formatting (it really is!).\n", - "\n", - "Here is the configuration of these formatters:\n", - "\n", - "```yaml\n", - " - repo: https://github.com/pycqa/isort\n", - " rev: 5.12.0\n", - " hooks:\n", - " - id: isort\n", - " args: [--profile, black, --filter-files]\n", - " - repo: https://github.com/PyCQA/autoflake\n", - " rev: v2.1.1\n", - " hooks:\n", - " - id: autoflake\n", - " - repo: https://github.com/psf/black\n", - " rev: 23.3.0\n", - " hooks:\n", - " - id: black\n", - " language_version: python3\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "15", - "metadata": {}, - "source": [ - "### Linters\n", "Linters are tools that check the code for errors and potential issues.\n", - "We recommend to use [flake8](https://flake8.pycqa.org/en/latest/) as a linter.\n", "\n", + "For both formatting and lintering purposes we recommend to use [ruff](https://github.com/astral-sh/ruff-pre-commit):\n", "\n", "```yaml\n", - " - repo: https://github.com/PyCQA/flake8\n", - " rev: 6.0.0\n", + " - repo: https://github.com/astral-sh/ruff-pre-commit\n", + " # Ruff version.\n", + " rev: v0.11.8\n", " hooks:\n", - " - id: flake8\n", + " # Run the linter.\n", + " - id: ruff\n", + " # Run the formatter.\n", + " - id: ruff-format\n", "```\n", "\n", - "Note, however, that with the default configuration `flake8` conflicts with `black`.\n", - "To avoid this conflict, we need to configure `flake8` to ignore the formatting issues.\n", - "Here is the configuration of `flake8` to be put to `.flake8` file in the root of the repository:\n", - "\n", - "```\n", - "[flake8]\n", - "extend-ignore =\n", - " E501\n", - " W503\n", - " E203\n", - "```\n", - "\n", - "The vanilla `flake8` provides only basic checks.\n", - "However, its functionality can be extended with plugins.\n", - "For example, we recommend to use the following plugins:\n", - "\n", - "- [flake8-bugbear](https://github.com/PyCQA/flake8-bugbear) for finding common bugs and design problems.\n", - "- [flake8-builtins](https://pypi.org/project/flake8-builtins/) for verifying that builtins are not used as variable names.\n", - "- [flake8-comprehensions](https://pypi.org/project/flake8-comprehensions/) for writing better and consistent comprehensions.\n", - "- [flake8-debugger](https://pypi.org/project/flake8-debugger/) for checking that there are no forgotten [breakpoints](https://docs.python.org/3/library/functions.html#breakpoint).\n", - "- [flake8-eradicate](https://pypi.org/project/flake8-eradicate/) to avoid “dead” or commented code.\n", - "- [flake8-logging-format](https://github.com/globality-corp/flake8-logging-format) for consistent logging.\n", - "- [pep8-naming](https://pypi.org/project/pep8-naming/) for verifying that PEP8 naming conventions are followed.\n", - "- [tryceratops](https://pypi.org/project/tryceratops/) to avoid exception anti-patterns.\n", - "\n", - "Here is the configuration of these plugins:\n", - "\n", - "```yaml\n", - " - repo: https://github.com/PyCQA/flake8\n", - " rev: 6.0.0\n", - " hooks:\n", - " - id: flake8\n", - " args: [--count, --show-source, --statistics]\n", - " additional_dependencies:\n", - " - flake8-bugbear\n", - " - flake8-builtins\n", - " - flake8-comprehensions\n", - " - flake8-debugger\n", - " - flake8-eradicate\n", - " - flake8-logging-format\n", - " - pep8-naming\n", - " - tryceratops\n", - "```" + "Ruff is fast and very simple to setup." ] }, { "cell_type": "markdown", - "id": "16", + "id": "15", "metadata": {}, "source": [ "### Type checks\n", @@ -498,7 +430,7 @@ }, { "cell_type": "markdown", - "id": "17", + "id": "16", "metadata": {}, "source": [ "\n", @@ -524,7 +456,7 @@ }, { "cell_type": "markdown", - "id": "18", + "id": "17", "metadata": {}, "source": [ "## Exercise on pre-commit\n", @@ -532,19 +464,18 @@ "For the exercise, we will use the `mypackage` package that we have created in the basic tutorial.\n", "\n", "1. Install pre-commit as a development dependency.\n", - "2. Enable basic pre-commit hooks (`end-of-file-fixer`, `trailing-whitespace`) and apply them to your repository.\n", - "3. Enable `isort` pre-commit hook and apply it to your repository.\n", - "4. Enable `autoflake` pre-commit hook and apply it to your repository.\n", - "5. Enable `black` pre-commit hook and apply it to your repository.\n", - "6. Enable `flake8` pre-commit hook and apply it to your repository.\n", + "1. Enable basic pre-commit hooks (`end-of-file-fixer`, `trailing-whitespace`, `check-yaml`, `check-added-large-files`) and apply them to your repository.\n", + "1. Install `pre-commit` for your local repository by running: `pre-commit install`.\n", + " If this step is successful, the pre-commit will run at every attempt to make a commit to the repo.\n", + "1. Enable ruff's pre-commit hooks (`ruff-format` and `ruff`) and apply them on your repo.\n", "What issues does it find?\n", "Can you fix them?\n", - "7. Enable pre-commit.ci for your repository." + "1. Enable pre-commit.ci for your repository." ] }, { "cell_type": "markdown", - "id": "19", + "id": "18", "metadata": {}, "source": [ "# Testing\n", @@ -559,7 +490,7 @@ }, { "cell_type": "markdown", - "id": "20", + "id": "19", "metadata": {}, "source": [ "## Types of tests\n", @@ -589,7 +520,7 @@ }, { "cell_type": "markdown", - "id": "21", + "id": "20", "metadata": {}, "source": [ "## Installation\n", @@ -605,7 +536,7 @@ }, { "cell_type": "markdown", - "id": "22", + "id": "21", "metadata": {}, "source": [ "## Write your first unit test\n", @@ -640,7 +571,7 @@ }, { "cell_type": "markdown", - "id": "23", + "id": "22", "metadata": {}, "source": [ "## Run the tests\n", @@ -657,7 +588,7 @@ }, { "cell_type": "markdown", - "id": "24", + "id": "23", "metadata": {}, "source": [ "## Test driven development (TDD)\n", @@ -681,10 +612,10 @@ }, { "cell_type": "markdown", - "id": "25", + "id": "24", "metadata": {}, "source": [ - "## Code coverage\n", + "## Test coverage\n", "\n", "Test coverage is defined as the percentage of code that is being executed during the tests with respect to the total amount of loaded code.\n", "To make this clear, let's consider the following example.\n", @@ -756,6 +687,16 @@ "
" ] }, + { + "cell_type": "markdown", + "id": "25", + "metadata": {}, + "source": [ + "## Exercise on testing\n", + "1. Test the `length` method of the `Line` class.\n", + " The `Line` object should receive two `Points` and check that the distance between them is correctly computed." + ] + }, { "cell_type": "markdown", "id": "26", @@ -810,7 +751,8 @@ "Comments can be helpful in the following cases:\n", "\n", "1. To describe what you plan to do.\n", - "It is often helpful to write a comment before writing the code to keep track of what you are doing.\n", + " It is often helpful to write a comment before writing the code to keep track of what you are doing.\n", + "\n", "```python\n", "# Step 1: Import the data.\n", "# Step 2: Clean the data.\n", @@ -819,23 +761,26 @@ "```\n", "\n", "2. To describe what the code does.\n", - "Often, even with your best effort, the code is not self-explanatory.\n", - "In this case, it is better to write a comment that explains what the code does.\n", + " Often, even with your best effort, the code is not self-explanatory.\n", + " In this case, it is better to write a comment that explains what the code does.\n", + "\n", "```python\n", + "\n", "# This part of the code is responsible for importing the data.\n", "with open(\"data.csv\", \"r\") as f:\n", " data = f.read()\n", "```\n", "\n", "3. Describe or reference an algorithm.\n", + "\n", "```python\n", "# This part of the code implements the Newton-Raphson method.\n", "...very-complex-code...\n", "```\n", "\n", "4. Tagging.\n", - "```python\n", "\n", + "```python\n", "# TODO: Revise this part of the code in the future.\n", "```\n", "\n", @@ -1149,7 +1094,9 @@ "source": [ "## Exercise on README.md\n", "\n", - "- Add a README file to your package and make sure it is properly displayed on GitHub." + "1. (optional) Add a README file to your package and make sure it is properly displayed on GitHub.\n", + "2. Add a small section in your readme and push it to the repository.\n", + " Check if it is correctly displayed." ] }, { @@ -1159,22 +1106,20 @@ "source": [ "## GitHub wiki\n", "\n", - "GitHub provides a free wiki for an open source project.\n", + "GitHub provides a free wiki for an open-source project.\n", "It can be found under the `Wiki` tab of the repository.\n", - "\n", - "GitHub Wiki is usually a great place to kickstart a more serious documentation.\n", + "GitHub Wiki is usually a great place to kickstart more serious documentation.\n", "It is easy to use and it is easy to collaborate on it.\n", "The only basic knowledge it requires is the knowledge of Markdown, which you have already acquired.\n", "\n", - "To start using the GitHub Wiki, simply click on the `Wiki` tab and then on the `Create the first page` button.\n", + "To start using the GitHub Wiki, simply click on the `Wiki` tab and then on the `Create the first-page` button.\n", "Then, you can start writing the documentation in Markdown.\n", "Keep in mind that all revisions of the file are stored in the repository.\n", - "\n", "With time the GitHub Wiki might become too small for your documentation.\n", "The Wiki, however, will still be a good place to write on the internal development policies, provide quick how-to guides, etc.\n", "\n", "All-in-all consider wiki more for the live documentation which is updated frequently and is not too big.\n", - "The more static (end established) documentation should be moved to a separate website, which we will discuss in the next section." + "The more static (end-established) documentation should be moved to a separate website, which we will discuss in the next section." ] }, { @@ -1205,9 +1150,9 @@ "id": "50", "metadata": {}, "source": [ - "### Configuration\n", + "### Configuring Read the Docs\n", "\n", - "To configure MkDocs, we need to create a `mkdocs.yml` file in the root of the repository.\n", + "To configure MkDocs, we must create a `mkdocs.yml` file in the repository's root.\n", "It can be as simple as:\n", "\n", "```yaml\n", @@ -1312,6 +1257,7 @@ "\n", "When developing a package, you usually put your changes to the `main` branch of your repository.\n", "So, in principle, the users can install the package directly from the repository mentioning the branch or the commit hash:\n", + "\n", "```bash\n", "pip install git+https://github.com/numpy/numpy@main # Install the latest version from the main branch.\n", "pip install git+hhttps://github.com/numpy/numpy@198f339c859d87e68dfe92c2a79429ac6e5074c1 # Install the version using the commit sha.\n", @@ -1336,7 +1282,7 @@ "id": "56", "metadata": {}, "source": [ - "## Update the version\n", + "## Update the version on GitHub\n", "\n", "The first thing we need to do is to update the version of the package.\n", "The version is usually stored in the `pyproject.toml` file as well as in the `__init__.py` file of the package.\n", @@ -1532,7 +1478,7 @@ "id": "63", "metadata": {}, "source": [ - "## Testing\n", + "## Automate testing\n", "\n", "Testing can be automated using [GitHub Actions](https://docs.github.com/en/actions).\n", "GitHub Actions are workflows that are triggered by certain events.\n", @@ -1543,6 +1489,7 @@ "Then, we need to create a `ci.yml` file with the following content:\n", "\n", "```yaml\n", + "---\n", "name: Continuous Integration\n", "\n", "on: push\n", @@ -1556,7 +1503,7 @@ " steps:\n", "\n", " - name: Check out repository\n", - " uses: actions/checkout@v3\n", + " uses: actions/checkout@v4\n", "\n", " - name: Set up Python 3.11\n", " uses: actions/setup-python@v4\n", @@ -1565,16 +1512,15 @@ "\n", " - name: Install python dependencies\n", " run: |\n", - " pip install --upgrade pip\n", - " pip install -e .[dev]\n", + " python3 -m pip install -e .[dev]\n", "\n", " - name: Run test suite\n", - " run: pytest\n", + " run: pytest --cov\n", "```\n", "\n", "In the example above, we have created a workflow that is triggered by a push event.\n", "The workflow runs on the latest version of Ubuntu.\n", - "Then, it checks out the repository, sets up Python 3.11, installs the package with the development dependencies, and runs the test suite.\n", + "Then, it checks out the repository, sets up `Python 3.11`, installs the package with the development dependencies, and runs the test suite.\n", "\n", "If pytest finds any errors, the workflow will fail." ] @@ -1586,8 +1532,8 @@ "source": [ "## Exercise on testing automation\n", "\n", - "- Create a GitHub Action workflow that runs the tests on every push.\n", - "- Check that the workflow is triggered and all the tests pass." + "1. Create a GitHub Action workflow that runs the tests on every push.\n", + "1. Check that the workflow is triggered and all the tests pass." ] }, { @@ -1595,7 +1541,7 @@ "id": "65", "metadata": {}, "source": [ - "## Version update\n", + "## Automate version update\n", "\n", "To automate the version update we can use [bumpver](https://github.com/mbarkhau/bumpver) tool.\n", "To install it, run the following command:\n", @@ -1610,7 +1556,7 @@ "\n", "```toml\n", "[tool.bumpver]\n", - "current_version = \"v0.2.3\"\n", + "current_version = \"v0.1.0\"\n", "version_pattern = \"vMAJOR.MINOR.PATCH[PYTAGNUM]\"\n", "commit_message = \"Bump version {old_version} -> {new_version}.\"\n", "commit = true\n", @@ -1651,17 +1597,19 @@ "source": [ "## Exercise on version update automation\n", "\n", - "- Configure `bumpver` in your package.\n", - "- Update the version of your package using `bumpver`.\n", - "- Go to GitHub and check that the version is updated." + "1. Configure `bumpver` in your package.\n", + "2. Update the version of your package using `bumpver`.\n", + "3. Go to GitHub and check that the version is updated." ] }, { "cell_type": "markdown", "id": "67", - "metadata": {}, + "metadata": { + "jp-MarkdownHeadingCollapsed": true + }, "source": [ - "## Releasing\n", + "## Automate releasing\n", "\n", "To automate the release process we will use [GitHub Actions](https://docs.github.com/en/actions).\n", "We already did big part of the work in the section above, when talking about version update.\n", @@ -1693,7 +1641,7 @@ "\n", " steps:\n", " - name: Checkout\n", - " uses: actions/checkout@v3\n", + " uses: actions/checkout@v4\n", "```\n", "\n", "In the above, we trigger the workflow only when the tag starts with `v`.\n", @@ -1708,7 +1656,7 @@ " python-version: 3.11\n", "\n", " - name: Install flit\n", - " run: pip install flit~=3.4\n", + " run: pip install flit~=3.12\n", "```\n", "\n", "Once the dependencies are installed, we can build and publish the package:\n", @@ -1731,11 +1679,11 @@ "Finally, we want to add a release to GitHub.\n", "```yaml\n", " - name: Create release\n", - " uses: softprops/action-gh-release@v0.1.14\n", + " uses: softprops/action-gh-release@v2.2.2\n", " with:\n", - " files: |\n", - " dist/*\n", - " generate_release_notes: true\n", + " files: |\n", + " dist/*\n", + " generate_release_notes: true\n", "```\n", "\n", "Once everything is done, and the workflow is configured, we can commit and push the changes to GitHub.\n", @@ -1755,21 +1703,17 @@ "id": "68", "metadata": {}, "source": [ - "## Exercises on releasing automation\n", - "\n", - "- Configure the release workflow in your package.\n", - "You can use all steps from the example above except from the `Build and publish` step.\n", - "This step should be replaced with the following one:\n", + "## Exercises on release automation\n", "\n", + "1. Configure the release workflow in your package.\n", + " You can use all steps from the example above except from the `Build and publish` step.\n", + " This step should be replaced with the following one:\n", "```yaml\n", " - name: Build and publish\n", " run: flit build\n", "```\n", - "- Update the version of your package using `bumpver`.\n", - "\n", - "**Optional:**\n", - "\n", - "- Only if you are working with your own package, you can try to release it on PyPI." + "2. Update the version of your package using `bumpver`.\n", + "3. (optional) Only if you are working with your own package, you can try to release it on PyPI." ] } ], @@ -1789,7 +1733,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.1" + "version": "3.10.10" } }, "nbformat": 4,