Devops

• DevOps
  • Poetry (package management system)
  • Pyenv
  • Precommit hooks
  • Linters (flake8, bandit, etc)
• Python underpinnings
  • Typehints and mypy
    • How mypy works
    • Basic types (dict/list/sequence/mapping/tuple/none/optional)
    • Generic types
• tox, nox (run tests, lints, e.g. on diff python versions)
• Pytest
• Semantic versioning (semver.org)
• Imperative vs declarative languages
• Pipx
• CICD
• Github actions
• Docker
  • Build docker image locally and see what happens
• Unit tests
  • Python Unit tests library
  • Test driven development
  • Skdh tests
• Google style
  • https://google.github.io/styleguide/pyguide.html
  • https://google.github.io/styleguide/pyguide.html#316-naming
• https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
• Git
  • https://www.atlassian.com/git/tutorials/what-is-version-control

Poetry

https://python-poetry.org/docs/

• For managing packages and dependencies: It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

• Here’s a good getting started with poetry video

  • Installation
  • Add to PATH
  • Create python project with poetry
  • Activate poetry virtual env (creates virtual envs by default, poetry env management)

• Some commands

  • poetry new my_project creates directory with starter files
  • poetry add pandas
  • poetry remove pandas
  • poetry install install all the packages, creates the poetry.lock file the 1st time it’s run or installs the versions defined in poetry.lock if it already exists (not the latest versions)
  • poetry show --tree

• Dependencies can be organized into groups, e.g. dev, test etc.

  • e.g. poetry add pytest --group test adds pytest to the test group of dependencies

• Then you can only install specific dependencies groups:

  • poetry install --without test,docs
  • poetry install --with docs
  • poetry install --only main only install runtime dependencies

• all poetry commands are documented here: https://python-poetry.org/docs/cli/

• pre-commit hooks: https://python-poetry.org/docs/pre-commit-hooks/

Pre-commit

https://pre-commit.com/

• Run on every commit, identify style related issues (trailing whitespace, missing semi-colons…) before the code is committed. Free up the reviewer focus on architecture.
  • A framework for managing and maintaining multi-language pre-commit hooks
  • This is a good video on getting started with pre-commit
• Quick start:
  • install
  • create a .pre-commit-config.yaml. This describes the hooks and repositories to install.
  • pre-commit install. This installs pre-commit into your git hooks. After this, pre-commit will run on every commit.
    • Every time you clone a project using pre-commit running pre-commit install should always be the first thing you do.
  • run against all pre-commit hooks on all files using: pre-commit run --all-files
    • pre-commit run black to run only black
    • this causes changes to files, so you need to git add the files again, and then re-commit. If you don’t like the changes made, git restore to go back.
• Structure and contents of the .pre-commit-config.yaml file:
  • https://pre-commit.com/#plugins
• All pre-commit commands https://pre-commit.com/#cli
• Other useful things you can do:
  • temporarily disable hooks
  • run hooks on different stages, e.g. on commit, or on merge, or on push
  • https://pre-commit.com/#advanced
• tons of hooks have been written:
  • e.g. trailing-whitespace, detect-aws-credentials, check-added-large-files
  • you add these to your .pre-commit-config.yaml file under the pre-commit repo, like so

  - repo: 'https://github.com/pre-commit/pre-commit-hooks'
    rev: v4.4.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: detect-aws-credentials
      - id: check-added-large-files

  • https://pre-commit.com/hooks.html

Git hooks

• pre-commit works because git has the notion of hooks
  • hooks are scripts that run when you perform an action like commit, push or pull
  • this info is stored in the .git/hooks folder inside a dir
  • pre-commit’s .pre-commit-config.yaml basically describes what hooks to run, and when you run pre-commit install pre-commit makes corresponding changes the contents of the .git/hooks folder

Linters

flake8

https://flake8.pycqa.org/en/latest/#