Created by: gwideman, Oct 23, 2013 8:09 pm
Revised by: gwideman, Oct 26, 2013 3:38 pm (10 revisions)


This is the main page in my ongoing (Fall 2013) attempt to create, at least for myself and colleagues, some orderly docs on the following interlocking subjects as they relate to Python:
  • Project Layout
  • Project packaging, deployment and installation
  • Python Virtual Environments
  • Python and application/script launch methods and considerations
These topics are briefly elaborated below, followed by links to subsidiary pages exploring the topics in more detail.

Project Layout

The arrangement of components of Python projects, such as main application modules and subsidiary libraries, and other libraries under development but shared with other projects. The subject of which alternative layouts are to be preferred is not widely documented as such. Instead the developer is left to discover plausible layouts by working backwards from these other topics:
  • Python import statement and its variants.
  • Python sys.path, and how it is composed.
  • In addition, project layout might or might not be constrained by what arrangements are found tractable (or not) by:
    • the Python deployment and installation tools.
    • Launch methods
    • Testing tools/framework
    • Version control
Because import, sys.path and deployment are complex topics, it is difficult for programmers, working their way toward more complex projects, to discern what are the most workable project layouts.

Project packaging, deployment and installation

All that's involved in gathering up the locally developed project components into a package that can be deployed to other machines and installed there.
  • Installation may entail finding (on the web, typically in PyPI) and installing other third-party libraries which are needed.
    • Projects which create applications which are intended to be self-contained and for non-Python-savvy end-users may have additional requirements relative to projects that produce libraries for other Python users.
  • The Python deployment tool landscape has been in turmoil for years. As of fall 2013 it seems to have some stability, and a somewhat established trajectory of improvement. Much of this captured in PEP 453.
  • The documentation of the Python deployment landscape is riddled with conflicting and outdated (not to mention undated) documents, and even the official most-current docs are incoherent relative to each other. This chaos presents a huge obstacle for programmers working toward a more sophisticated use of Python, and also a huge opportunity to improve the cultivation of productive Python developers if the chaos was resolved.

Python Virtual Environments

A Python installation assumes a directory structure that encourages third-party libraries to be installed in a single tree associated with that Python installation. However, programmers working on multiple projects need the ability to install and uninstall third-party libraries without risk of them conflicting with other installed libraries. There's a special risk on OSes that use Python for OS utilities, in which messing with the "system Python" puts OS operation itself at risk. These problems can be addressed with the Python Virtual Environment mechanism.
  • Python 3.3 has the formerly-separate virtualenvs apparatus built-in as venvs. Some docs say that the built-in venv subsume (and might even conflict with) the separate virtualenvs, and most of the prominent docs continue to recommend installing virtualenvs and don't even mention built-in venv. WTF?
  • The virtual environments feature substantially solves the library conflict problem, but does it play nicely with the other major topics, most notably packaging and deployment, and the various methods for launching Python and Python scripts?

Python and application/script launch methods and considerations

There are numerous ways to start Python, and scripts written in Python. Especially with the introduction in Python 3.3 (and subsequent revision) of Python Launcher for Windows, the situation is ostensibly simplified, though it has introduced a bunch of failure modes.

Subsidiary pages

Following is a list of subsidiary pages, some of which are attempts to get to the bottom of one or another set of issues (ie: reverse-engineering the Python docs, some Python module, or just plain experimentation), and some pages are attempts to construct coherent explanations of some of these topics.