Created by: gwideman, Dec 4, 2013 4:24 pm
Revised by: gwideman, Dec 6, 2013 4:25 pm (15 revisions)

.
  • Work in progress
  • Part of a series which starts here.

Overview

In previous parts of the series, I outlined the background needed to understand what the packaging/distribution/installation workflow needs to accomplish. In this part, the objective is to bring together key documentation needed to know how to operate the tools and to understand what they produce.
I do not regard this page as a "final solution" to this objective. It does supply some key topics, but also illuminates where there are gaps and conceptual problems in the current state of the tools or their documentation.
This page focuses on the tools supplied with the Python distribution, or managed by the Python Packaging Authority (PyPA):
  • distutils library
  • setuptools library, which includes the Easy Install installation tool
  • pip installation tool

The setup.py/setup( ) business

This section examines a somewhat technical detail, but because it is so central to distutils and setuptools (does it intersect with pip? I'm not sure), it is worth facing this issue head on.
Both distutils and setuptools (which is an elaborate extension to distutils) offer their features via a function called "setup( )". Both libraries expect the user to call setup( ) from a script (typically called setup.py) with some command-line parameters which the setup( ) function (or subsidiary functions) will interpret.
The full picture looks like this:
setup_py01.png
The picture for invoking setuptools looks exactly the same, except that the setup.py script imports the setuptools library instead of distutils:
setup_py_setuptools01.png
To be clear: The apparatus is invoked by the command line command "setup.py". This script calls function setup() (of distutils or setuptools), which expects both command-line arguments and function arguments.
So, in order to operate this apparatus, you will need to understand:
  • The command-line "setup commands" which the setup( ) function recognizes. These commands typically select one or more distribution related tasks to perform.
    • Example setup commands: sdist (create source distribution); upload (to PyPI); install.
    • You will need to understand the additional arguments for each of those commands.
    • The setuptools library implements additional commands beyond those of the base distutils library, and modifies some of the base commands.
  • The keyword arguments to the setup() function, which are typically used to specify metadata about the distribution (so remain more or less constant, regardless of which setup command is invoked).
    • Most keyword arguments take an additional argument (for example, specifying a name or a path)
    • The setuptools library recognizes more such arguments than the base distutils library.
  • Configuration files which may be used in the process
    • I have the impression that config files are relatively rarely used, so maybe study these only at a later stage.
    • Evidently these can contain lines with name-value pairs that bear some format relationship to commands that may be specified on the command line?

Notes

  • There's a MANIFEST file that is produced as output by some command(s), and sometimes used as input. Also a MANIFEST.in file. Seems like this file has been treated in different ways by different versions of some tools. Haven't tracked all that down.

Non-setup.py-based installation

There seems to be a move away from requiring the user to directly or indirectly run a setup.py script supplied by the distribution package, on the basis that the user ought not be obliged to run some supplied executable (which might do who-knows-what) in order to be able to inspect and gain confidence in the product being supplied. Although there are a fair number of articles floating around that mention this, is not clear to me what the full picture is.

Existing official docs

So, where to learn all these details?
There are a number of "official" existing documents relating to the packaging/distribution/installation process and tools. I have compiled a current snapshot of the ones I could find here:

Problems with official docs

  • Obsolete: Many of the official documents are out-of-date, way out-of-date, and/or conflict with each other. This is pernicious because of the rapid pace of recent (2013) change, and the chronic lack of credible dates on most documents.
    • The series on the python.org under the Installing Python Modules, and Distributing Python Modules, is particularly problematic.
    • The Hitchhiker's Guide to Packaging is well-meaning, but at this date (late 2013) complete rubbish.
  • Scattered and piecemeal: Because the operation of distutils and setuptools depends on command-line arguments, function arguments, and config files, there is no single coherent place to read docs regarding the complete set of inputs one can supply (and by extension, to learn the complete set of capabilities). Instead, this info is distributed over many pages of partial documentation, partly intermingled into tutorial narrative, and so on.
  • Setuptools - "a thing on a thing": Docs for setuptools are further hobbled because they skip documenting the commands offered by setuptools that come from the underlying distutils.
  • Lack of context: the docs don't identify typical use cases within which to make recommendations for the developer's project layout or the user's installed layout, and the deployment plans and workflow that would suit them.
  • Muddling of developer's and user's tasks: Because the same tools (setup.py/setup() ) are used for creating a distribution and installing the distribution, the docs are often unclear as to which phase of the process they are discussion. This is aggravated by some deployment scenarios performing build steps on the developer's system, and some scenarios performing build on the user's system.
As an attempt to gain traction on this chaotic situation, I embarked on some docs detail in the next section:

Comprehensive docs for commands, arguments and options for the main tools

To try to get to some sort of ground truth regarding the primary installation tools, I embarked on a process to try to distill authoritative docs from the tools' source code itself.
This is far from the complete set of information needed, because it doesn't go very far to relate the commands/arguments to the scenarios where they would be useful. But it's at least a model for one level of docs that is needed.

Distribution structure and format docs

To help understand what information is communicated in a distribution, and to ascertain whether the creation of a distribution produced a desired result, a developer needs documentation of the distribution structure and format.
Distribution format
Link to structure and format docs
Comments
zip file

Generic zip format with arbitrary contents and manual instructions... up to the developer to document.
zip with main.py
???

sdist
???
"Source" distribution
bdist
???
"Binary" or "Built" distribution
egg
pythonhosted.org/setuptools/formats.html
  • .egg format
  • .egg-info format
  • .egg-link file
wheel
PEP 427 -- The Wheel Binary Package Format 1.0

Notes:
  • Each structure/format has evolved over time, and continues to do so, as new PEPs have defined new features.

Note to self: don't forget...

  • Uninstall