Created by: gwideman, Mar 14, 2011 4:49 am
Revised by: gwideman, Mar 14, 2011 10:41 pm (5 revisions)


Overview

Mini-investigation into:
  • site-package dirs which play a particular role in sys.path
  • .pth files which can be used to add additional directories to sys.path
  • the site.py module which implements the behavior examined here

site.py

Module site.py adds a set of directories that are (according to PEP 250) particularly intended to hold locally-installed modules and packages, and locally-developed modules (and presumably packages). The module site.py is automatically imported (run) during initialization, and site.py source states that its purpose is to "append site-specific paths to the module search path." This basically includes:
  • Starting with one or two provided "prefix" directories, look for several standard subsidiary directories which, if they exist, are added to sys.path.
  • Inspect these subsidiary directories for *.pth files, which may contain additional paths to be added to sys.path

site-package directories

The directories to be added actually address two slightly different purposes:
  • For site-wide local packages, one or two directories subsidiary to the Python install directory
  • For each user's local packages, a directory subsidiary to the user's home directory
  • In each case, xxx.pth files can be used to ordain more directories for site-wide or per-user local packages
    • Note that the xxx.pth files cover almost the same use-case as the PYTHONPATH environment variable
      • could be site-wide, or per-user
      • but one or other may be a better fit to permissions and responsibilities.
The nuances of site-package can be understood from inspecting the source code of site.py, though the extensive comments at the beginning of that file have significant problems.

Directories which site.py examines

Note that currently site.py's docstring is incomplete on this (it omits mentioning the usersitepackages path, for example).
Relevant site.py code:
def main():
    ...
    known_paths = addusersitepackages(known_paths)
    known_paths = addsitepackages(known_paths)
    ...
main()
  • main()
    • addusersitepackages():
      • Compose USER_BASE, then USER_SITE = USER_BASE/something/site-packages
      • if USER_SITE is a dir
        • addsitedir(USER_SITE) (see below)
    • addsitepackages()
      • For each dir in prefix list (Python install dirs)
        • Append dir to local sitedirs list
        • Append dir+various suffixes to sitedirs
      • For each dir in sitedirs (that exists and is a dir)
        • addsitedir(dir) (see below)

  • addsitedir(dir)
    • Append dir to sys.path
    • Get list of *.pth files in dir
    • for each name.pth file
      • addpackage(dir, name,...)
        • Process name.pth file.
          • For a line that lists a name:
            • newdir = join(dir, line) (Makes relative path unless line is absolute path)
            • Add newdir to sys.path
          • If line is import statement, exec it

Directories examined by site.py are combinations as follows:
Windows
prefix from
Example

Suffix
Comment
sys.prefix
sys.exec_prefix
C:\Python\31\
or ???
combined
with
[nothing] or
lib\site-packages
Bug: actual directory is Lib not lib. (This is normally not an issue because Windows is case-insensitive, but could be an issue if using a case-sensitive server.)
USER_SITE
%APPDATA%\Python\
+
Python31\site-packages
31 is example version. (First and third char from version string)
See note [1] for example path
Note [1]
  • %APPDATA% is an environment variable that points a user specific directory.
    • Example on Windows 7: C:\Users\[username]\AppData\Roaming
  • Full path to "mypkg"
    • C:\Users\fred\AppData\Roaming\Python\Python31\site-packages\mypkg
Note [2]
  • On Windows, site.py treats the python install directory as a possible location for "site packages" (ie: location for pth files), even though there is no "site-packages" string in the path name.
    • I think this is a legacy alternative. I think use of that location fails to promote organization, so I think I will avoid that possibility.
Unix/Mac (non-framework)
prefix from
Example

Suffix

sys.prefix
sys.exec_prefix
/usr/local/
of ???
combined
with
lib/python3.1/site-packages or
lib/site-python
3.1 is example version. (First 3 chars of version string)
USER_SITE
~/.local/
+
lib/python3.1/site-packages

Processing of ".pth" files

  • site.py looks for .pth files only in the site-package directories (including the "bare-named" one on Windows).
    • .pth files elsewhere on the sys.path have no effect
    • In particular, .pth files never cascade -- there is no useful way for a .pth file to point to a location containing another .pth file and have that second one processed by site.py
  • pth files contain lines that are one of:
    • Comment starting with pound sign (#)
    • a string which is interpreted as an absolute path if possible, otherwise as a relative path (combined with the dir containing the pth file)
    • If it starts with "import " then exec it. This presumable is a way to provide some initialization for particular collection of files.

Observation

The current blurb in site.py's docstring, and the import feature of .pth files, suggest that the site-package feature may have been oriented to toward an older style of package which perhaps was simply a folder-with-files, lacking __init__.py files and accompanying automatic recognition of such a dir as a package.