A minor bug in the Hy programming language has led me down a rabbit hole of Python's internals, and I seem to have absorbed an awful lot of Python's import semantics. The main problem can best be described this way: In Python, you call the import function with a string; that string gets translated in some way into python code. So: what are the exact semantics of the python import command?

Over the next couple of posts, I'll try to accurately describe what it means when you write:

<code>import alpha.beta.gamma
from alpha import beta
from alpha.beta import gamma
from .delta import epsilon</code>

In each case, python is attempting to resolve the collection of dotted names into a module object.

module object: A resource that is or can be compiled into a meaningful Python module. This resource could a file on a filesystem, a cell in a database, a remote web object, a stream of bytes in an object store, some content object in a compressed archive, or anything that can meaningfully be described as an array of bytes (Python 2) or characters (Python 3). It could even be dynamically generated!

module: The organizational unit of Python code. A namespace containing Python objects, including classes, functions, submodules, and immediately invoked code. Modules themselves may be collected into packages.

package: A python module which contains submodules or even subpackages. The most common packaging scheme is a directory folder; in this case the folder is a module if it contains an __init__.py file, and it is a package if it contains other modules. The name of the package is the folder name; the name of a submodule would be foldername.submodule. This is called regular packaging. An alternative method (which I might cover later) is known as namespace packaging.

Python has a baroque but generally flexible mechanism for defining how the dotted name is turned into a module object, which it calls module finding, and for how that module object is turned into a code object within the current Python session, called module loading.

Python also has a means for module listing. Listing is usually done on a list of paths, using an appropriate means for finding (identifying) the contents at the end of a path as Python modules.

The technical definition of a package is a module with a __path__, a list of paths that contain submodules for the package. Subpackages get their own__path__. A package can therefore accommodate . and .. prefixes in submodules, indicating relative paths to sibling modules. A package can also and to its own __path__ collection to enable access to submodules elsewhere.

The problem I am trying to solve:

Python module listing depends upon a finder resolving a path to a container of modules, usually (but not necessarily) a package. The very last finder is the default one: after all alternatives provided by users have been exhausted, Python reverts to the default behavior of analyzing the filesystem, as one would expect. The default finder is hard-coded to use the Python builtin imp.get_suffixes() function, which in turn hard-codes the extensions recognized by the importer.

If one wants to supply alternative syntaxes for Python and have heterogenous packages (for examples, packages that contain some modules ending in .hy, and others .py, side-by-side)... well, that's just not possible.

Yet.

In the next post, I'll discuss Python's two different finder resolution mechanisms, the meta_path and the path_hook, and how they differ, and which one we'll need to instantiate to solve the problem of heterogenous Python syntaxes. The actual solution will eventually involve eclipsing Python's default source file handler with one that enables us to define new source file extensions at run-time, recognize the source file extensions, and supply the appropriate compilers for those source files, while falling back on the default behavior correctly when the extension is one Python can handle by itself.

My hope is that, once solved, this will further enable the development of Python alternative syntaxes. Folks bemoan the explosion of Javascript precompilers, but the truth is that it has in part led to a revival in industrial programming languages and a renaissance in programming language development in general.   Python, with its AST available and exposed at runtime, is eminently suitable as an alternative language research platform-- except for this one little problem.

I am trying to solve that problem.