A minor bug in the Hy programming language has led me down a rabbit hole of Python's internals, and I seem to have absorbed an awful lot of Python's
import semantics. The main problem can best be described this way: In Python, you call the import function with a string; that string gets translated in some way into python code. So: what are the exact semantics of the python
Over the next couple of posts, I'll try to accurately describe what it means when you write:
from alpha import beta
from alpha.beta import gamma
from .delta import epsilon</code>
In each case, python is attempting to resolve the collection of dotted names into a module object.
module object: A resource that is or can be compiled into a meaningful Python module. This resource could a file on a filesystem, a cell in a database, a remote web object, a stream of bytes in an object store, some content object in a compressed archive, or anything that can meaningfully be described as an array of bytes (Python 2) or characters (Python 3). It could even be dynamically generated!
module: The organizational unit of Python code. A namespace containing Python objects, including classes, functions, submodules, and immediately invoked code. Modules themselves may be collected into packages.
package: A python module which contains submodules or even subpackages. The most common packaging scheme is a directory folder; in this case the folder is a module if it contains an
__init__.py file, and it is a package if it contains other modules. The name of the package is the folder name; the name of a submodule would be
foldername.submodule. This is called regular packaging. An alternative method (which I might cover later) is known as namespace packaging.
Python has a baroque but generally flexible mechanism for defining how the dotted name is turned into a module object, which it calls module finding, and for how that module object is turned into a code object within the current Python session, called module loading.
Python also has a means for module listing. Listing is usually done on a list of paths, using an appropriate means for finding (identifying) the contents at the end of a path as Python modules.
The technical definition of a package is a module with a
__path__, a list of paths that contain submodules for the package. Subpackages get their own
__path__. A package can therefore accommodate
.. prefixes in submodules, indicating relative paths to sibling modules. A package can also and to its own
__path__ collection to enable access to submodules elsewhere.
The problem I am trying to solve:
Python module listing depends upon a finder resolving a path to a container of modules, usually (but not necessarily) a package. The very last finder is the default one: after all alternatives provided by users have been exhausted, Python reverts to the default behavior of analyzing the filesystem, as one would expect. The default finder is hard-coded to use the Python builtin
imp.get_suffixes() function, which in turn hard-codes the extensions recognized by the importer.
If one wants to supply alternative syntaxes for Python and have heterogenous packages (for examples, packages that contain some modules ending in
.hy, and others
.py, side-by-side)... well, that's just not possible.
In the next post, I'll discuss Python's two different finder resolution mechanisms, the meta_path and the path_hook, and how they differ, and which one we'll need to instantiate to solve the problem of heterogenous Python syntaxes. The actual solution will eventually involve eclipsing Python's default source file handler with one that enables us to define new source file extensions at run-time, recognize the source file extensions, and supply the appropriate compilers for those source files, while falling back on the default behavior correctly when the extension is one Python can handle by itself.
I am trying to solve that problem.