Why is importlib.metadata much slower than importlib.util.find_spec?

[jaraco]

In GitLab by @anntzer on Nov 17, 2019, 22:50

Not sure whether this should be reported here or on bugs.python.org.

In a clean Py3.8 venv with just ipython installed:

In [1]: import importlib.util, importlib.metadata                                                

In [2]: %timeit importlib.util.find_spec("pip")                                                  
93.5 µs ± 256 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [3]: %timeit importlib.metadata.distribution("pip")                                           
5.59 ms ± 58.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

As far as I know, importlib.util.find_spec does no caching (that's why I'm not actually timing the import itself), so my uneducated guess is that it should spend a similar amount of time looking for the importable module/package as importlib.metadata.distribution spends looking for the dist-info/egg-info. But importlib.metadata is ~50x slower. Why is this the case?

As a real application, consider packages who may want to set their __version__ at runtime by querying their own distribution version using importlib.metadata (e.g., if the version originally comes from git tags, using setuptools_scm). 5ms would be a not insignificant price to pay for that, especially if multiple dependent packages also do the same.

[jaraco]

In GitLab by @jaraco on Nov 30, 2019, 11:11

Thanks for the report (and data). Happy to explain and help investigate.

Not sure whether this should be reported here or on bugs.python.org.

Either is fine. I'm likely to address non-emergency issues here first.

Why is importlib.metadata much slower than importlib.util.find_spec?

The first thing to note is that distribution() and find_spec() operate on two fundamentally different entities. In the latter, "pip" refers to the Python Package (a special form of a Python Module), whereas in the former, "pip" refers to a Distribution Package (something published to PyPI). And while those two entities are related (by name and convention), they're very different concepts semantically and could be completely unrelated.

Consider, for example the path.py distribution that implements the path module or setuptools, which implements the setuptools package, pkg_resources package, and easy_install module. Some modules (like stdlib modules and packages) don't have any corresponding distribution package.

Moreover, find_spec is operating on importable modules, the functionality which has been highly optimized over decades, whereas distribution() was created on top of a de-facto spec to provide a reasonably-performant approach (compared to pkg_resources) to discovering installed (distribution) packages with minimal overhead.

In particular, find_distributions aims to eliminate the much more expensive operation of enumerating all packages that pkg_resources has on import.

So there are inherent differences between find_distributions and find_spec that means that their performance characteristics may not be comparable.

That said, I'm not sure if what find_distributions is doing is inherently expensive or could be further optimized. I'd be delighted if you would be interested in investigating further to identify unnecessary slowness or other optimizations that could be made.

[jaraco]

In GitLab by @anntzer on Nov 30, 2019, 17:27

Thanks, I am well aware of the difference between distributions and packages (although I realize this was not obvious from the original issue description).

A quick profiling shows that a lot of time is spent playing with regexs in MetaPathFinder._matches_{info,legacy}, and indeed, rewriting MetaPathFinder._search_path to

    @classmethod
    def _search_path(cls, root, pattern):
        if not root.is_dir():
            return
        normalized = pattern.replace('-', '_')
        for item in root.iterdir():
            if (item.name.lower().startswith((normalized + ".", normalized + "-"))
                    and item.suffix.lower() in [".dist-info", ".egg-info"]):
                yield item
            elif (root.name.lower().startswith((normalized + ".", normalized + "-"))
                  and root.suffix.lower() in [".dist-info", ".egg-info"]
                  and item.name.lower() == "egg-info"):
                yield item

yields a ~2.5x speedup of the test case described originally.

(Note that technically the string checks do not exactly match the regex checks, but I think the only difference is that this would additionally incorrectly match "name..dist-info" (with two (or more) dots); this can probably be also handled without compromising speed too much.

Probably more not-insignificant microoptimization can be done e.g. lifting the check on root.name/root.suffix out of the loop, and switching to os.listdir/os.scandir specifically here to avoid paying the cost of constructing Path objects.)

---

Edit: indeed,

    @classmethod
    def _search_path(cls, root, pattern):
        if not root.is_dir():
            return
        normalized = pattern.replace('-', '_')
        root_n_low = root.name.lower()
        root_is_info = (
            root_n_low.startswith((normalized + ".", normalized + "-"))
            and root_n_low.endswith((".dist-info", ".egg-info")))
        for name in os.listdir(root):
            n_low = name.lower()
            if (n_low.startswith((normalized + ".", normalized + "-"))
                    and n_low.endswith((".dist-info", ".egg-info"))):
                yield root / name
            elif (root_is_info and n_low == "egg-info"):
                yield root / name

is ~8.5x faster than the original implementation.

[jaraco]

In GitLab by @anntzer on Dec 8, 2019, 19:25

mentioned in merge request !103

[jaraco]

In GitLab by @jaraco on Jan 10, 2020, 09:43

mentioned in commit 969e955

[jaraco]

In GitLab by @jaraco on Jan 10, 2020, 22:09

mentioned in commit 7b5797e

[jaraco]

In GitLab by @jaraco on Jan 10, 2020, 22:10

mentioned in merge request !106

[jaraco]

In GitLab by @jaraco on Jan 10, 2020, 23:51

closed via merge request !106

[jaraco]

In GitLab by @jaraco on Jan 10, 2020, 23:51

mentioned in commit b22b8a7