close
Skip to content

Further optimize distribution() searching.#272

Merged
jaraco merged 3 commits into
python:mainfrom
anntzer:moreperf
Jan 10, 2021
Merged

Further optimize distribution() searching.#272
jaraco merged 3 commits into
python:mainfrom
anntzer:moreperf

Conversation

@anntzer

@anntzer anntzer commented Jan 6, 2021

Copy link
Copy Markdown
Contributor

Locally, this speeds up distribution("ipython") on a large-ish
environment by nearly 2-fold. (The GHA benchmark goes from ~1.6ms
to ~0.8ms.)

All the following changes matter for the performance gain:

  • avoid needlessly reinstanciating Prepareds.
  • switch from os.path.splitext (very slow) to str.rpartition
  • avoid having to add back a dot to the ext part given by rpartition
  • precompute egg_prefix and versionless_egg_name

(See https://gitlab.com/python-devs/importlib_metadata/-/issues/95 for
rationale why performance matters.)

@jaraco

jaraco commented Jan 10, 2021

Copy link
Copy Markdown
Member

Thanks for this. It's excellent. The only real wish I have (for future consideration) is if these optimizations had been committed and pushed separately, we could see the impact of each change independently.

Any idea why the Python 3.6 tests are timing out?

@jaraco

jaraco commented Jan 10, 2021

Copy link
Copy Markdown
Member

Any idea why the Python 3.6 tests are timing out?

Oh, dear. I see the Python 3.6 tests have been timing out for some time, although apparently only intermittently. I'll bet it's some bug with pips resolver and selective dependencies on self (importlib_metadata). I'll deal with that in another issue/pr.

@anntzer

anntzer commented Jan 10, 2021

Copy link
Copy Markdown
Contributor Author

if these optimizations had been committed and pushed separately, we could see the impact of each change independently.

Done.

@jaraco jaraco closed this Jan 10, 2021
@jaraco jaraco reopened this Jan 10, 2021
@jaraco

jaraco commented Jan 10, 2021

Copy link
Copy Markdown
Member

Looks like the second and third commits had the most impact:

importlib_metadata HEAD $ git checkout -q c1af1d6~1; tox -q -e perf
200 loops, best of 5: 1.47 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q c1af1d6; tox -q -e perf
200 loops, best of 5: 1.45 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q c9c0909; tox -q -e perf
200 loops, best of 5: 1.14 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q 7fc4f11; tox -q -e perf
200 loops, best of 5: 928 usec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

@jaraco jaraco closed this Jan 10, 2021
@jaraco jaraco reopened this Jan 10, 2021
@jaraco jaraco merged commit c93a9f7 into python:main Jan 10, 2021
@anntzer anntzer deleted the moreperf branch January 10, 2021 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants