Fixing a sudden 10x increase in Pip's build time

published on Nov 22, 2022

slow pip Time is money.

Building the Docker image of an app I'm working on suddenly became a big pain. What used to take a little less than two minutes, can no longer be completed in less than 15 minutes.

The process

At first I suspected my internet connection was at fault. But when I had the same problem while on a +100 Mbps network, I blamed my computer's slow hard drive. It was only after I had the same issue in an SSD-equipped VPS that I started to think something was wrong in the app itself.

I'm building the using Python (with Django) and this problem only manifested itself after I updated the base image from 3.10 to 3.11.

To understand what was going on, I started to inspect Pip's output as it was there where the 99% of the delay occurred. Spending more quality time on the issue I discovered that the main culprit is Pandas. It took minutes on end for Pip to install it.

For a second, I thought I was out of luck. Pandas is a huge package and it made sense for me that it took longer than other packages to install. Fortunately for me, I decided to look further into it; why on earth would the new version of Python render Pandas so difficult to install.

More time with Pip revealed that Pandas exhibited a different behavior than most of its peers. For other packages, Pip was merely downloading their .whl file. With Pandas, however, Pip was building the wheel file. "Something fishy is going on", I thought. After some Gooogling, I came across this answer on Stack Overflow. This section showed the light at the end of the tunnel:

It seems you're using CPython3.8. pandas==0.24.2 does not wheels built for your version, so your system builds them for itself each time. You can check the available download files from here.

The solution

Doing the same for Pandas revealed that the version I used (1.4.3) didn't have ready-made Wheels for Python 3.11. Pandas 1.5.1, the latest stable version at the moment offered however such a file.

To get to list of Wheel files available for any version of a Pip package, just visit the package's page on PyPI, and then:

  1. Go to Release history on the left navigation bar;
  2. Click on the version you're interest in;
  3. Go to the left bar again, but this time select the Download files link;

Once the page loads, search the (sometimes long) list of .whl files to see if any of them is made for your version of Python. But be aware that this can be a little tricky for lack of naming standardization.

For Pandas, for example, you have to look for cp311 in the filename if you're aiming for Python 3.11. For Django, on the other hand, the hint is marked as py3, i.e. the file is compatible with any version of Python 3.

To solve my problem, all I had to do is to update my requirements.txt file to require installing Pandas 1.5.1 as it's Wheel file for Python 3.11 was already available. Pip do no longer need to build the Wheel file itself, thereby getting the build time right where is was before bumping Python's version.

END OF FILE