[Python-Dev] Packaging and binary distributions

[Paul Moore]

On 31 October 2011 18:36, Ned Deily <nad at acm.org> wrote:
> In article
> <CACac1F_V6_6+uG9qfqBJtuokz0HXO53hsXX3Ptap=O8+pxTpJA at mail.gmail.com>,
>  Paul Moore <p.f.moore at gmail.com> wrote:
>> On 30 October 2011 18:04, Ned Deily <nad at acm.org> wrote:
>> > Has anyone analyzed the current packages on PyPI to see how many provide
>> > binary distributions and in what format?
>>
>> A very quick and dirty check:
>>
>> dmg: 5
>> rpm: 12
>> msi: 23
>> dumb: 132
>> wininst: 364
>> egg: 2570
>>
>> That's number of packages with binary distributions in that format.
>> It's hard to be sure about egg distributions, as many of these could
>> be pure-python (there's no way I know, from the PyPI metadata, to
>> check this).
>
> Thanks.  If you have access to the egg file name, you should be able to
> tell.  AFAIK, eggs with extension modules include the Distutils platform
> name in the file name preceded by a '-', so '-linux', '-win32',
> '-macosx' for the main ones.  Pure python eggs do not contain a platform
> name.  http://pypi.python.org/pypi/pyinterval/ is a random example of
> the former.

136 architecture-specific
2502 architecture independent

About 5%. The numbers don't quite add up, so there's some funnies in
there (possibly bad data that I'm not handling well) but it gives an
idea.

Counts by architecture:

win32                    70
linux-i686               43
win-amd64                33
linux-x86_64             26
macosx-10.3-fat          12
macosx-10.5-i386         11
macosx-10.6-universal    9
macosx-10.6-fat          8
macosx-10.3-i386         7
macosx-10.6-i386         6
macosx-10.7-intel        4
macosx-10.6-intel        3
macosx-10.6-x86_64       2
macosx-10.3-ppc          2
macosx-10.4-i386         2
macosx-10.4-ppc          2
py2.3-linux-i686         1
py2.4-linux-i686         1
gnu-0.3-i686-AT386       1
linux-ppc                1
cygwin-1.5.25-i686       1
py2.3                    1
py2.4                    1
py2.5                    1
macosx-10.7-x86_64       1
macosx-10.4-universal    1
py2.5-linux-i686         1

Most of the 1-counts are bad data in some form.

I'm not sure what this proves, to be honest, but what I take from it is:

- Nearly all binary distributions are for Windows
- Architecture-neutral eggs are common (but not relevant here as
packaging can install from source with these)
- Ignoring architecture-neutral eggs, most popular formats are
wininst, egg, dumb(!!!) and msi
- Even the most popular binary format (wininst) only accounts for 2%
of all packages.

Having said all of this, there are two major caveats I'd include:

- Not everything is on PyPI.
- This analysis ignores relative importance. It's hard to claim that
numpy is no more significant than, say, "Products.CMFDynamicViewFTI"
(whatever that might be - I picked it at random, so apologies to the
author :-))

Paul.