This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Overriding __new__ method with itself changes behaviour of the class
Type: behavior Stage:
Components: Interpreter Core Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: SilentGhost, alexey-muranov, orlnub123, rhettinger, veky
Priority: normal Keywords:

Created on 2019-05-07 11:42 by alexey-muranov, last changed 2022-04-11 14:59 by admin.

Messages (9)
msg341705 - (view) Author: Alexey Muranov (alexey-muranov) Date: 2019-05-07 11:42
I expect that overriding methods with themselves like in the following example should not change the behaviour of the class:

    class C:
        a = 1

        def __init__(self, b):
            self.b = b

        def f(self, x):
            return x*self.a + self.b + 1

        @classmethod
        def g(cls, y):
            return y*cls.a + 2

        @staticmethod
        def h(z):
            return z + 3

    class D(C):
        def f(self, *args, **kwargs):
            return super(__class__, self).f(*args, **kwargs)

        @classmethod
        def g(cls, *args, **kwargs):
            return super(__class__, cls).g(*args, **kwargs)

        @staticmethod
        def h(*args, **kwargs):
            return super(__class__, __class__).h(*args, **kwargs)

    c = C(7)
    d = D(7)
    assert c.f(42) == d.f(42)
    assert c.g(42) == d.g(42)
    assert c.h(42) == d.h(42)

(Moreover, I expect to be able to extend superclass method this way.)

However, this does not work with `__new__`:

    class C:
        def __init__(self, x):
            self.x = x
            print(x)

    class D(C):
        @staticmethod
        def __new__(*args, **kwargs):
            return super(__class__, __class__).__new__(*args, **kwargs)

    C(7) # fine
    D(7) # TypeError: object.__new__() takes exactly one argument

I think this is not a desirable feature.  I would call it a bug.

By the way, I understand why `object`'s `__init__` can complain about a wrong number of arguments, but I do not see a point in making `object`'s `__new__` complain about it.

Here is my current workaround:

    class T:
        @staticmethod
        def __new__(cls, *_args, **_kwargs):
            return object.__new__(cls)

    class E(C, T):
        @staticmethod
        def __new__(*args, **kwargs):
            return super(__class__, __class__).__new__(*args, **kwargs)

    C(42) # fine
    E(42) # fine

A possibly related issue: #32768 (https://bugs.python.org/issue32768)
msg341709 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2019-05-07 12:12
It seems you're misunderstanding the point of __new__ and your usage of it is a bit strange. You're getting an error because you're ultimately calling object.__new__ which doesn't accept any other arguments but the class for instantiation. I'd suggest the documentation for object.__new__ (https://docs.python.org/3/reference/datamodel.html#object.__new__) and the Data model page in general.
msg341710 - (view) Author: Alexey Muranov (alexey-muranov) Date: 2019-05-07 12:22
The issue is the following: i expect overriding a method with itself to not change behaviour of the class.  I do not see how my understanding of `__new__` or its point could be relevant.

Do we agree that overriding a method with itself should not change behaviour?  Is there a more correct way to do it than

    def foo(self, *args, **kwarg):
        # possible extensions
        # ...
        super(__class__, self).foo(*args, **kwarg)

(modified accordingly for class and static methods)?

When I do not override `__new__`, I expect Python to use `object`'s `__new__` (or at least pretend that it does). Therefore there should be no difference in behaviour.
msg341711 - (view) Author: Alexey Muranov (alexey-muranov) Date: 2019-05-07 12:25
Incidentally, the documentation gives the following signature of __new__:

    object.__new__(cls[, ...])

which suggests a variable number of arguments.
msg341728 - (view) Author: Alexey Muranov (alexey-muranov) Date: 2019-05-07 14:08
I've noticed some faults in my code examples: `super(__class__, __class__)` instead of a more appropriate `super(__class__, cls)`, forgotten `return` before `super(__class__, self).foo(*args, **kwarg)`, maybe there are more.  I cannot edit previous comments, but this does not affect the main point.

I've stumbled on this behaviour in a situation where it actually poses me a problem.  However, here is some analogy: if a calculator returns 0 as the result of a multiplication of any number by 1, this cannot be justified by saying that no one needs to multiply numbers by 1 anyway.
msg341783 - (view) Author: Alexey Muranov (alexey-muranov) Date: 2019-05-07 17:16
Here is an example of code where i got surprised by the current behaviour and had to apply some (ugly) workaround:

https://gist.github.com/alexeymuranov/04e2807eb5679ac7e36da4454a58fa7e
msg342185 - (view) Author: Alexey Muranov (alexey-muranov) Date: 2019-05-11 12:37
I see that I am not the only one who got bitten by this unexpected behaviour (though the others might have not realised it).  There is a question ["Creating a singleton in Python"][1] on StackOverflow which was posted in 2011 and by now has the total of 737 upvotes.  Here is a code snippet from the question:

    class Singleton(object):
        _instance = None
        def __new__(class_, *args, **kwargs):
            if not isinstance(class_._instance, class_):
                class_._instance = object.__new__(class_, *args, **kwargs)
            return class_._instance

    class MyClass(Singleton, BaseClass):
        pass

[1]: https://stackoverflow.com/q/6760685

Currently this does not work as expected, try:

    class Failed(Singleton):
        def __init__(self, _):
            pass

    Failed(42)  # TypeError: object.__new__() takes exactly one argument ...

There is a similar code snippet in the accepted [answer][2] with 507 upvotes:

    class Singleton(object):
        _instances = {}
        def __new__(class_, *args, **kwargs):
            if class_ not in class_._instances:
                class_._instances[class_] = super(Singleton, class_).__new__(
                    class_, *args, **kwargs
                )
            return class_._instances[class_]

    class MyClass(Singleton):
        pass

[2]: https://stackoverflow.com/a/6798042

This does not work either, for the same reason.
msg344661 - (view) Author: Vedran Čačić (veky) * Date: 2019-06-05 01:30
The point is, constructing via __new__ and initializing via __init__ are two mechanisms that aren't really orthogonal, and various heuristics are trying to guess whether the arguments you call your class with are meant for __new__ or for __init__. [Usually, immutable objects are __new__able, and mutable ones are __init__able -- and crucially, objects are usually not both. For example, list is __init__able, but tuple is __new__able.] Those heuristics assume you won't have both __init__ and __new__ implemented on your object. It mostly works if you have a hierarchy of __new__able objects, or a hierarchy of __init__able objects, but as you noticed, it usually breaks if you have an __init__able class derived from a __new__able one, or vice versa. If you're really trying for a most general solution, please note that __init__ is redundant: everything __init__ does can be done with __new__ (but not vice versa). So, you should just have a hierarchy of __new__able classes.
msg344675 - (view) Author: (orlnub123) * Date: 2019-06-05 06:09
The comments in object's source explain the rationale behind this:

/* You may wonder why object.__new__() only complains about arguments
   when object.__init__() is not overridden, and vice versa.

   Consider the use cases:

   1. When neither is overridden, we want to hear complaints about
      excess (i.e., any) arguments, since their presence could
      indicate there's a bug.

   2. When defining an Immutable type, we are likely to override only
      __new__(), since __init__() is called too late to initialize an
      Immutable object.  Since __new__() defines the signature for the
      type, it would be a pain to have to override __init__() just to
      stop it from complaining about excess arguments.

   3. When defining a Mutable type, we are likely to override only
      __init__().  So here the converse reasoning applies: we don't
      want to have to override __new__() just to stop it from
      complaining.

   4. When __init__() is overridden, and the subclass __init__() calls
      object.__init__(), the latter should complain about excess
      arguments; ditto for __new__().

   Use cases 2 and 3 make it unattractive to unconditionally check for
   excess arguments.  The best solution that addresses all four use
   cases is as follows: __init__() complains about excess arguments
   unless __new__() is overridden and __init__() is not overridden
   (IOW, if __init__() is overridden or __new__() is not overridden);
   symmetrically, __new__() complains about excess arguments unless
   __init__() is overridden and __new__() is not overridden
   (IOW, if __new__() is overridden or __init__() is not overridden).

   However, for backwards compatibility, this breaks too much code.
   Therefore, in 2.6, we'll *warn* about excess arguments when both
   methods are overridden; for all other cases we'll use the above
   rules.

*/
History
Date User Action Args
2022-04-11 14:59:14adminsetgithub: 81008
2019-06-05 06:09:16orlnub123setnosy: + orlnub123
messages: + msg344675
2019-06-05 01:30:53vekysetnosy: + veky
messages: + msg344661
2019-05-11 12:37:59alexey-muranovsetmessages: + msg342185
2019-05-07 17:16:59alexey-muranovsetmessages: + msg341783
2019-05-07 14:08:26alexey-muranovsetmessages: + msg341728
2019-05-07 14:00:20SilentGhostsetnosy: + rhettinger
2019-05-07 12:25:44alexey-muranovsetmessages: + msg341711
2019-05-07 12:22:06alexey-muranovsetmessages: + msg341710
2019-05-07 12:12:16SilentGhostsetnosy: + SilentGhost
messages: + msg341709
2019-05-07 11:42:08alexey-muranovcreate