New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implicit List conversion breaks backwards compatibility + numpy support #514
Comments
|
Is this the issue you are experiencing? |
|
As far as I can tell, this isn't like #451 because the error in #451 occurs in .NET. The error in this issue occurs in Python. But, the same underlying problem may be causing both issues. Given the .NET code above, here is a different demonstration of the poor behavior: That assertion used to (Computer 1, version 2.3) succeed. Now (Computer 2, ce14424+ on master) it fails. From #451, I see you commented, "The reason is because before the list is passed back to C#, it gets converted to Python list." It seems like this is a new (since 2.3) behavior, right? I suggest that this breaks backward compatibility and is generally undesirable. An explicit helper function to easily convert a generic .NET list to a Python list might be nice, but the conversion should not be performed automatically. But if it is, nonetheless, performed automatically, then the symmetric operation (automatically converting a Python list to a generic .NET list) MUST also happen. Also from #451: "But this is of course a lot of conversion back and forth and maybe a performance issue." I agree, which is why I strongly suggest not performing automatic conversion. EDIT: The reason I say a helper function to convert a .NET List to a Python list only might be nice is that the conversion is trivial: |
|
@BenjaminPelletier I agree that converting the CLR At the same time we need to get this sample with numpy arrays working seamlessly: https://github.com/pythonnet/pythonnet#example This is why the conversion was introduced. |
|
Calling Python from .NET is outside my use case so I'm not familiar with the logistics doing so. But "just like I'd expect" would be for pythonnet to expose explicit .NET types that map to Python types for the user to instantiate (in .NET) by some means before calling Python with them. These types would probably have helper functions for converting/wrapping from common corresponding .NET types. FWIW, here's how I'm marshalling arrays between .NET and NumPy, and it seems like it would have been nice to not have to figure out & write this myself: Python: .NET: |
|
@BenjaminPelletier if you would like to submit a helper function for numpy arrays, then pull requests are welcome! Here is another example of passing numpy arrays: https://stackoverflow.com/questions/43910749/efficiently-convert-system-single-to-numpy-array |
|
@denfromufa I also have a numpy array converter here With this converter, the example https://github.com/pythonnet/pythonnet#example can be replaced with this one,
|
|
@yagweb do you suggest that we should revert back the auto-conversion due to this regression and use your numpy converter? |
|
which commit broke this ? Was it the one I added to fix the example? |
|
@vmuriart this pull request broke C# List behavior: |
|
ok to revert behavior until better solution is found |
|
Hi, I wrote a Python-side Numpy <-> .NET conversion functions that use The I don't see any evidence of memory-leaking. Edit: one can do a zero-copy with |
|
@robbmcleod I just wanted to say thanks for your code and give feedback - it works great! I especially appreciate not having .NET wrapper and handling all conversion in Python/pythonnet/ctypes. |
|
This issue is a showstopper for us and required us to make a fork of the repo, which commented out the code in converter.cs in the Since the implicit conversion behavior might be useful to some but unacceptable to others, is it worth considering some run-time configuration option to obtain the desired behavior? |
|
Yeah, I'm more and more inclined to just deactivate it by default. A runtime switch is possible via the |
|
Tangentially related, but in our fork I just added a way to extend how .NET types are converted to Python and vice versa (so far without disabling conversions, that are coming out of the box, but I am planning this too). See losttech@c0ffc29 |
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for pythonnet#514
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for pythonnet#514
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for pythonnet#514
|
The new |
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for #514
|
This should now be solved by codecs: Codecs: .NET <-> Python conversions |
|
Thanks. |
Now Python host can force raw encoding for autoconverted .NET types. Enables workaround for pythonnet#514
|
@lostmsu Hi there. Would you be able to provide an example of how to use Codecs to prevent the conversion from a System.Collections.Generic.List to a python list? I read both the Codec wiki page and this example but I didn't get it, sorry. |
|
Sorry, I don't do that. I think I got associated with #514 because I reported that Pythonnet 2.x had broken support for .NET Remoting, and whoever fixed it (maybe?) introduced an issue related to what you are talking about. So that was holding up the release containing the .NET Remoting fix. Since the fix for me has been released, I'm guessing your issue also got resolved.
--------- Original Message --------- Subject: Re: [pythonnet/pythonnet] Implicit List conversion breaks backwards compatibility + numpy support (#514)
From: "ddsleonardo" <notifications@github.com>
Date: 10/30/20 5:38 am
To: "pythonnet/pythonnet" <pythonnet@noreply.github.com>
Cc: "jcarole" <github@jnpcarole.com>, "Manual" <manual@noreply.github.com>
@lostmsu Hi there. Would you be able to provide an example of how to use Codecs to prevent the conversion from a System.Collections.Generic.List to a python list? I read both the Codec wiki page and this example but I didn't get it, sorry.
Is there a way to just set this behaviour by default anytime I import pythonnet?
-
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
I do not understand your reply @jcarole. |
|
@ddsleonardo have you checked the new test named |
|
Sorry to confuse you. I mean to say that I don't know anything about the issue you're referring to and tried to explain why you might have thought I was at all involved in that topic.
…--------- Original Message --------- Subject: Re: [pythonnet/pythonnet] Implicit List conversion breaks backwards compatibility + numpy support (#514)
From: "ddsleonardo" <notifications@github.com>
Date: 10/30/20 9:41 am
To: "pythonnet/pythonnet" <pythonnet@noreply.github.com>
Cc: "jcarole" <github@jnpcarole.com>, "Mention" <mention@noreply.github.com>
I do not understand your reply @jcarole.
To make it clear, I am having the same problem as #1153 and also mentioned here. In both instances there are mentions either of RawProxyEncoder or the Codecs but I could not understand the usage examples. The only working solution I have found is this comment previously made in this discussion. This comment sounded encouraging, it would be really nice if we could make the conversions optional.
-
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
@lostmsu Thanks for the reply. My main question was relative to at what scope should I use the Codecs, i.e. do I need to define and register an instance of the Encoder class every time I open a Python Interactive and want to manipulate some C# Lists? (which I believe is the case, correct)? I think I managed to follow Still following on the same question, it sounds like this has been made a lot more convoluted than maybe it should be. I had been running a previous version of pythonnet for a while to avoid this, and only encountered the issue again as I had to reset my environment. Now I appreciate that this package isn't written just for me, but I'd be interested to know the motivations behind this, as it seems that I'm not the only one who has struggled with this and the workarounds to make it work are not trivial. At the same time, I am happy to run my own fork of this repo without the conversion, and set up a pipeline that keeps everything else up to date with the main repo. |
|
@ddsleonardo don't worry, the default conversion is going to be off by default in the next major version. I was not here around the time Python.NET started, but I suspect this conversion was added to make most APIs easily callable: |
|
Updated my code snippet for PythonNet 2.5.1, Python 3.9.4, and NumPy 1.20.2 for those who are looking for a working solution: https://gist.github.com/robbmcleod/73ca42da5984e6d0e5b6ad28bc4a504e |
|
@robbmcleod I adapted your last code so that complex arrays in python convert back-and-forth with SystemArray in .NET whose element type are directly System.Numerics.Complex (instead of adding extra dimension): https://gist.github.com/CitizenInsane/c8d3ddc5b14faceec433af3e940e22a8 |
|
@CitizenInsane thanks although in my case I actually do need the 64-bit complex as that's what the counterparty software that I'm talking to uses. |
|
@robbmcleod Yes it all depends. The .NET API i try to connect with is using System.Numerics.Complex arrays directly. Thanks for the great job, first time I'm doing python and this to interop with .net so it helped a lot to have your converters :). |
|
BTW, @robbmcleod , @CitizenInsane , in pythonnet 3.0 .NET arrays will implement buffer protocol! |
Good to know, thanks, that should be a big improvement over my memcpy solution. |


BenjaminPelletier commentedJul 20, 2017
Environment
Computer 1:
Computer 2:
Details
Description: Python calls .NET function which returns a List. Python then passes the return value, without modification, to a second .NET function which accepts a List. Computer 1 executes this code just fine. On Computer 2, there is no .NET function found that matches the arguments because the return value of the .NET function has been transformed into a Python list.
Python code:
.NET code:
The Python code works fine on Computer 1. On Computer 2, the PrintDoubleList call produces
TypeError: No method matches given arguments
If I print type(x) in Python, Computer 1 gives me a .NET List type while Computer 2 gives me a Python list. I can print x.Count on Computer 1, but I get a missing attribute error on Computer 2.
If I build manually from the 2.3 tag, I get the same (good) behavior as on Computer 1.
It seems that some feature has been partially added to automatically convert .NET objects into Python objects when possible. I suppose this is ok (though I would prefer that not happen because I don't want the mandatory performance hit of converting even when I don't want to convert), but if that's the intention, there must be automatic conversion of Python objects to .NET objects also. One without the other is a critical bug.
The text was updated successfully, but these errors were encountered: