Skip to content

feat: tensor type for protobuf deserialization#1645

Merged
JoanFM merged 9 commits into
mainfrom
feat-protobuf-tensor-type
Jun 14, 2023
Merged

feat: tensor type for protobuf deserialization#1645
JoanFM merged 9 commits into
mainfrom
feat-protobuf-tensor-type

Conversation

@JohannesMessner

@JohannesMessner JohannesMessner commented Jun 13, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

This allows DocVoc to be deserilzed to a specific tensor_type, i.e. torch, tf, or numpy:

# look at the unit test for more comprehensive example
class MyDoc(BaseDoc):
    tensor: TensorFlowTensor

da = DocVec[MyDoc](...)  # doesn't matter what tensor_type is here

proto = da.to_protobuf()
da_after = DocVec[MyDoc].from_protobuf(proto, tensor_type=TensorFlowTensor)

assert isinstance(da_after.tensor, TensorFlowTensor)

Note that the tensor_type passed to from_protobuf() does not need to match the tensor type from before serialization.
Since all tensor are represented the same way in the proto, any proto can be deserialized to any tensor type.

TODO:

  • docs

@JohannesMessner
feat: tensor type for protobuf
e70f033 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner JohannesMessner mentioned this pull request Jun 13, 2023
Merged
9 tasks
@JohannesMessner
docs: explain tensor type from protobuf
ddd7b19 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
test: fix tf tensor import
eaf2a0f 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner JohannesMessner requested a review from samsja June 13, 2023 14:05
samsja
samsja reviewed Jun 13, 2023
View reviewed changes

@samsja samsja left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so what happened if I do to_profobuf with a TorchTensor in my doc_type ? It will be loaded as numpy tensor right ?

@JohannesMessner
test: try to fix tf test
8a32574 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner

JohannesMessner commented Jun 13, 2023

Copy link
Copy Markdown
Member Author

so what happened if I do to_profobuf with a TorchTensor in my doc_type ? It will be loaded as numpy tensor right ?

yes, if not tensor_type is specified in from_protobuf it will attempt to load it as numpy. This mirrors the behaviour of DocVec.__init__().

@JohannesMessner
test: another go at the tf test
ae65251 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
test: another go at the tf test
76d4fc8 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
test: another go at the tf test
026a8ef 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner
test: fing tf pls work now
c2812cf 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@JohannesMessner JohannesMessner marked this pull request as ready for review June 13, 2023 15:28
@JohannesMessner
test: remove proto mark
f583129 
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
@github-actions

github-actions Bot commented Jun 13, 2023

Copy link
Copy Markdown

📝 Docs are deployed on https://ft-feat-protobuf-tensor-type--jina-docs.netlify.app 🎉

@JohannesMessner JohannesMessner requested a review from samsja June 14, 2023 07:19
@JoanFM JoanFM merged commit eb3f857 into main Jun 14, 2023
@JoanFM JoanFM deleted the feat-protobuf-tensor-type branch June 14, 2023 09:23
@samsja samsja mentioned this pull request Jun 19, 2023
Closed
@scott-martens scott-martens mentioned this pull request Jun 20, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@samsja samsja samsja approved these changes

Assignees

No one assigned

Labels

area/core area/docs area/testing component/array size/m

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JohannesMessner @samsja @JoanFM