Skip to content

feat(redis): implement Redis storage backend and unit tests#452

Merged
hanxiao merged 107 commits into
docarray:mainfrom
AnneYang720:feat-redis-backend
Aug 26, 2022
Merged

feat(redis): implement Redis storage backend and unit tests#452
hanxiao merged 107 commits into
docarray:mainfrom
AnneYang720:feat-redis-backend

Conversation

@AnneYang720

@AnneYang720 AnneYang720 commented Jul 25, 2022

Copy link
Copy Markdown
Contributor

This PR will implement Redis storage backend and its unit test according to #445

@bwanglzu bwanglzu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some comments

Comment thread docarray/array/storage/redis/backend.py Outdated
Comment thread docarray/array/storage/redis/backend.py Outdated
Comment thread docarray/array/storage/redis/backend.py Outdated
Comment thread docarray/array/storage/redis/backend.py Outdated
Comment thread docarray/array/storage/redis/backend.py Outdated
Comment thread docarray/array/storage/redis/backend.py Outdated
Comment thread docarray/array/storage/redis/backend.py Outdated

@JoanFM JoanFM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these unit tests are excessive.You are only testing private methods. In general, we should test the public interface. These tests may make changing underlying implementation very hard

Comment thread docarray/array/storage/redis/backend.py Outdated
@bwanglzu

bwanglzu commented Jul 27, 2022

Copy link
Copy Markdown

please also fix the commit messages, see error in commit-lint, use git amend

@codecov

codecov Bot commented Jul 27, 2022

Copy link
Copy Markdown

Codecov Report

Merging #452 (5f94757) into main (7c91c7b) will increase coverage by 0.32%.
The diff coverage is 94.13%.

@@            Coverage Diff             @@
##             main     #452      +/-   ##
==========================================
+ Coverage   86.53%   86.86%   +0.32%     
==========================================
  Files         134      140       +6     
  Lines        6704     6994     +290     
==========================================
+ Hits         5801     6075     +274     
- Misses        903      919      +16     
Flag Coverage Δ
docarray 86.86% <94.13%> (+0.32%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
docarray/array/document.py 70.76% <62.50%> (-1.17%) ⬇️
docarray/array/storage/redis/seqlike.py 86.66% <86.66%> (ø)
docarray/array/storage/redis/find.py 94.36% <94.36%> (ø)
docarray/array/storage/redis/backend.py 96.19% <96.19%> (ø)
docarray/array/storage/redis/getsetdel.py 96.77% <96.77%> (ø)
docarray/array/redis.py 100.00% <100.00%> (ø)
docarray/array/storage/redis/__init__.py 100.00% <100.00%> (ø)
docarray/array/storage/base/helper.py 92.30% <0.00%> (+2.56%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@bwanglzu bwanglzu marked this pull request as ready for review August 1, 2022 09:02
@bwanglzu bwanglzu linked an issue Aug 1, 2022 that may be closed by this pull request
@bwanglzu bwanglzu mentioned this pull request Aug 1, 2022
7 tasks

@JoanFM JoanFM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of all these tests for now I would keep only test_init_storage

Comment thread docarray/array/storage/redis/backend.py Outdated
Comment thread tests/unit/array/storage/redis/test_backend.py Outdated
@bwanglzu bwanglzu marked this pull request as draft August 1, 2022 11:56
@bwanglzu

bwanglzu commented Aug 1, 2022

Copy link
Copy Markdown

@AnneYang720 let's commit everything on this branch without creating seperate PRs, we'll make it ready-to-review until everything is here

@alaeddine-13 alaeddine-13 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides the documentation, LGTM, well done !

Comment thread docs/advanced/document-store/redis.md Outdated

### Vector search with filter query

One can perform Vector Similarity Search based on FLAT or HNSW algorithm and pre-filter results using a filter query that is based on [MongoDB's Query](https://www.mongodb.com/docs/manual/reference/operator/query/). We currently support a subset of those selectors:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wouldn't mention MongoDB here. BTW we can refer it as domain specific language in the future or DSL

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's fine we already mention it here: https://docarray.jina.ai/fundamentals/documentarray/find/

@bwanglzu bwanglzu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work on the docs! left some minor comments

Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated

### Vector search with filter query

One can perform Vector Similarity Search based on FLAT or HNSW algorithm and pre-filter results using a filter query that is based on [MongoDB's Query](https://www.mongodb.com/docs/manual/reference/operator/query/). We currently support a subset of those selectors:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's fine we already mention it here: https://docarray.jina.ai/fundamentals/documentarray/find/

Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md

@JohannesMessner JohannesMessner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some rephrasings

Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
Comment thread docs/advanced/document-store/redis.md Outdated
@hanxiao hanxiao merged commit c2235de into docarray:main Aug 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Redis storage backend - backend.py

6 participants