Skip to content

feat: subindex for document index#1428

Merged
AnneYang720 merged 75 commits into
mainfrom
feat-subindex
May 11, 2023
Merged

feat: subindex for document index#1428
AnneYang720 merged 75 commits into
mainfrom
feat-subindex

Conversation

@AnneYang720

@AnneYang720 AnneYang720 commented Apr 20, 2023 •
edited by samsja
Loading

Copy link
Copy Markdown
Contributor

This PR is related to issue #1235

ToDo:
Subindex

  • init
    • abstract
    • change schema (add parent_id dynamically)
    • hnswlib
    • elastic
  • index
    • abtract
    • hnswlib
    • elastic
  • get
  • del
  • find
  • find (return both root and subindex results)
  • filter
  • tests
    • abtract common methods
    • hnswlib
    • elastic (v7&v8)
  • weaviate
  • qdrant
  • Documentation

Example

class SimpleDoc(BaseDoc):
    simple_tens: NdArray[10]
    simple_text: str

class MyDoc(BaseDoc):
    docs: DocList[SimpleDoc]

my_docs = [MyDoc(id='0', docs=DocList[SimpleDoc]([SimpleDoc(simple_tens=np.ones(10) * (j + 1), simple_text=f'hello {j}',)for _ in range(10)])]
index = ElasticDocIndex[MyDoc](index_name='idx')
index.index(my_docs)# index with name 'idx' and 'idx__docs' will be generated

doc = index['0] #this doc should be complete
print(doc) 
print(doc.docs[0])

# you can find on subindex level, and return results will be subindex docs
docs, scores = index.find(query, search_field='docs__simple_tens', limit=1) 
# or you can use find_subindex, which can return both root and sub results
root, sub, stores = index.find_subindex(query, search_field='docs__simple_tens', limit=5)

# filter_subindex is for filter on subindex level, return results are subindex docs
query = {'match': {'simple_text': 'hello 0'}}
docs = index.filter_subindex(query, subindex='docs', limit=5)

del index['0'] # delete all subindex docs as well

@AnneYang720
feat: subindex init
603012d 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720 AnneYang720 marked this pull request as draft April 20, 2023 07:48
@AnneYang720
feat: subindex init and index for elastic
8c11e5e 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
feat: add parent_id to nested doclist
64a823c 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
feat: get with subindex
867ab36 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
feat: subindex index for hnsw and del
9c66ba8 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@github-actions github-actions Bot added size/m and removed size/s labels Apr 21, 2023
@AnneYang720
feat: _validate_search_field subindex check
b514e3a 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
Merge remote-tracking branch 'origin/main' into feat-subindex
f8a00ed 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
feat: find support subindex
ff7f5d5 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
feat: subindex for find_batched
b3eb8f2 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
feat: have dynamic parent_id
741c0de 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
fix: type judge in _convert_dict_to_doc
def048e 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
feat: subindex del for hnswlib
4cc8521 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
feat: filter_subindex
3bfc976 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
Merge branch 'main' into feat-subindex
9d53e33
@AnneYang720
feat: subindex find that returns two results
3726410 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
Merge branch 'main' into feat-subindex
4959fb0 
Signed-off-by: Anne Yang <evangeline-lun@foxmail.com>
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
fix: no parent_id in nest schema
b6c2bd4 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
fix: mypy
d1b3ada 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
test: subindex tests for elastic
44b656e 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
fix: mypy
8bcf7cc 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
fix: __getitem__
aaef7b8 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@AnneYang720
test: subindex tests for hnswlib
34af8a8 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@github-actions github-actions Bot removed the size/m label Apr 26, 2023
@JoanFM JoanFM linked an issue May 10, 2023 that may be closed by this pull request
Subindex search + navigate and traverse nested structures with id, parent_id and root_id #1235
Closed
@AnneYang720
refactor: minor adjustment
722944f 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@github-actions

github-actions Bot commented May 10, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@AnneYang720 @JohannesMessner
Update docs/user_guide/storing/docindex.md
011178f 
Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Anne Yang <evangeline-lun@foxmail.com>
@github-actions

github-actions Bot commented May 10, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@AnneYang720
docs: fix link
0eb0112 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@github-actions

github-actions Bot commented May 10, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@AnneYang720

AnneYang720 commented May 10, 2023

Copy link
Copy Markdown
Contributor Author

you can find on subindex level, and return results will be subindex docs

Can we change it such that this one return on the root level, not the subindex level? Or is that difficult to do?
The reason I am asking is because normal nested search (not subindex) returns on the root level, so this should do the same, otherwise it will be confusing imo

@AnneYang720 what is the status on this?

This one is hard to change for two reasons:

  1. The find() is recursively called. For example search_field='list_docs__docs__simple_tens', when find is called on subindex docs level, we already lose the root level info.
  2. find_subindex() use find() to get the subindex results as well.

But we can let find() return on the root level and implement another new function like find_subindex_only() as a substitue.

@JohannesMessner

JohannesMessner commented May 10, 2023

Copy link
Copy Markdown
Member

you can find on subindex level, and return results will be subindex docs

Can we change it such that this one return on the root level, not the subindex level? Or is that difficult to do?
The reason I am asking is because normal nested search (not subindex) returns on the root level, so this should do the same, otherwise it will be confusing imo

@AnneYang720 what is the status on this?

This one is hard to change for two reasons:

1. The `find()` is recursively called. For example `search_field='list_docs__docs__simple_tens'`, when `find` is called on subindex `docs` level, we already lose the root level info.

2. `find_subindex()` use `find()` to get the subindex results as well.

But we can let find() return on the root level and implement another new function like find_subindex_only() as a substitue.

I see. Maybe we should drop the subindex functionality in .find() altogether? I would rather it not being there than it being confusing. We could detect if a user passes a field that belongs to a subindex and then raise an error clearly stating that they should use find_subindex instead.
@AnneYang720 @samsja what do you think?

@samsja

samsja commented May 10, 2023

Copy link
Copy Markdown
Member

agree on failing on the find method rather than being confusing. We can always make it compatible with find later. (But not the otherway around if we commit to it now we have to always support it)

But this need to be documented when to pick find when to pick find_subindex

@AnneYang720
feat: remove subindex from find
255d59a 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@github-actions

github-actions Bot commented May 11, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@AnneYang720
docs: update subindex part
12cbc3d 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@github-actions

github-actions Bot commented May 11, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@github-actions

github-actions Bot commented May 11, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

1 similar comment
@github-actions

github-actions Bot commented May 11, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@AnneYang720
docs: docstring for hnswlib _get_items
dcb7863 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@github-actions

github-actions Bot commented May 11, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@github-actions

github-actions Bot commented May 11, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@AnneYang720
fix: fix docstring
6154a18 
Signed-off-by: AnneY <evangeline-lun@foxmail.com>
@github-actions

github-actions Bot commented May 11, 2023

Copy link
Copy Markdown

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@github-actions

github-actions Bot commented May 11, 2023

Copy link
Copy Markdown

📝 Docs are deployed on https://ft-feat-subindex--jina-docs.netlify.app 🎉

@AnneYang720 AnneYang720 merged commit 44977b7 into main May 11, 2023
@AnneYang720 AnneYang720 deleted the feat-subindex branch May 11, 2023 14:14
@samsja samsja mentioned this pull request May 15, 2023
Closed
@JoanFM JoanFM mentioned this pull request May 16, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JoanFM JoanFM JoanFM left review comments

@samsja samsja samsja approved these changes

@JohannesMessner JohannesMessner JohannesMessner approved these changes

Assignees

No one assigned

Labels

area/core area/docs area/testing size/xl

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Subindex search + navigate and traverse nested structures with id, parent_id and root_id

4 participants

@AnneYang720 @samsja @JohannesMessner @JoanFM