Skip to content

Azure.storage.getFiles - ignores paging options. #629

@osher

Description

@osher

Disclaimer: I'm not totally complete with that description, because the problem is bigger,
but lets start with that and have a proper mature discussion.

I basically expected behavior that matches other adapters provided in the package (namely Azure), and kept the same option names.

Here are 3 problems, (all are solved with the PR linked).
All of which consider the following setup:

Consider

  • an azure client initiated with valid credentials
  • a blob container that has
    • 1 file under path a/
    • 5 files under path b/
    • 1 file under path c/

In node shell:

> client = require('pkgcloud').storage.createClient({provider: 'azure', process.env.AZURE_ACCOUNT, process.env.AZURE_API_KEY});1
1
> container = process.env.AZURE_CONTAINER;1
1

Problem one - limit

Scenario

When I list files and provide options.limit and options.prefix

> client.getFiles(container, {limit:2}, (e,f) => console.log('Result', e || f.length));1
1

Expected

I expect only as much files as the provided limit

Result: 2

Found

I get all the files

Result: 7

Problem 2 - paging marker

According to Azure,
A paged list should return an opaque cursor identifier - aka marker - by which paging may continue:

marker - Optional. A string value that identifies the portion of the list to be returned with the next list operation. The operation returns a marker value within the response body if the list returned was not complete. The marker value may then be used in a subsequent call to request the next set of list items.

The marker value is opaque to the client.

See: list-blobs#uri-parameters

While Rackspace can use the file.name as a marker - Azure cannot, while the present API does not allow the marker to float back to the user.

Problem 3 - selector by a prefix

Scenario

When I provide a prefix

> client.getFiles(container, {prefix: 'a/'}, (e,f) => console.log('Result', e || f.length));1
1

Expected

I expect only files under the provided prefix

Result: 5

Found

I get all files

Result: 7

NOTE: If the bucket contains more than 10K files - there is no way to list files beyond the 10Kth file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions