Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scrubber: add remote_storage based listing APIs and use them in find-large-objects #8541

Merged
merged 9 commits into from
Jul 30, 2024

Conversation

arpad-m
Copy link
Member

@arpad-m arpad-m commented Jul 29, 2024

Add two new functions stream_objects_with_retries and stream_tenants_generic and use them in the find-large-objects subcommand, migrating it to remote_storage.

Also adds the size field to the ListingObject struct.

Part of #7547

@arpad-m arpad-m requested a review from a team as a code owner July 29, 2024 14:27
@arpad-m arpad-m requested review from problame, skyzh and yliang412 and removed request for problame July 29, 2024 14:27
Copy link
Member

@skyzh skyzh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. It's worth noting that list_streaming could be an infinite stream if there is an error. The user of this stream should control the number of next calls and stop if there is a repeated error.

@arpad-m
Copy link
Member Author

arpad-m commented Jul 29, 2024

I have pushed two commits for yielding errors and ending the stream. Note that we already would end the stream when encountering an error: retryable errors would be retried but any other errors we'd yield immediately, and higher up the chain we'd exit at any error immediately (retried errors would not show up if a later attempt is successful).

Copy link

github-actions bot commented Jul 29, 2024

3138 tests run: 3017 passed, 0 failed, 121 skipped (full report)


Code coverage* (full report)

  • functions: 32.8% (7022 of 21389 functions)
  • lines: 50.0% (55881 of 111716 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
58e38a0 at 2024-07-30T09:09:15.740Z :recycle:

storage_scrubber/src/lib.rs Outdated Show resolved Hide resolved
@arpad-m arpad-m enabled auto-merge (squash) July 30, 2024 08:27
@arpad-m arpad-m merged commit 9fabdda into main Jul 30, 2024
61 checks passed
@arpad-m arpad-m deleted the arpad/azure_scrubber_2 branch July 30, 2024 09:00
arpad-m added a commit that referenced this pull request Jul 31, 2024
Uses the newly added APIs from #8541 named `stream_tenants_generic` and
`stream_objects_with_retries` and extends them with
`list_objects_with_retries_generic` and
`stream_tenant_timelines_generic` to migrate the `find-garbage` command
of the scrubber to `GenericRemoteStorage`.

Part of #7547
arpad-m added a commit that referenced this pull request Aug 5, 2024
…large-objects (#8541)

Add two new functions `stream_objects_with_retries` and
`stream_tenants_generic` and use them in the `find-large-objects`
subcommand, migrating it to `remote_storage`.

Also adds the `size` field to the `ListingObject` struct.

Part of #7547
arpad-m added a commit that referenced this pull request Aug 5, 2024
Uses the newly added APIs from #8541 named `stream_tenants_generic` and
`stream_objects_with_retries` and extends them with
`list_objects_with_retries_generic` and
`stream_tenant_timelines_generic` to migrate the `find-garbage` command
of the scrubber to `GenericRemoteStorage`.

Part of #7547
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants