-
Notifications
You must be signed in to change notification settings - Fork 418
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
scrubber: add remote_storage based listing APIs and use them in find-large-objects #8541
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. It's worth noting that list_streaming
could be an infinite stream if there is an error. The user of this stream should control the number of next
calls and stop if there is a repeated error.
I have pushed two commits for yielding errors and ending the stream. Note that we already would end the stream when encountering an error: retryable errors would be retried but any other errors we'd yield immediately, and higher up the chain we'd exit at any error immediately (retried errors would not show up if a later attempt is successful). |
3138 tests run: 3017 passed, 0 failed, 121 skipped (full report)Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
58e38a0 at 2024-07-30T09:09:15.740Z :recycle: |
Add two new functions
stream_objects_with_retries
andstream_tenants_generic
and use them in thefind-large-objects
subcommand, migrating it toremote_storage
.Also adds the
size
field to theListingObject
struct.Part of #7547