pageserver: do vectored read on each dio-aligned section once

[yliang412]

Part of #8130, closes #8719.

Problem

Currently, vectored blob io only coalesce blocks if they are immediately adjacent to each other. When we switch to Direct IO, we need a way to coalesce blobs that are within the dio-aligned boundary but has gap between them.

Summary of changes

• Introduces a VectoredReadCoalesceMode for VectoredReadPlanner and StreamingVectoredReadPlanner which has two modes:
  • AdjacentOnly (current implementation)
  • Chunked(<alignment requirement>)
• New ChunkedVectorBuilder that considers batching dio-align-sized read, the start and end of the vectored read will respect stx_dio_offset_align / stx_dio_mem_align (vectored_read.start and vectored_read.blobs_at.first().start_offset will be two different value).
• Since we break the assumption that blobs within single VectoredRead are next to each other (implicit end offset), we start to store blob end offsets in the VectoredRead.
• Adapted existing tests to run in both VectoredReadCoalesceMode.
• The io alignment can also be live configured at runtime.

Testing

See #8779 for a matrix build of the regression test with alignment requirement = [1, 512].

Performance

Benchmark Results

TLDR: No significant difference between using different chunk sizes.

Rollout

• The adjacent-only merge is enabled by default (io_buffer_alignment=0).
• Run Rust unittest running with alignment requirement = [0, 1, 512].

We will test the new chunked vectored read code path in pre-prod later this week after release.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3780 tests run: 3674 passed, 0 failed, 106 skipped (full report)

---

Code coverage* (full report)

• functions: 32.3% (7328 of 22654 functions)
• lines: 50.4% (59285 of 117523 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
2443e9a at 2024-08-28T14:18:55.634Z :recycle:}

[VladLazar]

Generally looks good

---

Are we supporting these two modes to allow for a staged cut-over to dio?

[yliang412]

Are we supporting these two modes to allow for a staged cut-over to dio?

Currently doing perf testing to see if this is mergable without the actual O_DIRECT changes. I will report back findings once I'm done.

[problame]

This review pertains to code that is taken by the Adjacent path, i.e., this review is to ensure we're not regressing anything.

[problame]

Reviewed the ChunkedVectoredReadBuilderInner.

Seems correct, some style comments.