pageserver: add metric pageserver_secondary_heatmap_total_size

[jcsp]

Problem

We don't have a convenient way for a human to ask "how far are secondary downloads along for this tenant".

This is useful when driving migrations of tenants to the storage controller, as we first create a secondary location and want to see it warm up before we cut over. That can already be done via storcon_cli, but we would like a way that doesn't require direct API access.

Summary of changes

Add a metric that reports to total size of layers in the heatmap: this may be used in conjunction with the existing pageserver_secondary_resident_physical_size to estimate "warmth" of the secondary location.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2172 tests run: 2108 passed, 0 failed, 64 skipped (full report)

---

Flaky tests (4)

Postgres 16

• test_combocid: debug
• test_forward_compatibility: debug
• test_lfc_resize: debug
• test_lagging_sk: debug

Code coverage* (full report)

• functions: 32.4% (7233 of 22312 functions)
• lines: 50.4% (58441 of 115937 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
eae36d1 at 2024-08-20T13:35:44.093Z :recycle:}