Fix negative replication delay metric

[tristan957]

In some cases, we can get a negative metric for replication_delay_bytes. My best guess from all the research I've done is that we evaluate pg_last_wal_receive_lsn() before pg_last_wal_replay_lsn(), and that by the time everything is said and done, the replay LSN has advanced past the receive LSN. In this case, our lag can effectively be modeled as 0 due to the speed of the WAL reception and replay.

[tristan957]

I've also given thought to something along the lines of:

DO $$
DECLARE
  receive_lsn pg_lsn;
  replay_lsn pg_lsn;
  replication_delay_bytes numeric;
BEGIN
  SELECT pg_last_wal_replay_lsn() INTO replay_lsn;
  SELECT pg_last_wal_receive_lsn() INTO receive_lsn;

  SELECT pg_wal_lsn_diff(receive_lsn, replay_lsn) INTO replication_delay_bytes;
END $$;

Then we can guarantee that receive LSN will always be larger than replay LSN at the expense of a tiny bit larger replication, the amount of WAL replayed between the time replay_lsn is set and when receive_lsn is set. In practice, this is likely negligible.

[github-actions]

3150 tests run: 3029 passed, 0 failed, 121 skipped (full report)

---

Code coverage* (full report)

• functions: 32.5% (7022 of 21597 functions)
• lines: 49.9% (55877 of 111995 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
759186d at 2024-07-30T22:48:37.765Z :recycle:}

[tristan957]

Discussion with Laurenz Albe, here: https://dba.stackexchange.com/questions/341186/postgres-hot-standby-has-a-negative-replication-delay

[tristan957]

The more I think about this, the more I think the current PR is probably the best way to solve this problem. A negative value really means that we are processing WAL so fast, that there is essentially no lag.

I guess if you're a really smart user, would seeing a small non-zero value be more believable than seeing 0?