Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Known issues with git mirrors #2208

Open
DrJosh9000 opened this issue Jul 17, 2023 · 1 comment
Open

Known issues with git mirrors #2208

DrJosh9000 opened this issue Jul 17, 2023 · 1 comment

Comments

@DrJosh9000
Copy link
Contributor

DrJosh9000 commented Jul 17, 2023

This issue exists to billboard known problems with git mirrors across different agent versions.

Checkouts sometimes fail with 'bad object', 'did not send all necessary objects' errors

Issue: Checkouts sometimes fail because the mirror appears to contain the commit needed for a job, but the checkout phase fails with a 'bad object' or '{repo} did not send all necessary objects' error. As a recovery step the agent may try to fetch all refs, but still fails again. This could be due to corruption in the mirror repo, or due to objects removed normally from the mirror but still referenced by the checkout directory.

Affected versions: Unknown, possibly since v3.23.0.

Status: A workaround (#2286) is in v3.52.0.

Workarounds/debugging:

  • Update to the latest agent release.
  • Use the --clean-checkout flag or set the environment variable BUILDKITE_CLEAN_CHECKOUT=true.
  • Delete the affected mirror directory, allowing the agent to make a fresh clone.
  • Manually run git fsck --full in the checkout directory and manually remove any bad objects.
  • In the mirror, manually run git fsck --full, remove bad objects, and then run git remote update to ensure the mirror is not corrupt.
  • Disable git mirrors.

Mirrors of submodules not fetched

Issue: When the agent runs jobs that check out a repository that uses git submodules, the submodules should also be mirrored. The main repository mirror is updated (fetched) to ensure the commit or branch being built was present, but a bug meant that submodule mirrors was not fetched. As a result, submodule mirrors might become out -of-date, missing commits or objects needed by the main repo.

Affected versions: Between v3.44.0 and v3.49.0, inclusive.

Status: Fixed with #2203, which was released as part of v3.50.

Workarounds/debugging:

  • Update to the latest agent release.
  • If each host only runs one agent, include a manual git fetch origin for all submodule mirror directories in your pipeline.
  • Use a custom checkout hook.
  • Disable git mirrors.

Mirrors of submodules pull from wrong remote URL

Issue: When the agent runs jobs that check out a repository that uses git submodules, the submodules should also be mirrored. A bug caused the remote URL in the mirror directory for each submodule to be the wrong URL.

Affected versions: Between v3.44.0 and v3.48.0, inclusive.

Status: Fixed with #2144, but because the submodule mirror might not be re-fetched, it can still contain the wrong git history and objects. The missing fetch is fixed by #2203, which was released as part of v3.50.

Workarounds/debugging:

  • Update to the latest agent release.
  • If each host only runs one agent, update to at least agent v3.49 and include a manual git fetch origin for all submodule mirror directories in your pipeline.
  • Use a custom checkout hook.
  • Disable git mirrors.
@rajatvig
Copy link

rajatvig commented Sep 5, 2024

When using mirrors, we occasionally see

Error: cloning git repository: signal: bus error

which fixes on a retry. Could this be handled the same way fatal: bad object was handled?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants