Monorepo Management with Git – Best Practices and Workflows

From Qiki
Revision as of 12:25, 7 October 2025 by Ryan (talk | contribs) (7.1 Merge conflicts across modules)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Monorepo Management with Git – Best Practices and Workflows

1) Introduction

What is a monorepo?

A monorepo is a single Git repository that stores many projects (apps, services, libraries, tools) together.

Example: apps/web, apps/mobile, libs/ui, libs/auth, infra/terraform.

Why teams use it

  • One place for all code → easier discovery and code search.
  • One set of tools → shared linters, formatters, CI jobs.
  • Atomic changes → update many modules in one commit.
  • Strong reuse → shared libraries stay in sync.



2) Benefits & Drawbacks

When a monorepo is a good choice

  • You share code across many apps.
  • You do frequent cross-module refactors.
  • You want one CI system and one set of rules.
  • You prefer one version of dependencies per language.

Potential challenges

  • Repository size can become very large.
  • Full clone can be slow (we fix this below with partial clone + sparse-checkout).
  • Conflicts can affect many teams if boundaries are unclear.
  • Ownership and permissions are trickier (use CODEOWNERS and docs).

Rule of thumb

If modules are tightly related and often change together → monorepo is helpful.

If modules are unrelated, have different lifecycles, or strict isolation → consider multiple repos.



3) Repository Setup

3.1 Folder structure examples

Option A: Apps + Libs (common)

.
├─ apps/
│  ├─ web/
│  ├─ mobile/
│  └─ api/
├─ libs/
│  ├─ ui/
│  ├─ auth/
│  └─ data/
├─ tools/            # scripts, generators, code mods
├─ infra/            # IaC / Terraform / k8s manifests
├─ docs/
├─ .github/          # CI workflows (GitHub Actions) or .gitlab-ci.yml
├─ package.json / pyproject.toml / etc.
└─ README.md

Option B: Services by domain

.
├─ services/
│  ├─ billing/
│  ├─ identity/
│  └─ notifications/
├─ shared/
│  ├─ logging/
│  ├─ metrics/
│  └─ test-utils/
└─ ...

Tips

  • Put a README.md in each module (apps/*, libs/*) with: purpose, owners, how to build/test.
  • Keep module boundaries clear: what is public API vs. internal.
  • Store module metadata (owners, build/test commands) in a small file (e.g., module.yaml).

3.2 .gitignore setup

Root .gitignore (example):

# OS junk
.DS_Store
Thumbs.db

# Editors
.vscode/
.idea/

# Build outputs
/build/
/dist/
/out/
/coverage/
/logs/
/tmp/

# Languages/tools (add more as needed)
node_modules/
*.log
*.tsbuildinfo
.venv/
__pycache__/
target/          # Rust
bin/
obj/

# Keep empty dirs tracked
!.gitkeep

Per-language ignores can also live inside each module (e.g., apps/web/.gitignore) if needed.

Why?

Keeping build outputs and local files out of Git keeps the repo clean and fast.

3.3 Tips for subfolders and shared libraries

  • Keep shared code in libs/. Publish a clear API; avoid deep imports between modules.
  • Use language workspace features:
    • JS/TS: npm/yarn/pnpm workspaces.
    • Python: editable installs (pip install -e .), Poetry workspaces.
    • Go: multi-module or workspaces as needed.
  • Add CODEOWNERS at the root to route reviews by path.
  • Consider versioning shared libs internally (e.g., tags or changelogs) so consumers know what changed.



4) Managing Large Monorepos with Git

4.1 git sparse-checkout (concept + quick start)

Concept

Only materialize selected folders in your working tree. You see and build only what you need.

Steps (cone mode is easiest):

# 1) Partial clone + no checkout yet (faster)
git clone --filter=blob:none --no-checkout <REPO_URL> monorepo
cd monorepo

# 2) Enable sparse-checkout in "cone" mode (fast path-based)
git sparse-checkout init --cone

# 3) Choose the folders you want
git sparse-checkout set apps/web libs/ui

# 4) Checkout the branch
git checkout main

Common commands

# Add more folders later
git sparse-checkout add apps/api

# Replace the set (switch modules)
git sparse-checkout set apps/mobile libs/auth

# Reapply patterns after branch changes
git sparse-checkout reapply

# Turn it off and get full tree (careful with size)
git sparse-checkout disable

Why?

You avoid checking out huge directories you don’t need → faster status/build/test.

4.2 Partial clone (--filter=blob:none)

Concept

Download commits and trees, but skip file blobs until you actually need a file. Git will lazily fetch the blob on demand.

Examples

# Fresh clone
git clone --filter=blob:none <REPO_URL>

# Pull with filter (keep future fetches light)
git pull --filter=blob:none

# Fetch with filter
git fetch origin main --filter=blob:none
git switch main

Why?

Saves bandwidth and disk space, especially for large file histories.

💡 Best combo: Partial clone plus sparse-checkout gives the biggest speedup.

4.3 git worktree for multi-branch work

Work on multiple branches or reviews at the same time without re-cloning.

# From your main checkout (e.g., ~/src/monorepo)
# Create a new working tree in a sibling folder on a new branch
git worktree add ../monorepo-doc-fix -b docs/fix origin/main

# List worktrees
git worktree list

# Remove when done (from the main repo)
git worktree remove ../monorepo-doc-fix

Combine with sparse-checkout inside each worktree:

cd ../monorepo-doc-fix
git sparse-checkout init --cone
git sparse-checkout set docs tools

4.4 Switch between modules efficiently

Approach A: one-liner with sparse-checkout

# Switch focus to apps/mobile + libs/auth
git sparse-checkout set apps/mobile libs/auth

Approach B: helper script

Create scripts/use-module.sh:

#!/usr/bin/env bash
set -euo pipefail
mod="$1"
git sparse-checkout init --cone >/dev/null 2>&1 || true
git sparse-checkout set "$mod" libs/shared
echo "Now focused on: $mod (and libs/shared)"

Usage:

./scripts/use-module.sh apps/web

5) Branching Strategy

5.1 Recommended: Trunk-Based Development (TBD)

  • Short-lived branches (hours or 1–2 days).
  • Small PRs with fast reviews.
  • Feature flags for incomplete work instead of long-lived branches.
  • Protected main branch with required checks.

Example branch names:

feat/apps-web-header
fix/libs-auth-token-refresh
docs/infra-runbook

Example commit message (Conventional Commits style helps in monorepos):

feat(apps/web): add sticky header to product page

5.2 Isolate module-specific changes

Only stage and commit files in the target module:

# Stage only module A
git add apps/web

# Show diffs for that module
git diff -- apps/web

# Commit
git commit -m "fix(apps/web): correct cart subtotal"

This keeps history clean and makes reviews easier.



6) CI/CD Integration Tips

6.1 Fetch only needed folders in CI

Idea: Check out only the paths used by a job (faster checkout, faster build).

GitHub Actions example

name: build-web

on:
  push:
    paths:
      - 'apps/web/**'
      - 'libs/ui/**'
      - '.github/workflows/build-web.yml'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          # Limit working tree to the paths we care about
          sparse-checkout: |
            apps/web
            libs/ui
          sparse-checkout-cone: true
      - name: Speed up network
        run: git fetch --filter=blob:none
      - name: Install deps (example for JS workspaces)
        run: npm ci --workspaces=false && npm ci --workspace apps/web --workspace libs/ui
      - name: Build
        run: npm run -w apps/web build
      - name: Test
        run: npm test -w apps/web

GitLab CI example

build_web:
  image: node:20
  rules:
    - changes:
        - apps/web/**/*
        - libs/ui/**/*
  script:
    - git config --global advice.detachedHead false
    - git sparse-checkout init --cone
    - git sparse-checkout set apps/web libs/ui
    - git fetch --filter=blob:none
    - npm ci --workspaces=false
    - npm ci --workspace apps/web --workspace libs/ui
    - npm run -w apps/web build

Other tips

  • Cache per-path (apps/web/.cache, libs/ui/.cache) to avoid cross-module cache pollution.
  • Split jobs by path to keep pipelines parallel and fast.



7) Common Issues & Troubleshooting

7.1 Merge conflicts across modules

Why it happens

Two branches touched the same files or shared APIs changed without coordination.

Fix (step by step)

  1. Pull latest main.

    git switch your-branch
    git fetch origin
    git rebase origin/main
    
  2. Resolve conflicts locally (use your IDE’s merge tool).

  3. Run affected tests only (see below).

  4. Commit the resolution.

    git add .
    git rebase --continue
    
  5. Push.

    git push --force-with-lease
    

Tip: Enable Git’s conflict reuse:

git config --global rerere.enabled true

This lets Git remember how you resolved a conflict before.

7.2 Sparse-checkout sync issues

Symptoms

  • Files appear missing after a branch switch.
  • Build tools complain a folder is not present.

Fix

# Ensure cone mode + reapply patterns
git sparse-checkout init --cone
git sparse-checkout reapply

# If patterns changed, set them again
git sparse-checkout set apps/web libs/ui

If you really need a full tree (rare), disable:

git sparse-checkout disable

Gotcha: Sparse patterns are per-working-tree. If you use git worktree, set patterns in each worktree.

7.3 Partial clone “missing blob” during build

This is normal. Git fetches file blobs on demand. Ensure network access and try:

git fetch --filter=blob:none --recurse-submodules=no

7.4 Accidentally staged the whole repo

Unstage everything, then add only your module:

git reset
git add apps/mobile

8) Best Practices Summary

8.1 Naming conventions

  • Branches: feat/<path>, fix/<path>, docs/<path>. Example: feat/apps-web-header.
  • Commits: Conventional Commits with module scope: feat(apps/web): …, fix(libs/auth): ….
  • Tags/Releases (optional): tag per app if you release apps separately (e.g., web-v2.3.0).

8.2 Scripts for common developer tasks

Create top-level scripts (Bash, Makefile, npm scripts, or Taskfile):

scripts/
  use-module.sh         # set sparse-checkout to a module
  affected.sh           # list modules changed since main
  fmt.sh                # run formatters across repo
  test.sh               # run tests for selected modules

Example “affected” (very simple):

#!/usr/bin/env bash
base=${1:-origin/main}
git diff --name-only "$base"...HEAD | cut -d/ -f1-2 | sort -u

8.3 Keep the repo clean and fast

  • Prefer partial clone + sparse-checkout for devs and CI.

  • Use Git LFS for large binary assets (images, media, big models).

  • Do periodic cleanup:

    git gc --prune=now
    
  • Avoid committing build outputs and lockfiles from other tools accidentally.

  • Add CODEOWNERS and module READMEs to document ownership and boundaries.

  • Keep PRs small and focused on one module when possible.



9) Step-by-step Quickstart (copy/paste)

  1. Clone fast
git clone --filter=blob:none --no-checkout <REPO_URL> monorepo
cd monorepo
  1. Focus on a module
git sparse-checkout init --cone
git sparse-checkout set apps/web libs/ui
git checkout main
  1. Create a short-lived feature branch
git switch -c feat/apps-web-header
  1. Work only in your module
# edit files under apps/web and libs/ui
git add apps/web libs/ui
git commit -m "feat(apps/web): add sticky header"
git push -u origin HEAD
  1. Open a PR (small, fast review), then merge.



10) Collaboration Habits

  • Use presets: shared prettier/eslint/black/gofmt configs at the root.
  • Keep docs up to date: each module’s README shows build, test, and owners.
  • Respect ownership: CODEOWNERS routes reviews to the right team.
  • Communicate breaking changes in shared libs; consider changelogs.
  • Prefer feature flags over long-lived branches.



Appendix: Useful Commands (cheat sheet)

# Show only changes under a path
git status -- apps/web
git diff -- apps/web

# Stage only a path
git add -- apps/web

# Reapply sparse-checkout patterns (after switching branches)
git sparse-checkout reapply

# Add another folder to your sparse set
git sparse-checkout add libs/auth

# Create another working tree on a different branch
git worktree add ../mono-exp -b exp/new-idea origin/main