Skip to content

Content-Addressed Storage Vault (Go)

Goal

Build a local storage vault that ingests files, chunks them, hashes content, deduplicates storage, tracks metadata, and verifies integrity over time.

This is core backup plumbing, not a full backup product.

Scope (Strict)

  • Local filesystem only
  • No encryption (yet)
  • No networking
  • No cloud
  • No UI
  • CLI is thin and boring

Focus: correctness, resiliency, and explainable design choices.

High-Level Features

  • File ingestion
  • Fixed-size chunking
  • Content-addressed storage (CAS)
  • Deduplication
  • Metadata index
  • Resumable jobs
  • Integrity verification
  • IO-aware concurrency

Directory Layout

vault/
 ├── cmd/
 │    └── vaultctl/        # CLI entrypoint (thin)
 ├── internal/
 │    ├── ingest/         # file walking, chunking orchestration
 │    ├── chunk/          # chunking logic
 │    ├── store/          # chunk storage (CAS)
 │    ├── index/          # metadata DB (SQLite)
 │    ├── verify/         # integrity verification
 │    ├── scheduler/      # resumable jobs
 │    └── limits/         # IO & concurrency control
 ├── pkg/
 │    └── hash/           # hashing utilities
 └── vault.md

Business logic never lives in cmd.

Core Libraries

Standard Library

  • os, io, filepath
  • crypto/sha256
  • sync, context
  • log/slog
  • database/sql

External

  • SQLite driver:
  • github.com/mattn/go-sqlite3 or
  • modernc.org/sqlite

No worker pool libraries. No magic abstractions.

Storage Model

Chunking

  • Fixed-size chunks (e.g. 4MB)
  • Deterministic chunk boundaries
  • Chunk hash = SHA-256(content)

Chunk Storage Layout

vault/
 ├── chunks/ab/cd/abcdef...
 └── index.db

Chunks are stored once, referenced many times.

Database (SQLite)

Tables (Conceptual)

  • files
  • chunks
  • file_chunks
  • jobs

Must support: - transactions - crash recovery - resumable ingestion

Required Engineering Concepts

Context Propagation

Every operation accepts context.Context.

Example: 

func IngestFile(ctx context.Context, path string) error

IO-Aware Concurrency

  • Limit concurrent file reads
  • Separate CPU-heavy hashing from IO
  • Use buffered channels as semaphores

No uncontrolled goroutines.

Resumability

  • Restart-safe ingestion
  • Skip already stored chunks
  • Resume partially ingested files

Integrity Verification

  • Background verification job
  • Re-hash stored chunks
  • Detect and report corruption

Logging

  • Structured logging with log/slog
  • No printf-debugging

CLI Interface

vaultctl init
vaultctl ingest /path
vaultctl verify
vaultctl stats

CLI is a driver, not the product.

Explicit Non-Goals

  • Encryption
  • Networking
  • REST API
  • Cloud storage
  • Competing with real backup software

Milestones

Week 1

  • Vault initialization
  • Chunk storage
  • Single-file ingestion

Week 2

  • Directory ingestion
  • Concurrency limits
  • SQLite schema

Week 3

  • Resumability
  • Verification job
  • Statistics & reporting

Stop here. Document decisions.

Why This Matters

This project demonstrates: - Chunking strategies - Deduplication models - Integrity guarantees - Failure recovery - Go concurrency under IO pressure

These are core systems and backup-engineering skills.

Guiding Principle

If you cannot explain why a decision was made, the decision does not count.