Appendix A-Production Hardening Reference¶
This appendix consolidates the hardening slices distributed throughout the curriculum. By week 24 the reader's hardening/ template should contain a working example of every section below.
A.1 Build & Release¶
A.1.1 go build flags worth knowing¶
- ** - trimpath`-strips local file paths from the binary. Always on** in release builds; required for reproducible builds.
- ** - ldflags="-s -w"`**-strips DWARF and symbol tables. ~30% size reduction. Only enable for production releases (debugging is harder; core dumps less useful).
- ** - ldflags="-X main.version=v1.2.3"
**-embeds version info. Pair with - X main.commit=$(git rev-parse HEAD)and - X main.buildDate=...`. - ** - buildmode=pie`**-position-independent executable. Required for ASLR on hardened deployments.
- ** - buildvcs=true
**-embed VCS info (default on with modules);go version -m` reads it back. - ** - tags=netgo,osusergo`**-pure-Go DNS/user resolvers. Required for fully static binaries on Linux.
A.1.2 Build tags for cross-platform code¶
- Tags gate file-level compilation. - Common patterns://go:build linux, //go:build !windows, //go:build integration (for slow tests), //go:build debug.
- Avoid runtime runtime.GOOS checks where a build tag would do-the dead-code path costs binary size.
A.1.3 Cross-compilation¶
GOOS=linux GOARCH=amd64 go build -trimpath -o bin/svc-linux-amd64 ./cmd/svc
GOOS=linux GOARCH=arm64 go build -trimpath -o bin/svc-linux-arm64 ./cmd/svc
GOOS=darwin GOARCH=arm64 go build -trimpath -o bin/svc-darwin-arm64 ./cmd/svc
GOOS=windows GOARCH=amd64 go build -trimpath -o bin/svc-windows-amd64.exe ./cmd/svc
zig cc via CGO_ENABLED=1 CC="zig cc -target aarch64-linux-musl" for the simplest setup.
A.1.4 Static linking¶
CGO_ENABLED=0produces a fully static binary on Linux. The default for containerized Go services unless you specifically need CGO (sqlite, libsystemd, etc.).- For services that must CGO: link against musl via Alpine or use
gcc -staticcarefully; glibc-static-linking is fragile.
A.1.5 Reproducible builds¶
- Pin toolchain via
go.modtoolchain go1.22.X. -
- trimpath`.
- Avoid
time.Now()ininit()or `build.go - equivalent. - Build inside a deterministic image: a pinned Alpine,
golang:1.22.X-alpine@sha256:...with content hash. - Confirm reproducibility:
sha256sum bin/svcshould match across machines and builds of the same commit.
A.1.6 goreleaser¶
- The de-facto Go release tool. One config file produces: cross-compiled binaries,
tar.gz/ziparchives, Homebrew tap, Linux packages (deb/rpm), Docker images, GitHub Releases, SBOM, signatures. - Replaces ~500 lines of
Makefile+CI-script glue. Adopt early.
A.2 Linting and Static Analysis¶
A.2.1 golangci-lint baseline configuration¶
A reasonable starting .golangci.yml:
run:
timeout: 5m
go: "1.22"
linters:
disable-all: true
enable:
- errcheck
- govet
- staticcheck
- gosimple
- ineffassign
- unused
- revive
- gocritic
- gosec
- bodyclose
- rowserrcheck
- sqlclosecheck
- nilerr
- prealloc
- unconvert
- unparam
- misspell
- depguard
- contextcheck
- errorlint
- exhaustive
- forbidigo
- goerr113
- testifylint
- tparallel
- thelper
- paralleltest
- fieldalignment
- copyloopvar
- intrange
linters-settings:
errcheck:
check-blank: true
govet:
enable-all: true
depguard:
rules:
domain-purity:
list-mode: lax
files: ["**/internal/domain/**"]
deny:
- pkg: net/http
desc: domain must not import HTTP
- pkg: database/sql
desc: domain must not import SQL
issues:
max-issues-per-linter: 0
max-same-issues: 0
A.2.2 The race detector is non-negotiable¶
- ~5–10× slowdown, ~5–10× memory. - Catches data races by adding a happens-before tracking layer. - Never commit code that has not been tested under - race`.A.2.3 go vet¶
- Subset of
golangci-lint(which runsvetinternally), but the standalone command is fast and honest. - Critical analyzers:
printf,lostcancel,copylocks,loopclosure,nilness,shadow,unsafeptr.
A.2.4 staticcheck¶
- The most rigorous Go linter. Maintained separately from
go vet. Documented atstaticcheck.io. - High-value codes:
SA1015(time.Tickleak),SA1029(context.WithValuecollisions),SA4006(unused write),SA6002(sync.Poolnon-pointer).
A.3 Profiling and Tracing¶
A.3.1 pprof endpoints-production setup¶
import _ "net/http/pprof"
// ...
go func() {
log.Fatal(http.ListenAndServe("127.0.0.1:6060", nil)) // admin port, never public
}()
kubectl port-forward for ad-hoc access.
A.3.2 The pprof commands you will run weekly¶
go tool pprof -http=:0 http://host:6060/debug/pprof/profile?seconds=30 # CPU
go tool pprof -http=:0 http://host:6060/debug/pprof/heap # heap (inuse)
go tool pprof -http=:0 -alloc_objects http://host:6060/debug/pprof/heap # allocations
go tool pprof -http=:0 http://host:6060/debug/pprof/goroutine # goroutines
go tool pprof -http=:0 http://host:6060/debug/pprof/block # block (after SetBlockProfileRate)
go tool pprof -http=:0 http://host:6060/debug/pprof/mutex # mutex contention
A.3.3 runtime/trace¶
- View with go tool trace trace.out.
- Use when pprof doesn't explain a latency stall-trace shows the exact timeline of every G across every P.
A.3.4 PGO (Profile-Guided Optimization)¶
- Run a representative load against your service.
- Capture:
curl -o cpu.pprof http://host:6060/debug/pprof/profile?seconds=60. - Place at
default.pgoin the package containingmain. - Rebuild:
go build -pgo=auto. - Expect ~5–15% throughput win on hot paths. Combine with PGO-update cadence in your release flow.
A.4 Observability Standards¶
A.4.1 Logging¶
- Use
log/slog(stdlib, since 1.21). - JSON handler in production; Text handler locally.
- Per-request scoped logger via
context.Context. - Levels:
Debug(off in prod),Info,Warn(something to watch),Error(a human should look). NeverPanicorFatalfor recoverable errors. - Sensitive-attribute redaction at the handler.
A.4.2 Metrics¶
- Use
prometheus/client_golangwithcollectors.NewGoCollector(collectors.WithGoCollections(...))for Go runtime metrics fromruntime/metrics. - The four golden signals: latency (histogram), traffic (counter), errors (counter), saturation (gauge).
- Never unbounded labels.
A.4.3 Traces¶
- OpenTelemetry SDK + OTLP gRPC exporter.
- Auto-instrument with
otelhttp,otelgrpc,otelsql. - Sampling: head-based (e.g., 1%) for high-QPS services; tail-based (via collector) for systems where rare errors matter most.
A.4.4 The "useful errors" hardening pass¶
- Wrap with
fmt.Errorf("doing X: %w", err)at every layer, preserving%wforerrors.Is/errors.As. - Sentinel errors at domain boundaries:
var ErrNotFound = errors.New("not found"). - Structured errors only when you need typed fields:
type ValidationError struct{ Field, Reason string }withError()method. - Never
panicfor recoverable conditions. Reserve panic for "the program's invariants are violated" (e.g., a nil pointer that should never be nil).
A.5 Memory Tuning¶
A.5.1 The two knobs¶
- `GOGC - heap-growth ratio. Default 100 (next-GC = 2× live). Lower = more frequent GC = less memory; higher = less frequent = more throughput, more memory.
- `GOMEMLIMIT - soft memory ceiling. Default off. Set this in containers to ~90% of cgroup memory.
A.5.2 The setup pattern¶
import _ "go.uber.org/automaxprocs" // honor cgroup CPU
import "runtime/debug"
func init() {
if v := os.Getenv("MEMORY_LIMIT_BYTES"); v != "" {
if n, err := strconv.ParseInt(v, 10, 64); err == nil {
debug.SetMemoryLimit(n)
}
}
}
A.5.3 automaxprocs¶
- Uber's small library that sets
GOMAXPROCSbased on cgroup CPU quota. Without it, a container limited to 0.5 CPUs still sees the host's full CPU count and spawns too many P's. - Adopt by default in all containerized services.
A.6 The Hardening Template¶
By week 24, the hardening/ template should contain:
hardening/
.golangci.yml
.goreleaser.yaml
Dockerfile # multi-stage, scratch or distroless final
Makefile # fmt, vet, lint, test, race, bench, profile
cmd/svc/main.go # idiomatic composition root
internal/
platform/
observability/ # slog + prom + otel + pprof wiring
memlimit/ # GOMEMLIMIT from env
shutdown/ # graceful shutdown helper
ci/
test.yml # fmt + vet + lint + test -race
bench.yml # benchstat against baseline
fuzz.yml # nightly fuzz
release.yml # goreleaser on tag
RELEASE_CHECKLIST.md
RUNBOOK.md
SECURITY.md
THREAT_MODEL.md
This is the artifact that should accompany every Go service you ship after week 24.