Most of the cost of a software product is paid after launch, in the changes it has to absorb. A small number of early decisions determine how expensive those changes are, and they are rarely the decisions teams spend the most time debating. Framework choice matters far less than where the boundaries sit, who owns which data, and whether the system can be observed when it misbehaves.
Put boundaries where the data ownership is
Service boundaries drawn around team structure or around nouns tend to produce distributed systems that require a coordinated release to change anything. Boundaries drawn around data ownership — one component is authoritative for a set of records and everything else reads a copy or asks it — survive reorganizations and feature changes considerably better.
- Identify the entities with independent lifecycles and different consistency requirements; those are candidate boundaries.
- A single writer per record class is worth more than any amount of clever conflict resolution.
- Shared mutable database tables between services are a boundary violation regardless of how the code is organized.
- Start with a modular monolith unless there is a specific scaling, isolation or team-independence reason not to. Extracting a well-bounded module later is routine; unpicking a distributed ball of mud is not.
Decide the consistency story explicitly
Almost every painful production surprise in a distributed application traces back to an implicit consistency assumption. Write the assumption down at design time, in one sentence per interaction, and the failure modes become testable rather than theoretical.
- 1.For each cross-component write, state whether the caller needs the result immediately or can tolerate eventual propagation.
- 2.Make every side-effecting operation idempotent, keyed by a client-supplied identifier. Retries are not an edge case; they are the normal behavior of every network.
- 3.Choose deliberately between a transactional outbox and dual writes — dual writes will eventually lose a message, and it will be the one that mattered.
- 4.Define what the user interface shows during the eventual-consistency window, because the default is a confusing empty state.
If a feature cannot tolerate a duplicated message or a five-second delay, that is an architectural requirement and belongs in the design, not a bug report filed six months after launch.
Background work is a first-class subsystem
Scheduled jobs and queue consumers are routinely treated as scripts, then become the least observable and most fragile part of a system. They deserve the same treatment as request-serving code: versioned deployment, structured logging, metrics, alerting and a documented retry and dead-letter policy.
- Every consumer needs an explicit retry limit and a dead-letter destination somebody is alerted about.
- Long-running jobs need to be resumable, or they need to be split until they are.
- Queue depth and consumer lag belong on the same dashboard as request latency; they are usually the earliest warning of trouble.
- Jobs that mutate customer data need the same audit trail as the API endpoints that do.
Build the diagnostic path before you need it
The question that matters in an incident is not whether the system is up, but why a specific request failed for a specific customer. Answering that requires correlation identifiers threaded through every layer, structured logs, and traces that cross service and queue boundaries. Added after the first serious incident, this work is expensive and partial; added at the start, it is a few days.
- Generate a correlation identifier at the edge and propagate it through HTTP calls, queue messages and background jobs.
- Log structured events, not sentences, so failures can be aggregated by cause rather than grepped.
- Instrument the handful of business outcomes that matter — signups, orders, syncs — not only technical metrics.
- Alert on user-visible symptoms with clear ownership; alerts nobody owns get muted and then ignored.
Write down the decisions and their trade-offs
Architecture decision records are the cheapest durable artifact in software engineering. A page per significant decision — the context, the options considered, the choice, and what would make the team revisit it — is what lets a future team change the system confidently instead of preserving choices whose reasoning nobody remembers.
None of this is exotic, and that is rather the point. Systems that stay changeable are usually built from ordinary components with clear boundaries, explicit assumptions and good diagnostics — not from novel technology choices that were interesting at the time.