Schema driven development

I see two meanings of “schema-driven development” - literal and general.

The literal meaning is the capitalized Schema-Driven Development (SDD) process, which applies to the development of back end services / REST APIs. The general meaning is the idea of using “schemas” as a first step of a development process.

Schema-Driven Development for APIs

In this sense, advocates state that one must start with a schema - e.g., OpenAPI (or similar) spec for REST APIs. The argument is that the schema is the contract under which all concerned parties operate, and how would any one individual know what to expect, or how to behave in various circumstances without a contract?

I love the use of the word contract here, because it evokes great analogies of other contexts where the contract (schema) is also the starting point. Most apt is the use of architectural drawings when building a home - the number of tasks and people that pivot independently around those drawings is massive, not to mention the ability to estimate time and cost with a fair degree of accuracy up front.

In the context of developing APIs, the schemas serve as the pivot point around which all other work can be done independently and concurrently. With a clear, precise, unambiguous schema, a development timeline can look like this:

Problems Schema-Driven Development solves

I have worked on many features where a schema wasn’t the starting point. Image a back end developer that works in a silo on a new service, marks their ticket as “closed” when done, and hand their shiny new endpoint off to the next team. This rarely ends well. This timeline looks more like this:

Inappropriate initial implementations causes early refactoring

When the next developer in this waterfall starts to work on their part of the feature, in my experience there are questions - e.g.:

• Can we change the structure of the data?
• What does field X mean?
• Can field Y be an integer rather than a string?

If lucky, the back end dev is able to make changes for you - but this is a slow down. If unlucky, the back end dev isn’t able to make changes - either because of higher priority work, or because other teams have already started ingesting the endpoint.

Ambiguous / missing schemas cause overcomplicated client implementations

As an iOS engineer, I see the consequences of not having [good] schemas during PR review. When I’m looking at a proposed data model I’ll see some code smells indicative of a service having no schema. One example is when all the fields in a structure are optional:

struct Thing: Codable {
  let name: String?
  let value: String?
  let something: String?
  let fubar: String?
}

Clients need two branches of code, plus two sets of unit tests for each optional value.

Another example is when date, numeric, or boolean fields are represented as strings:

struct: Thing: Codable {
  let name: String
  let startDate: String?
  let postCount: String?
  let hasAvailability: String?
}

When these code smells appear in a PR, it starts a conversation that usually leads to the discovery of missing or ambiguous schemas, which is technical debt that needs to be paid off. A healthy response is to pay off the debt as soon as it is discovered, and the client PR can be updated with more appropriate data models once the schema has been corrected. If not paid off, the client is adding to the technical debt by propagating the ambiguity into the client code, and should at least properly handle the various code paths.

Missing / inaccurate schemas cause crashes

I have seen crashes in large numbers due to some piece of data changing in a service - missing fields, changed data types, etc. - all where there was no schema in place at the time of development, or the “schema” was just a documentation artifact created early on, and never updated.

Faux Schema-Driven Development

Schema-Driven Development is not:

• JSON: examples are not schemas.
• Post-hoc documentation: schemas are the driver, and should never be driven by code - either lazily (by manually writing a schema after the service has been implemented), or actively (by using a generator to generate schemas from code)

Benefits of Schema-Driven Development

• Enables teams to work independently
• Encourages thinking ahead of time rather than by the seat of your pants. Solving problems ahead of time is worth 10x more than solving them in production.
• Reduces micro-decisions (the kind that no one bubbles up (and maybe not even aware) as they make them, but oftentimes surface later as a subtle bug).
• Fosters communication and collaboration - cross-team, and within teams

Guidelines for writing good schemas

Start discussing data types and structures in the ideation / planning phase of a project.

The first implementation task of a project should be to write the schema.

A schema should be an unambiguous declaration of all data types and structures.

• Mark fields that will always have a value as required.
• Use the most specific data type available (see Data Types - note that Swagger became OpenAPI).
  • When using a string, add more constraints whenever possible.
    • Use a format, like date date-time, uuid, etc.
    • Use enums when appropriate.
    • Use a pattern if string values can follow a RegEx.
  • For number and integer types, include min/max values.
• Lean into adding descriptions.
  • At the very least, document date format & time zone expectations.
• Include example values.

Read more about Schema-Driven Development

• API design is stuck in the past (Buf, November 2020)
• Schema-driven Development (Matt Rickard, April 2022)
• Schema-driven development in 2021 (Eddy Nguyen, 99designs, 2021)
• What is OpenAPI? Likely the most used standard for specifying REST APIs

Schema-driven development as an idea

The second meaning of schema-driven development to me is the idea of starting a project with the process of designing your data structures and semantics. This involves taking a schema-driven approach to things that are not REST APIs.

I have seen this work really well in practice. On my current team, I have adopted a process of having a 30-60 minute discussion to think about and nail down data types and/or methods whenever more than one platform is involved. Half of the time we can even do this over Slack, asynchronously. This works great, and the team really enjoys not having to revisit things because these weren’t thought about ahead of time.

As an individual contributor, I also start by thinking about the data inputs & outputs of a new module, the required behaviors, and what the module’s surface area (the API) looks like for other developers. I try to take the perspective of someone using the module, and think about the code I would want to write as a user. I’m lazy and want to write one-liners without a lot of ceremony, but also want to have more options when needed. This perspective informs the shape of a module’s APIs and data structures so that there are low-touch entry points, yet flexible methods to gain progressive control of the module. Once I have the surface area figured out - i.e., the schema, or contract - it’s secondary how the internal implementation is put together.

Conversely, I have also seen things go awry when a schema-driven approach was not taken. A common occurrence is when the iOS and Android engineers work on things independently, there can be a discrepancy between them. Some are unavoidable platform differences - especially in the UI layer, but some are not.

Large discrepancies, like completely different architectures (think object-oriented vs. functional), are hard to solve without refactoring, and many times you’re almost forced to just live with it. When this is something like an in-house SDK that will be documented and used by many people outside of your team, this will be a problem.

It’s very hard to build new layers of abstraction on top of existing systems when those systems are not consistent with each other.

Smaller discrepancies, like something as simple as two different approaches to localizing a specific piece of text, can lead to one platform needing to revisit their implementation to bring the two platforms into alignment. Not a huge deal, but still requires a context switch back to what was already considered to be “solved”.

Summary

I place a very high value on planning - the kind of planning that is either literally Schema-Driven Development or generally embraces the idea of it. A small amount of time spent doing this kind of planning can:

• Save a lot of time by not having to refactor things later, because you’ve thought out the edge cases up front;
• Reduce technical debt, because your up-front considerations mean you don’t have to live with the consequences of poor design choices;
• Reduce code complexity, because you’ve made things as unambiguous as possible ahead of time;
• Make your team faster, because you’re not spending extra time to refactor things;
• Make your team happier and more collaborative, because everyone’s opinions and perspectives are important and valuable when planning.