Data Guide

There are many kinds of software out there, the kind I'm primarily engaged is Enterprise Applications. One of the enduring problems we need to tackle in this world is managing data, since such applications are all about using rapid access to large amounts of data to speed workflows, and provide information to the humans involved. Data management includes a whole host of aspects: storing data in data bases or other data storage mechanisms, moving data between applications, and modeling the data so we can understand what it means.
A guide to material on martinfowler.com about data management.
Over the last few decades, it's become increasingly obvious that data has huge value to the enterprise that holds it. But data is also a complicating factor to the software that manipulates it. We need to change our software regularly to generate more value for our users, but the very data that fuels this value also makes it more difficult to change.
In the early days of agile software development, we were told that we couldn't use evolutionary design techniques for database systems, because database schemas were difficult to change, and thus needed careful up-front design. Fortunately my colleague Pramod Sadalge devised an approach to evolve a database design through small but frequent database migrations. This allows teams to evolve their data the same way that they refactor their code - even if this database is in production.
Over the last decade we've developed and refined a number of techniques that allow a database design to evolve as an application develops. This is a very important capability for agile methodologies. The techniques rely on applying continuous integration and automated refactoring to database development, together with a close collaboration between DBAs and application developers. The techniques work in both pre-production and released systems, in green field projects as well as legacy systems.
by Pramod Sadalage and Martin Fowler
agile refactoring application architecture database evolutionary design
Many important software systems rely on persistent data, stored in relational databases. To evolve this software, and add new features, it's necessary to change the structure of these databases: changing the data schema, its access code, and migrating any data in the database. Fortunately the basic philosophy of refactoring still applies: make very small, behavior preserving changes - and compose them together to make large changes. This book details these database refactorings, with examples of how to do them.
by Scott J Ambler and Pramod J. Sadalage
My colleague Pramod Sadalage has been my authoritive source on data matters for two decades. He developed the evolutionary database techniques that are a core part of our application development approach and co-authored a book on NoSQL databases.
At the beginning of my career, the dominant theme in data management was a single, unified view of data - integrating all aspects of a enterprise's operations with a single data model. (And often a single database.) While that view is still common, I've moved in a different direction, believing that we have to accept that different parts of an enterprise need different data models: they look at different aspects of the enterprise, and even common elements are often viewed in different ways.
Bounded Context is a central pattern in Domain-Driven Design. It is the focus of DDD's strategic design section which is all about dealing with large models and teams. DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships.
team organization requirements analysis application integration domain driven design
Ubiquitous Language is the term Eric Evans uses in Domain Driven Design for the practice of building up a common, rigorous language between developers and users. This language should be based on the Domain Model used in the software - hence the need for it to be rigorous, since software doesn't cope well with ambiguity.
domain driven design
Our aspiration to augment and improve every aspect of business and life with data, demands a paradigm shift in how we manage data at scale. While the technology advances of the past decade have addressed the scale of volume of data and data processing compute, they have failed to address scale in other dimensions: changes in the data landscape, proliferation of sources of data, diversity of data use cases and users, and speed of response to change. Data mesh addresses these dimensions, founded in four principles: domain-oriented decentralized data ownership and architecture, data as a product, self-serve data infrastructure as a platform, and federated computational governance. Each principle drives a new logical view of the technical architecture and organizational structure.
data analytics data mesh
I use the term Application Database for a database that is controlled and accessed by a single application, (in contrast to an IntegrationDatabase). Since only a single application accesses the database, the database can be defined specifically to make that one application's needs easy to satisfy. This leads to a more concrete schema that is usually easier to understand and often less complex than that for an IntegrationDatabase.
application integration database
Many enterprises are investing in their next generation data lake, with the hope of democratizing data at scale to provide business insights and ultimately make automated intelligent decisions. Data platforms based on the data lake architecture have common failure modes that lead to unfulfilled promises at scale. To address these failure modes we need to shift from the centralized paradigm of a lake, or its predecessor data warehouse. We need to shift to a paradigm that draws from modern distributed architecture: considering domains as the first class concern, applying platform thinking to create self-serve data infrastructure, and treating