NEP 41 — First step towards a new datatype system — NumPy Enhancement Proposals

First step towards a new Datatype System
https://mail.python.org/pipermail/numpy-discussion/2020-April/080573.html and https://mail.python.org/pipermail/numpy-discussion/2020-March/080495.html
This NEP is second in a series:
NEP 40 explains the shortcomings of NumPy’s dtype implementation.
NEP 41 (this document) gives an overview of our proposed replacement.
NEP 42 describes the new design’s datatype-related APIs.
NEP 43 describes the new design’s API for universal functions.
Datatypes in NumPy describe how to interpret each element in arrays. NumPy provides int, float, and complex numerical types, as well as string, datetime, and structured datatype capabilities. The growing Python community, however, has need for more diverse datatypes. Examples are datatypes with unit information attached (such as meters) or categorical datatypes (fixed set of possible values). However, the current NumPy datatype API is too limited to allow the creation of these.
This NEP is the first step to enable such growth; it will lead to a simpler development path for new datatypes. In the long run the new datatype system will also support the creation of datatypes directly from Python rather than C. Refactoring the datatype API will improve maintainability and facilitate development of both user-defined external datatypes, as well as new features for existing datatypes internal to NumPy.
The user impact section includes examples of what kind of new datatypes will be enabled by the proposed changes in the long run. It may thus help to read these section out of order.
One of the main issues with the current API is the definition of typical functions such as addition and multiplication for parametric datatypes (see also NEP 40) which require additional steps to determine the output type. For example when adding two strings of length 4, the result is a string of length 8, which is different from the input. Similarly, a datatype which embeds a physical unit must calculate the new unit information: dividing a distance by a time results in a speed. A related difficulty is that the current casting rules – the conversion between different datatypes – cannot describe casting for such parametric datatypes implemented outside of NumPy.
This additional functionality for supporting parametric datatypes introduces increased complexity within NumPy itself, and furthermore is not available to external user-defined datatypes. In general the concerns of different datatypes are not well-encapsulated. This burden is exacerbated by the exposure of internal C structures, limiting the addition of new fields (for example to support new sorting methods [new_sort]).
Currently there are many factors which limit the creation of new user-defined datatypes:
Creating casting rules for parametric user-defined dtypes is either impossible or so complex that it has never been attempted.
Type promotion, e.g. the operation deciding that adding float and integer values should return a float value, is very valuable for numeric datatypes but is limited in scope for user-defined and especially parametric datatypes.
Much of the logic (e.g. promotion) is written in single functions instead of being split as methods on the datatype itself.
In the current design datatypes cannot have methods that do not generalize to other datatypes. For example a unit datatype cannot have a .to_si() method to easily find the datatype which would represent the same values in SI units.
The large need to solve these issues has driven the scientific community to create work-arounds in multiple projects implementing physical units as an array-like class instead of a datatype, which would generalize better across multiple array-likes (Dask, pandas, etc.). Already, Pandas has made a push into the same direction with its extension arrays [pandas_extension_arrays] and undoubtedly the community would be best served if such new features could be common between NumPy, Pandas, and other projects.
The proposed refactoring of the datatype system is a large undertaking and thus is proposed to be split into various phases, roughly:
Phase I: Restructure and extend the datatype infrastructure (This NEP 41)
Phase II: Incrementally define or rework API (Detailed largely in NEPs 42/43)
Phase III: Growth of NumPy and Scientific Python Ecosystem capabilities.
For a more detailed accounting of the various phases, see “Plan to Approach the Full Refactor” in the Implementation section below. This NEP proposes to move ahead with the necessary creation of new dtype subclasses (Phase I), and start working on implementing current functionality. Within the context of this NEP all development will be fully private API or use preliminary underscored names which must be changed in the future. Most of the internal and public API choices are part of a second Phase and will be discussed in more detail in the following NEPs 42 and 43. The initial implementation of this NEP will have little or no effect on users, but provides the necessary ground work for incrementally addressing the full rework.
The implementation of this NEP and the following, implied large rework of how datatypes are defined in NumPy is expected to create small incompatibilities (see backward compatibility section). However, a transition requiring large code adaption is not anticipated and not within scope.
Specifically, this NEP makes the following design choices which are discussed in more details in the detailed description section:
Each datatype will be an instance of a subclass of np.dtype, with most of the datatype-specific logic being implemented as special methods on the class. In the C-API, these correspond to specific slots. In short, for f = np.dtype("f8"), isinstance(f, np.dtype) will remain true, but type(f) will be a subclass of np.dtype rather than just np.dtype itself. The PyArray_ArrFuncs which are currently stored as a pointer on the instance (as PyArray_Descr->f), should instead be stored on the class as typically done in