Accelerating Biological Analysis with Xarray

Thursday, June 5th, 2025 (over 1 year ago)
SciPy 2025 Sprint! If you will be attending the SciPy 2025 conference join our Xarray for Biology Sprint! It's not required but if you are interested let us know via this form.
Hi! I'm Ian, a multimodal microscopist, and the new "Xarray Community Developer." I am funded by the Chan Zuckerberg Institute to support the use of Xarray in biological and biomedical applications.
I believe Xarray can be immensely valuable to the biological community. I discovered Xarray during my PhD where I used it to keep track of complex microscopy image metadata and execute larger-than-memory jobs on the cluster. It is so useful that it is now my default tool for all analyses in both structural and microbiology. Using Xarray means I no longer ask myself questions like: Which axis was what? Where did the exposure times go? I know I changed the media at 32 minutes, but what time index was that?
You have likely experienced similar challenges when working with array data (e.g., microscopy images, genomic sequences, or anything else you might currently analyze using NumPy). Xarray is a great tool for dealing with those challeneges. However, while Xarray is heavily used in the geosciences community, where it was initially developed, biological applications are only recently gaining steam.
To understand why and figure out a roadmap for the future of Xarray in Biology I have spent the last several months interviewing scientists and software developers across many fields of Biology. I also contributed biology related fixes to Xarray and Zarr, attended conferences and studied existing use cases of Xarray.
This post contains a summary of my findings. I will introduce the concepts of Xarray at a high level with biological context and give examples where it is already in use. Then, based on the interviews I conducted I will explain what has limited adoption. Finally, I will describe what we (Biologists and Xarray contributors) can do to increase adoption.
Biological data almost always has rich context and metadata associated with the actual measurements. For example, sample conditions, genetic modifications, buffer composition per well in a plate, time points, and spatial coordinates. While NumPy is a powerful tool, it has limitations when it comes to working with these datasets.
Selecting data based on array indices rather than the physical values can be confusing. You know you switched the growth medium at 32 minutes, but which array index is that? Similarly, keeping track of which dimension is which can be difficult without labels. You have a five-dimensional array, but there are a few transposes in this code from last week, and now you don't remember which axis is which in the output. Managing a collection of multiple related arrays with slightly different shapes can be tricky. Imagine sending data into a batch job and trying to keep segmentations and raw images together. Or maybe you've tried to follow poorly commented analysis from an interesting paper and gotten lost in the details?
Xarray’s data structures provide a solution to all these problems. To quote the docs,
Xarray introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like multidimensional arrays, which allows for a more intuitive, more concise, and less error-prone developer experience.
Xarray accomplishes this by providing the DataArray, Dataset and DataTree. Which keep track of it metadata and are coordinate aware (e.g., can hold the values of timepoints in seconds for you).
Here is an example of what a DataArray of a small stack of microscopy images might look like:
Looking at the repr, you can probably understand a lot about the experiment without any explanation. You no longer need to mentally keep track of transposes, axis labels, and metadata because you can always check the current state! This Xarray feature makes it easier for you to develop your analysis and also means that your code will be more easily understandable by others. Or if you receive a notebook from someone, you will have an easier time deciphering it. You can also use the metadata to expressss operations, writing dim="time" is much clearer than axis=0
When you collect data, you likely collect more than just one type. You may have sequence data in addition to growth curves, fluorescence data, or some kind of spectral data. If you collect multiple arrays of data that share some dimensions (e.g., time) but not all, how do you keep them organized? They can't fit in one array as they don't share all their dimensions!
Happily, Xarray solves this problem for us as well! The Dataset object stores collections of related DataArray's. For example, for our microscopy image stack, we might also create a segmentation layer that will share some of the dimensions of our image (time, X, Y, [Z]) but not all (channel, [Z]) and it would be great to keep them together.
Finally, the DataTree object allows you to keep trees of related datasets, for example, a representation of multiscale images allows for natural support of OME-NGFF data.
Having labeled dimensions and the ability to organize multiple arrays is often reason enough to start using Xarray. But there are more benefits:
As great as Xarray sounds, it does have limitations. Xarray is an array library; it's in the name! So, if your data is tabular and the tabular ecosystem is working well for you then keep using that!
To better understand if you have tabular or array data, I recommend reading Ryan Abernathy's excellent post Tensors vs. Tables. If you have array-like data and have been using Pandas to work with it and want to try out Xarray you can do so easily with pandas_df.to_xarray().
Many parts of biology generate array data. And scientists in some areas are already benefiting from Xarray.
Given the benefits of switching to Xarray, why aren't more biologists using it? Is it secretly not as good as this blog claims? I spent February and April 2025 reaching out