The Creator of Pandas on AI, Apache Arrow, and the Future of Software Engineering – Wes McKinney

This transcript and summary were AI-generated and may contain errors.
I joined Dan Beech on the Data Engineering Central podcast for a conversation that covered my whole career arc and then spent most of its energy on what AI is doing to software engineering.
We started with Apache Arrow, which Dan had picked up through PyArrow around 2015, back when almost nobody was talking about it. I described Arrow as one of those projects where the reaction on first hearing about it splits two ways: people say “that’s obviously a good idea, why doesn’t it exist already,” and then immediately follow with “you’ll never get anyone to adopt it.” The standards problem is real — there’s the xkcd comic about 13 competing standards becoming 14 — but Arrow worked because it was the only credible attempt at the problem, and because adoption compounds. The format hasn’t changed much in five or six years. We mostly had to wait. Now DataFusion alone underpins something like 30 or 40 companies, Cloudflare acquired Arroyo, Apple uses DataFusion Comet to accelerate Spark, and dbt’s new engine is built on Arrow and ADBC.
That led into what I think is the most interesting technical claim in the conversation: projects like DataFusion and DuckDB are the ones most resistant to being vibe-coded. LLMs are averaging machines. I compared the difference to injection-molded plastic toys versus a fine Swiss watch. I’ve talked to Hannes Mühleisen about this and he just laughed at the idea. What I did try to convince Hannes of is that he should use AI for the unpleasant parts — generating test cases — rather than hand-coding everything.
The trust argument connects to that. A lot of open source value is a track record of taste and reliability built over years, and people relying on you to guard against supply chain attacks and fix serious bugs. I said I feel like my credibility is on the line every day, which means I can’t ship vibe-coded slop — I have to hold to the standard I set before. Doing that in a world where you can generate 10,000 lines of code a day is genuinely hard. I also noted that open source ends up being more of a cult of personality than it should be: people still call me the pandas guy even though I haven’t been heavily involved in development since 2013, and they should be thanking and sponsoring the large group who actually maintain it.
On origins: I was a gamer, got into GoldenEye 007 speedrunning in the late nineties, and maintained speedrunning fan sites by hand on GeoCities without knowing how to program — which is a little painful to think about now. Watching Derek Kisman automate all of it with C++ gave me my first look at what a genuinely good programmer could do. I studied pure math at MIT, felt inferior to the child prodigies around me, learned enough Java to decide programming was terrible, and only got serious in 2007 at a quant hedge fund where I expected differential equations and found Excel spreadsheets. The reaction that started everything was simply: this should be code. Not MATLAB, because you’d have to buy a license for every production server — it should be open source and in a language built for systems.
We talked about why pandas ended up built on NumPy — it was the only game in town, and interoperability was mandatory to reach Matplotlib and SciPy — and how that forced anything non-numeric into Python objects inside NumPy arrays, with all the indirection and overhead that implied. That limitation became a direct motivation for Arrow. I also admitted pandas was not especially well engineered internally. It was focused on human ergonomics, and the internal messiness created real maintenance burden. Projects don’t have to be architecturally perfect to succeed.
Asked whether data problems have actually changed, I said we’re fighting many of the same ones: moving data from A to B, converting formats, loading into memory, running queries efficiently. I didn’t anticipate pandas becoming the load-bearing infrastructure it became, and there is probably more pandas code being written by LLMs now than was ever written by humans. My summary of the last two decades of progress is that we mostly just needed to listen to the database people — read the papers, go to the conferences, implement what’s in them. We went through a painful NoSQL era and produced scalable systems that a good single-threaded implementation on one machine could trounce, which is the point of the McSherry “Scalability! But at what cost?” paper. Now the modern data stack is essentially ruled by database technology and everything is columnar.
On advice for new graduates I’m still optimistic, though I had a small existential crisis last year about whether I had a future as a software engineer. My view is that AI tools filter people by agency. They hand you a B-plus, middling approach to every problem, and if you don’t bring taste and judgment you end up in what I’ve called the agentic tar pit. So the study time moves from learning to write Python or Java toward software and data architecture — knowing how systems should be designed, and being able to articulate precisely what you want. If you arrive at the prompt and can’t explain the problem or recognize when the agents are heading the wrong way, you’re in deep water with no compass.
The failure mode I keep hearing about is decision fatigue. Work used to be socialized — sprint planning, story points, shared conviction built with colleagues, and then you just did the implementation. Now all of that agile process is compressed into plan mode, and there’s nobody else in the room. Engineers are making ten times as many decisions a day. AI will not fix that. Relatedly, giving AI to someone without taste or experience mostly turns them into a slop cannon, and a lot of businesses struggling to see returns added AI while the people stayed the same.
We finished on token economics. I mentioned that at API rates I’d used $37,000 in tokens over the previous 30 days, far more than I actually