Naveen Rao discovered artificial intelligence through science fiction in middle school, devouring the works of Asimov while his peers in Kentucky were doing whatever kids in Kentucky normally did. Growing up in a family of doctors, he chose a less conventional path, following his early interests in programming and circuit building.
After establishing himself as an engineer, he stepped away to pursue a neuroscience PhD at Brown, finishing in record time. This combination of engineering expertise and biological understanding shaped his approach to AI hardware development at his companies Nervana (acquired by Intel) and MosaicML (acquired by Databricks).
In this conversation, the VP of AI at Databricks breaks down what's actually needed for meaningful progress in machine reasoning (hint: it's not just bigger models), and why deep tech development needs a different playbook than what we're used to.
Join us in the full episode of Barrchives (above), and read on for five highlights from the interview.
In Naveen's Words:
"SaaS product books actually put everyone in the wrong direction for deep tech. It doesn't work that way in deep tech, it's not about customer interviews from day one... You may not be able to know what to build until you understand the problem space well, so understanding the problem space is the key, and then actually going and figuring out how to build that with the right team first... When your innovation is in the tech part of it, first you got to innovate there. You got to build the tech."
Deep tech product development inverts traditional SaaS strategy by prioritizing fundamental research and technical innovation. At Mosaic, this meant building an exploratory organization staffed with academic talent - PhDs and post-docs who could tackle core challenges in neural network training and scalability. Rather than rushing to market validation, the team focused first on understanding emerging technological trends through open-source software releases, engaging with technical builders across sectors from startups to financial institutions. When your innovation is on the tech side, you have to innovate there versus being guided solely by immediate customer feedback.
"If I start with a bridge, probably the first bridge was someone put a slab of stone across some body of water... LLMs are no exception. They're big inscrutable units that everyone kind of mistook for being intelligence really... So Compound AI system is basically saying, well, I want to take this big monolithic thing... break it apart into constituent components... Think an engineered bridge versus a slab of stone. They both do the same thing, but the engineered bridge is precise, and each component is well characterized."
Today's focus is on breaking down monolithic LLMs into compound AI systems with well-characterized components that can be independently developed and tested. This architectural shift enables more precise control and understanding of each component, moving away from treating AI as a black box toward viewing it as an engineered system with definable, testable properties.
"Databases, data warehouses, ETL, those are much more stable things. And you can put down processes that span six months or a year. AI is not stable. It's constantly at your feet and what works today may not be that important in six months... You need to be able to experiment and go down a few different dead end paths when the world is shifting under you versus planning out everything and just executing."
While stable technologies can follow predictable development cycles, AI requires organizations to embrace rapid experimentation and accept that some paths will lead nowhere. At Databricks, this means maintaining fast shipping cycles while simultaneously investing in foundational research to anticipate where the technology is heading. Success comes from balancing customer feedback with forward-looking technical exploration – recognizing patterns in current needs while constantly evaluating new techniques that might better solve tomorrow's problems. This requires building an organization that can move quickly while remaining grounded in deep technical understanding.
"In AI systems, think building systems that can really do true decision-making, counterfactual reasoning, and breaking a problem down with sub-goaling in mind... So essentially giving a machine intention... These neural networks can't do counterfactual reasoning. They don't do simulation of outcomes and make decisions based on that. They're estimators of, LLM specifically, are estimators of conditional probability distributions... AGI is a stupid term because it's not general. It's constrained. It's constrained math."
The next frontier in AI comes down to fundamental breakthroughs in how machines process and act on information. Current systems excel at pattern matching and probability estimation but fall short of true reasoning capabilities. The challenge lies in bridging the gap between human intention and machine action: when users interact with AI, they express intent, but systems lack the ability to break this down into meaningful sub-goals or simulate potential outcomes before acting. This limitation points to a crucial distinction between current AI capabilities and genuine intelligence, suggesting that progress requires rethinking our approach to machine learning rather than simply scaling existing architectures.
.svg)