Vanta is in an interesting position when it comes to AI: they’re both helping customers navigate how to build their AI securely and compliantly, while also building their own, homegrown AI features for these very same customers. I sat down with Vanta’s CEO and Co-founder (Christina Cacioppo) and their VP of Engineering (Iccha Sethi) to talk about everything Vanta and AI, from how they’ve built their internal products to how the regulatory landscape for AI is shifting.
“One thing I'm very proud of that we've built is integrating our eval system into existing engineering workflows in our CI/CD systems. When I'm coding and I push up new code which impacts an AI feature, automatically in our CI/CD, our LLM-as-a-judge system runs evaluations against different golden datasets we’ve gathered to give the engineers a signal: hey, your AI quality may be dipping, or it may be doing better with this change that you made.”
“We were very proactive early on about writing down our principles around how we build with AI. And one of the primary and top ones is we will not train on customer data. And so I see that in our customers, and I see that in our customers' customers. We work with a lot of AI companies that are working on demonstrating their own policies. And this issue – will you train on my data – comes up a lot. And honestly, I think it is often a blocker, especially at the top of the market and especially with European companies.”
“Big enterprises have concerns about how their data is used. The software vendors building with AI want to satisfy those concerns, but no one quite knows exactly how, like the magical incantation to say to either ask for or provide to provide a bunch of certainty. And so with that, we're starting to see some of the standards bodies create new regulations, like the NIST AI RMF and the US ISO 42001. And I think more of these will gain traction over time as we just kind of figure out how to talk about some of this stuff.
.svg)