AI SaaS Tools: Don’t Trust, Definitely Verify

Everybody’s trying to sell you AI. Your legal software? AI-assisted. Your customer service? AI-enhanced. We haven’t seen a janitorial service with AI superpowers yet, but I suspect one is on the horizon.

And why not? The latest AI innovations are unbelievably amazing.

Here’s the dirty secret: AI SaaS is mostly selling snake oil. It barely works, doesn’t deliver the promised features, and lags ages behind the state of the art. They are just putting a little AI lipstick on a dubious pig.

I get it. These companies are under tremendous pressure to incorporate the most transformational technology the world has seen since the dawn of the Internet. Their customers, shareholders, and probably their janitors are asking them why they haven’t started using AI yet. There’s a lot of pressure.

Purchasing SaaS has always required a healthy skepticism. With AI, you’ll need to be smarter and sharper than ever.

As a leader, you face a dilemma. You can’t afford to ignore AI or wait on the sidelines while your competitors improve their business. Yet you want to avoid spending piles of money and getting locked into a product that doesn’t deliver. Many SaaS companies are still learning, which is understandable. But you don’t want to be the sucker paying for their education.

We recently worked with a vendor that showed us an incredible AI-backed product, backed by amazing promises. I won’t name them, or even the space they are in, because they seemed well intended. The AI features of their tool sounded great, and the demo looked solid. We announced we were trialing it in front of our whole company, and people were excited! It sounded amazing. There was genuine delight about adopting a SaaS tool company-wide, which last happened to me approximately never.

Then we tried it.

It was slow. It was inaccurate. Perhaps worst of all, it was just not very bright. The stuff we were cobbling together ourselves just to learn the technology was working better than this production service. The more we used it, the more it became apparent that they’d taken their old product, slapped some ChatGPT buzz on it in a half-baked interface, and shipped it as something new. We rolled it out to 20 eager beta testers anyway to be sure we weren’t just being overly picky.

It was met with resounding apathy and disappointment.

Fortunately, we had negotiated a trial period where we could cancel our contract – and that’s what we did.

But not everybody’s puffing vaporware. We found another tool (in a different problem space) that looked equally promising. They also had an amazing demo, and they also got the whole team excited.

But this time, it was legit.

After some months of experimenting with Github Copilot, IDE plugins, and an internal ChatGPT-powered tool, we’ve chosen Sourcegraph’s Cody as the AI tool for our software team. I could not be happier with our decision.

From the beginning of the conversation, their team was kind, transparent, and helpful. They readily agreed to set up a demo server that indexed all of our code, so we could see how it would work for us, and set up a shared Slack channel for when we had questions.

As we continued the conversation and got our hands on the tools, they were honest about what did and didn’t work well. They shared their roadmap with us so we could know whether our top priorities were coming soon. They demonstrated a sophisticated understanding of LLM technology. Unlike some partners, we could have a technical discussion without feeling like we were teaching them their business. That’s probably why, even in the brief time we’ve been using it already, we’ve seen them roll out significant improvements, addressing some of our biggest requests.

But it’s hard to switch tools. It was important to us that we work with someone who could not just deliver now but would be poised to be the leader for many years to come. To that end, they did a great job explaining why their static code analysis background was the “peanut butter and chocolate” to work with modern LLM technology. That code analysis and search product was already mature and useful – the AI just unlocked new capabilities.

Not every SaaS vendor is a Sourcegraph. Working with both them and their unfortunate counterparts has helped us set standards for what we look for in future AI SaaS vendors. In addition to all of the usual best practices for working with SaaS vendors, we’ll look for: