Startups

Context.ai wants to merge product analytics sensibilities with LLMs

Comment

Illustration representing collecting data for large language models.
Image Credits: a-image / Getty Images (Image has been modified)

Since the release of ChatGPT at the end of last year, we’ve seen companies developing generative AI tooling to help customers interact with their products and services in a more natural way. Yet in many cases, these vendors have no idea how well the underlying large language models are performing, or how good the answers are.

Context.ai launched earlier this year to help companies better understand how users are interacting with their LLMs. Today, the company announced a $3.5 million seed investment to fully develop the idea.

CEO Henry Scott-Green and his co-founder, CTO Alex Gamble, spent several years working at Google: Scott-Green on product and Gamble as a software engineer. Together, they recognized the need for a service that measures how well these models are behaving, and there was very little tooling out there to help.

“We’ve spoken to hundreds of developers who are building LLMs, and they have a really consistent set of problems. Those problems are that they don’t understand how people are using their model, and they don’t understand how their model is performing. The phrase that I always hear is that ‘my model is a black box,’” Scott-Green told TechCrunch.

In many ways, it’s not unlike product analytics tools such as Amplitude or Mixpanel, which measure how users are interacting with a product interface such as where they click or how long they stay on a page. In Context’s case, however, it’s about digging into the data generated by the LLM, and figuring out if it is producing truly useful content that helps users answer customer questions. The ultimate goal is building a more effective model.

The way it works is customers share chat transcripts with Context via an API. It then analyzes the information using natural language processing (NLP). The software groups and tags conversations based on topic, and then analyzes each conversation to determine from the signals available if the customer was satisfied with the response.

Contex.ai analyzes the information in chat transcripts generated by generative AI tools, and returns data like this to measure the effectiveness of the information being delivered by the model.
After it analyzes the text from chat transcripts, Context.ai delivers an analysis like this. Image Credits: Context.ai

“We believe there is a big shift happening [with the rise of LLMs], and there’s going to be a huge number of these chat experiences built over the next few years. And in that new world, where there is a huge amount of textual interface that users are engaging with via text, rather than graphical user interfaces, there is a need for a different set of tools,” he said.

They began by building an initial prototype and shared it with early customers and design partners, and have been iterating to improve and refine the product ever since. Scott-Green indicates it is an ongoing process, but they have been generating a lot of interest and have paying customers.

It’s worth noting for those concerned about security and privacy that Context strips out PII at ingestion. It doesn’t use the content for model building or marketing purposes, and it holds content for no more than180 days, after which it is deleted, according to Scott-Green.

The company is small right now, with six employees, but he sees a future with a growing organization, and he believes it’s never too early to be thinking about building a diverse company.

“It’s obviously a challenge that the startup ecosystem has, and the tech ecosystem has in general when it comes to building representative, diverse, inclusive teams. It’s something we both believe strongly in, and I think more importantly, it’s something that we’re both acting on as well, and really making efforts to ensure that we have an inclusive representative diversity [in our employee base],” he said.

Today’s investment was co-led by GV (Google’s venture arm) and Theory Ventures.

More TechCrunch

Google says it’s developed a new family of generative AI models “fine-tuned” for learning: LearnLM. A collaboration between Google’s DeepMind AI research division and Google Research, LearnLM models — built…

LearnLM is Google’s new family of AI models for education

The official launch comes almost a year after YouTube began experimenting with AI-generated quizzes on its mobile app. 

Google is bringing AI-generated quizzes to academic videos on YouTube

Around 550 employees across autonomous vehicle company Motional have been laid off, according to information taken from WARN notice filings and sources at the company.  Earlier this week, TechCrunch reported…

Motional cut about 550 employees, around 40%, in recent restructuring, sources say

The keynote kicks off at 10 a.m. PT on Tuesday and will offer glimpses into the latest versions of Android, Wear OS and Android TV.

Google I/O 2024: Watch all of the AI, Android reveals

It ran 110 minutes, but Google managed to reference AI a whopping 121 times during Google I/O 2024 (by its own count). CEO Sundar Pichai referenced the figure to wrap…

Google mentioned ‘AI’ 120+ times during its I/O keynote

Here are quick hits of the biggest news from the keynote as they are announced.

Google I/O 2024: Here’s everything Google just announced

Google Play has a new discovery feature for apps, new ways to acquire users, updates to Play Points, and other enhancements to developer-facing tools.

Google Play preps a new full-screen app discovery feature and adds more developer tools

Soon, Android users will be able to drag and drop AI-generated images directly into their Gmail, Google Messages and other apps.

Gemini on Android becomes more capable and works with Gmail, Messages, YouTube and more

Veo can capture different visual and cinematic styles, including shots of landscapes and timelapses, and make edits and adjustments to already-generated footage.

Google gets serious about AI-generated video at Google I/O 2024

In addition to the body of the emails themselves, the feature will also be able to analyze attachments, like PDFs.

Gemini comes to Gmail to summarize, draft emails, and more

The summaries are created based on Gemini’s analysis of insights from Google Maps’ community of more than 300 million contributors.

Google is bringing Gemini capabilities to Google Maps Platform

Google says that over 100,000 developers already tried the service.

Project IDX, Google’s next-gen IDE, is now in open beta

The system effectively listens for “conversation patterns commonly associated with scams” in-real time. 

Google will use Gemini to detect scams during calls

The standard Gemma models were only available in 2 billion and 7 billion parameter versions, making this quite a step up.

Google announces Gemma 2, a 27B-parameter version of its open model, launching in June

This is a great example of a company using generative AI to open its software to more users.

Google TalkBack will use Gemini to describe images for blind people

Firebase Genkit is an open source framework that enables developers to quickly build AI into new and existing applications.

Google launches Firebase Genkit, a new open source framework for building AI-powered apps

This will enable developers to use the on-device model to power their own AI features.

Google is building its Gemini Nano AI model into Chrome on the desktop

Google’s Circle to Search feature will now be able to solve more complex problems across psychics and math word problems. 

Circle to Search is now a better homework helper

People can now search using a video they upload combined with a text query to get an AI overview of the answers they need.

Google experiments with using video to search, thanks to Gemini AI

A search results page based on generative AI as its ranking mechanism will have wide-reaching consequences for online publishers.

Google will soon start using GenAI to organize some search results pages

Google has built a custom Gemini model for search to combine real-time information, Google’s ranking, long context and multimodal features.

Google is adding more AI to its search results

At its Google I/O developer conference, Google on Tuesday announced the next generation of its Tensor Processing Units (TPU) AI chips.

Google’s next-gen TPUs promise a 4.7x performance boost

Google is upgrading Gemini, its AI-powered chatbot, with features aimed at making the experience more ambient and contextually useful.

Google reveals plans for upgrading AI in the real world through Gemini Live at Google I/O 2024

Veo can generate few-seconds-long 1080p video clips given a text prompt.

Google’s image-generating AI gets an upgrade

At Google I/O, Google announced upgrades to Gemini 1.5 Pro, including a bigger context window. .

Google’s generative AI can now analyze hours of video

The AI upgrade will make finding the right content more intuitive and less of a manual search process.

Google Photos introduces an AI search feature, Ask Photos

Apple released new data about anti-fraud measures related to its operation of the iOS App Store on Tuesday morning, trumpeting a claim that it stopped over $7 billion in “potentially…

Apple touts stopping $1.8B in App Store fraud last year in latest pitch to developers

Online travel agency Expedia is testing an AI assistant that bolsters features like search, itinerary building, trip planning, and real-time travel updates.

Expedia starts testing AI-powered features for search and travel planning

Welcome to TechCrunch Fintech! This week, we look at the drama around TabaPay deciding to not buy Synapse’s assets, as well as stocks dropping for a couple of fintechs, Monzo raising…

Inside TabaPay’s drama-filled decision to abandon its plans to buy Synapse’s assets

The person who claimed to have stolen the physical addresses of 49 million Dell customers appears to have taken more data from a different Dell portal, TechCrunch has learned. The…

Threat actor scraped Dell support tickets, including customer phone numbers