Using AI to upgrade Enterprise Search

Since the advent of ChatGPT, the vision of a chatbot capable of addressing any question related to internal knowledge has captivated companies worldwide. However, this aspiration faces significant hurdles due to inherent limitations of chat-based assistants: they can only work with a handful of document passages at a time, easily hallucinate details, and cannot meet the demands of high-privacy environments.

Looking closer, what companies really need is a new kind of enterprise search solution: one that not only retrieves the right information, but can also execute meaningful work by processing the retrieved information – just like an intern would do.

We are working on this vision, leveraging many of the recent advances in GenAI. Here's how.

Better retrieval: bridge the gap with web search

Enterprise Search solutions have historically lagged behind Web Search in effectiveness, and often fail to retrieve high-value unformation, even within the best productivity tools around (Dropbox, Google Drive, OneDrive, etc).

However, advancements in AI — and more recently in GenAI — can help bridge the gap.

Websites want to be found, documents don't.

On the web, every site employs SEO strategies to ensure it is discoverable and deemed authoritative by search engines. All web pages use the same structured format — HTML — making the components of the page obvious (title, level of headings, metadata, …). Web pages are also interconnected through hyperlinks, enabling the identification of authoritative sites with the following core idea: “A website is important if other important websites are connected to it”. This recursive approach to determining a site's importance, famously known as PageRank¹, leverages the web's interconnected nature.

Files, on the other hand, are isolated (not interlinked), disorganised, and for the vast majority have no structured format. They don't care about being found. Nobody is doing their “SEO” to enhance their discoverability.

Use AI to do “the SEO of documents” to make them discoverable

Advances in AI can help your documents discoverability, as if full-time employees were doing “SEO” on them. It can be framed as a two steps process:

1. Document understanding: AI enables to process any file type, and extract more information from the file content.
   Show how we handle different file formats
2. Linking documents together: GenAI can be leveraged to detect implicit references, either as entities mentionned in both documents — think NER here, like organisations, people, places — or as semantic proximity that can result of comparing each document against every other one⁴.
   Show how documents can be tied together

These are two avenues of research that we are exploring at Dotprod AI, to dramatically improve the discoverability of corporate documents.

Use AI to understand queries and retrieve relevant information

The typical search box in productivity tools only supports keyword search from documents titles, and sometimes from the content in the files, with great limitations for PDFs, scans and images.

Experience proves keywords are not enough to effectively search within millions of corporate documents. People often don't remember the exact keywords, or need to find related information (not only exact matches). This is why we experimented with advanced AI retrieval methods (see below) giving users more flexibility in expressing what they are looking for in natural language.

With these techniques we can retrieve much more relevant documents aligned with the user query. We then needs to decide how to order them.

Use AI to simulate queries to optimise search algorithms

Public search engines benefit from billions of queries to optimise their ranking of documents. Every click on a Google search result is a signal that this site was the most relevant to the query, adding this (query, site) pair to a dataset later used to improve ranking algorithms.

Enterprise Search companies don't have such datasets, as they don't have the scale of public search engine, and also because they can't really collect one and guarantee privacy at the same time.

We intend to bridge the gap here with synthetic datasets generated by a range of LLMs configured with different user profiles. While indexing the documents of a company we can simulate millions of relevant (query, document) pairs, as if thousands of employees were searching documents all day long!

As demonstrated lately, textbook-quality synthetic datasets⁵ are much more efficient than real-word-noisy datasets to train SLMs. So, these would be perfect candidate to train our custom AI models for each customer, while still preserving the utmost level of privacy and confidentiality for their corporate data.

Handle information processing tasks

Beyond Search: Knowledge Work Applications

Once you can effectively retrieve relevant information from your files, you can use it as the input to different Generative-AI models to perform actions on them.

We believe that every employee will soon routinely delegate information processing tasks to AI models, as they would to an intern. It will be a bit like using Excel today, delegating heavy calculations to a machine. And it will certainly reduce the need for ad-hoc software.

This is our mid-term goal: progressively support such “knowledge work” applications and offering teams around the world a close to infinite pool of interns ready to crunch tasks for them.

If you read this far and are interested, let's chat about your use case! We are looking for design partners to work on high-value use cases.