From Concept to Workflow: The Realities of Gen AI Integration

Generative AI is like electricity! Just like how electricity can power several appliances like a stove, light, or a fan, similarly gen AI can be leveraged to power multiple use cases from sales to HR, increasing operational efficiency across the company.

But what are the challenges in integrating gen AI into multiple workflows across an enterprise? In this blog, we will go over considerations, processes, and challenges of integrating gen AI into your company workflows.

If you haven’t read already, please read my previous blog on “Gen AI real-world use cases for enterprise” that goes over use cases across multiple departments. Here in this blog we dive into the challenges of integrating gen AI into workflows across the company.

What to consider while integrating gen AI?

There are several considerations while adopting gen AI within your company workflows to ensure you have a successful implementation.

Use case

Start from the department or business user, define the use case, then gauge if and how gen AI can benefit that use case. For example, it could be enabling sales people to derive insights about their customers or ramping up onboarding by giving instant answers for your employees via a chatbot. Read my previous article to review the criteria for selecting use cases.

User experience

You might identify a great use case, but the value is delivered only when it is usable by the user. Consider the right user experience based on the use case (hint: it is not always a chatbot). Is it easily discoverable by the user? Does it have the right elements to make the user successful? Is it cumbersome to use? It could be a native chatGPT like web interface or it could be a co-pilot bolted on to an existing tool, or it could be an existing UI powered by AI in the backend.

Data source integration

It is important to consider the process of connecting your source data to the model. A few questions to ask yourself: Where is your data? What is the format? How frequently does it change? What is the size? How clean is the data? Is your data tabular, unstructured blobs, documents, or requires API integration into an existing tool? Is it distributed across multiple sources?

Security and privacy

Every company is different when it comes to security and privacy constraints. Is there user data like health, finance, or other private data? Do you need to ensure no data leaves your environment? Do you need to clean PII before providing data? With so many different proprietary and open source LLMs, you get to decide on the level of security control over your data.

Governance

As you might already know gen AI is not perfect, yet. It has a tendency to provide hallucinatory responses. How do you ensure that the responses are work appropriate, obeys your policies? Are there specific workflows that ensure which user queries are answered by gen AI and reviewed by humans before sending the responses to users?

Efficacy

Implementing any IT project required a trade-off between quality, time, and cost. It is especially true with a new technology like gen AI. How will you measure the quality of responses? What is the size of the development team to implement gen AI? What is the level of expertise required? If you are outsourcing to a consulting firm, do they have expertise in gen AI? (hint: do not go after named brands, go deeper and review the team background and experience with AI). Finally, do you have a budget for the project?

What is the process of integrating gen AI?

With all the considerations in providing a cost effective gen AI solution, let’s take a look at the process of integrating gen AI with an example.

Let’s take a simple example of providing a sales pitch chatbot that assists sales people with the latest contextual product information. Sales people ask product questions, chatbot provides answers personalized to sales persona, and helps them to pitch the product to customers.

Data

First and foremost, you need to build a data connector, so that you can ingest the data. The data could be in a tool, a database, a blob store, or a document repository. For our sales chatbot use case, let’s assume product documentation is in PDF, slides, and document format in Google drive or sharepoint. You’d then have to build connectors with those tools.

ETL

Once you have the raw data from the source you’d need to extract the right data from multiple data formats, then clean and transform the data so that it can be embedded into a vector store, and retrieved by LLMs (large language models).

In our example, you’d need to extract text content from PDFs, slides, and documents on a continuous basis, divide the text at the right boundaries (a.k.a. chunking, i.e. one paragraph, one page etc, so that context is preserved), then store it as an embedding in a vector store like Weaviate, Pinecone, or ChromaDB.

LLM

Select the right LLM based on your use case. You can either go with GPT3.5/4.0 if you are ok sharing data with OpenAI or you can pick from any of the hundreds of open source models like Mixtral, Llama, Falcon etc. Optionally you can also pick a smaller model from HuggingFace. Let’s say we pick Llama 2 for our sales chatbot use case.

Testing

Just like building any traditional software application, you’d need to define quantitative and qualitative metrics for success from a user’s point of view. These could be response accuracy, latency, availability to name a few. Have a golden set of example queries and responses, define an A/B test frameworks, then refine the workflow from data source through LLM.

In addition to a golden set, have a way to incorporate live feedback from users and adjust the workflow. For our sales use case, we can have specific queries and responses validated by sales people, including a thumbs up and down user experience within the chatbot interface to take live feedback from sales people.

Tuning

Define several knobs and checkpoint across the workflow, review the test results, and then play with the parameters until they meet your success criteria. These knobs could span the entire workflow: Try different data extraction methods, chunking techniques, vector store retrieval methods, prompts, LLMs etc.

For our sales use case, based on your accuracy cost tradeoffs, you might even need to fine tune the model for the AI to understand sales specific terminology and provide responses in a style that resonates with the sales persona.

Deployment

Finally, you need to deploy the use case considering your environment, cloud options, and user interface. The environment could expose minimal knobs to the user or heavy customizations. You can deploy in your private cloud whether it is Azure, AWS, or GCP, considering the scale of concurrent requests from users, cost of running the LLM, and security policies. Regarding the UI, this could be a plugin within an existing tool, a web chat interface, or powering an existing UI.

For our sales use case, let’s say we need to deploy a salesforce plugin, so that the AI can provide guidance right within the sales workflow. For this you’d need to build an app within the salesforce environment.

As you can see above, to build a simple sales assistant, you’d need to build data connectors to Google Drive and/or Sharepoint, extract, transform, load the data into the a vector store, pick the right LLM, define a test framework, tune the entire workflow, and deploy considering security and user experience.

Now imagine this entire workflow at enterprise scale, with several concurrent user sessions, with the product documentation changing frequently, while adhering to your security and compliance policies.

What are the potential challenges?

Yes, it might be easy to build a prototype for a single user, but integrating gen AI at enterprise scale is a different story. Now, think about this entire process for deploying at multiple departments from sales through HR. There are hundreds of trade-offs here that impact time to market, quality, and cost.

If you are embarking on this journey of integrating gen AI in a secure, scalable, cost effective way across multiple departments, then below are a few potential challenges you might face.

Fragmentation

You’ve identified a great use case for support, you build an awesome gen AI internal tool for them. Now, sales also has a gen AI use case. Do you need to build from scratch or can you resume the framework that you have used or support? HR already uses a HR tool and they want to upgrade to a gen AI subscription provided by the HR tool. Now imagine every department asking for a similar upgrade. What is the total cost of such upgrades annually? And if every tool has its own AI, how would you ensure there are no AI silos limiting company wide transformation?

Data orchestration

Your data might come in multiple formats from multiple sources across your company. It could be a SQL database, a tool, or a document repository. How would you ensure the right extraction, chunking, and transformation for text, image, tabular data, code etc? How would you handle multiple sources of data for one use case? How would you ensure continuous syncing across multiple sources every time data changes?

Cost effective LLMs

You figured LLama2 works best for your use case, but it costs a ton for running it in your private cloud, now what? Or, you have tried several models, but none of them is providing the results you want. Now, should you use RAG or fine tune the model or try a different LLM? Moreover, how do you A/B test the performance while changing knobs across data ETL through LLM options?

Maintenance and upgrades

You’ve deployed an end to end solution that meets all your requirements, but how do you ensure that the model stays accurate while your data keeps changing? How do you make sure your end to end workflow from data ingestion through LLM inference doesn’t break? How do you upgrade your model, if there’s a new open source model?

Compliance

Well, your entire workflow is working perfectly, but how do you ensure the AI is not sending inappropriate or hallucinatory messages? How do you ensure there are no security data leaks and user access controls are respected by AI? How do you make sure there is proper governance of the end to end data flow?

Usability

You’ve built a chatbot interface successfully! But when you see usage logs, there are crickets! How do you ensure the UI is usable? How do you ensure that the model learns and improves from user feedback? How do you ensure the user can trust the information provided by gen AI?

Without proper planning, you might end up spending millions of dollars and months in time to market to get one use case out of the door. Then spending both time and money to keep getting accurate results. And then keep reinventing the wheel for every use case across your company.

Conclusion

Gen AI can be leveraged for several use cases across departments such as sales, marketing, support, HR, Operations, and IT. As seen in this blog, there are multiple considerations, processes, and challenges in integrating gen AI at enterprise scale. How do you bring gen AI into your company while avoiding the common pitfalls? That’s the topic of our next blog, where we explore potential solutions for integrating gen AI into your company workflows.