The extravagant promise is within reach – everyone says so. All you need is a subscription to an AI model, point it at your data, and you can get almost any information you can think of from it without a programmer!  Just ask away, and AI will figure it out.

So, you go buy your subscription, play with pulling data from the internet, ask a few questions, and you’re getting cool responses!  So, time to let it in on your ERP’s data!

And after burning through a huge allocation of tokens, you find that it’s not quite so simple.  Most of your allocation of tokens for the week is gone, and you have a report of your top 10 customers by sales.

Now, it’s worth noting that this list is readily available to you in your ERP or your basic BI tool with no tokens to spend.  In fact, most of the examples I see from sales pitches or conference presentations fall into this category: things that took forever, used a ton of tokens, but that are also readily available already, without AI.

Now of course, there is satisfaction in having asked a natural language question and getting a response.  Knowing that AI interpreted your question, found the data in your ERP, aggregated it, and sent you a response is cool!  It’s new and fun. And it’s a necessary step on the learning curve as well.  This is the “Hello World” of AI.

But then you ask it for something a little more complex – what are my top 10 customers by gross margin?

ERPs are replete with options and flags and settings that control things, even something as simple as gross margin.  For example, do you include rebates in your margin calculation, and if so, what types of rebates, and rebates of what status (e.g. reconciled, pending, etc)?

Did your AI agent figure that out and follow the ERP’s definition?  Did it get confused between gross margin and gross profit?

Fairly quickly you realize that you have to fill in a lot of blanks and give AI a lot of instructions on what your definitions are.  Your executive, who you can’t (and shouldn’t) expect to learn the intricacies of filtering a complex report in your BI solution, is not going to write an AI prompt that tells the AI agent exactly how to calculate your gross margin.  And gross margin is a simple one…

One approach that everyone using AI needs to master is building effective markdown files.

Markdown files (commonly referred to as an MD file because of their .md file extension) give the LLM important information about the data environment and how it should behave.  This is information that a user then can assume the LLM understands, so they don’t have to repeat it in conversations.

The type of information you include in the MD files, and the structure of the MD files, will grow over time, and you should consider them to be an ongoing “development” effort.  Here are some of the types of information you should consider including in your MD files:

  • Synonyms – terms users use and what they correspond to in your data
  • Database structure
    • Table and view names and definitions
    • Table relationships, foreign keys
    • Location and use of policy flags
  • Rules the agent should follow
    • What it can and cannot do
    • What it should do with uncertainty
  • Instructions on the tone or style of the responses
  • Institutional data it should assume to be factual

Also, MD files are evaluated with each session, and that process consumes tokens, so the content should be succinct and stored in simple to “read” formats, plain text instead of HTML for example.

Additionally, when working with data as broad as an ERP’s, you should use multiple MD files, with a top level that refers to other MD files for different sets of data (like AR vs AP vs GL).  Segmenting them out will allow the LLM to select only the MD files that are relevant to the specific question being asked, instead of having to load the entire set of information for all questions.

Because these files are fundamental to the LLM providing accurate and consistent information, save them, store them in a source control environment, and protect them.

As you find AI returning incorrect data or misinterpreting a question, use that as an opportunity to permanently document within the MD file the clarifications needed so that it gets it right the next time (as opposed to only correcting it in that session’s prompt).  And of course, you can ask your AI agent to update the MD file for you – just double-check it.

If you are working with an ERP’s or BI vendor’s data, ask them if they have a good starting sample MD file you can implement to save you time and effort – then make it your own by incorporating your own directions, synonyms, etc.

Before long, you will be on a path to good results from your LLM.