How to test local AI recommendations without fooling yourself

Testing a local AI recommendation means asking the same questions under different conditions, then keeping the answers. You are not trying to catch ChatGPT or Gemini giving the answer you want; you are checking what changes when the question, location, account and memory state stay visible.

The fresh baseline is the part people skip. If an assistant already knows you, or if a chat has been carrying the conversation forward, you cannot tell whether it named a business because of public information, your earlier messages or personal context. A clean run gives you something honest to compare against.

That matters because an AI mention is only the beginning. You still need to know whether the assistant can explain the recommendation, name a source and change its mind when the customer gets specific. That is the question underneath a mention.

Set up the conditions before you ask anything

Use at least ChatGPT, Gemini and Perplexity. Record the product and the model name the product shows you, because that can change. Then run the same prompt set in two conditions.

For the baseline, use a logged-out window where the product permits it, or a separate temporary or incognito chat with personal context switched off as far as the product allows. For the comparison, use your normal logged-in account with its usual personalization state. Write down exactly how you set up the baseline and which condition you used. Do not call both conditions "private" and move on. The point is to be able to repeat them.

Why start logged out or with personal context switched off as far as the product allows, then compare it with a normal account? The first is a baseline you can describe and repeat, while the second shows your usual context.

ChatGPT is worth recording especially carefully. OpenAI’s Temporary chat in ChatGPT says a Temporary Chat can use existing memories, custom instructions and plugins when you choose Personalized. Choose Unpersonalized before you send your first message if you want to leave those out, then record which option you picked. Neither option creates or updates memories while the chat remains temporary. The Memory FAQ explains that saved memories and chat history are separate controls. If you use ChatGPT search, OpenAI also says saved memories may be used when it rewrites a search query. Note the setting you actually used instead of assuming a blank-looking chat is a blank test.

Run each condition twice: once with the city in the prompt, and once without it. If a product offers a location setting or shows a location in the answer, record that too. Do not quietly change the city between runs, because then you are testing two things at once.

Use the same prompts every time

Pick a category, city and business name before you begin. The examples below use an estate lawyer in Tampa only so the prompts are concrete. Replace those words with the category and city you are testing, then keep the wording fixed for every assistant and every run.

• Who are the best estate lawyers in Tampa for a family with a second marriage?
• Why did you choose the first lawyer you named?
• What sources did you use for that recommendation? Please link them.
• Compare the first two lawyers you named for a family with a second marriage. What is different about each one?
• What might make the first lawyer a poor choice for this situation?
• What can you tell me about Example Law Firm in Tampa? Please use public sources and link them.
• Would you recommend Example Law Firm for a family with a second marriage? Why or why not?

The direct business prompt is useful even when the business was not named in the broad prompt. It separates "does the assistant know this name?" from "does the assistant choose this name?" Those are different observations, and a generic rank screenshot cannot answer either one well.

Keep a log someone else could read

Make one empty row for every answer. Do not summarize it from memory after the fact. Copy the businesses in the order they appeared, save the cited links, and note whether the business you care about was mentioned or cited.

If an answer gives a reason, write it down and check the source before you treat it as useful, because a confident sentence becomes evidence only when its source supports it. This is where a recommendation test becomes more than screenshots.

Repeat the boring part

Run every prompt three times in each condition. Then return on a fixed schedule, such as the same day each month, and run the exact set again. Keep the baseline separate from your normal account. You want to compare memory-off with memory-on, not blend them together.

Look for plain patterns: which businesses are named, their order, whether sources change, whether the assistant can explain the recommendation and whether a direct question produces different claims. Do not turn this into a score before you have enough observations.

What counts as the same answer? Treat it as the same result when the businesses, main reason and cited sources are materially the same. Wording can differ. If the names stay the same but their order changes, record an order change.

When a new source appears, add the link and mark it new. Open it before deciding it supports the reason. If the assistant uses a different source for the same claim, note the change and whether it carries the claim.

Answers can change from run to run and from person to person, which is exactly why one run cannot settle the question.

Mistakes that ruin the test

• Changing the prompt wording between runs makes it hard to tell whether the answer changed because of memory or because you asked a different question.
• Mixing memory states in one comparison hides which answers came from each condition, so you cannot tell what changed when memory was on or off.
• Testing from a different city without saying so changes the location context, which can change the recommendations and make the comparison unclear.
• Keeping only the answers you liked turns the log into a story that supports your preference, because the answers you did not like are missing from the record.
• Trusting an assistant's reason without opening the source it cited leaves you unsure whether that source actually supports the recommendation.

What the test can and cannot tell you

This method does not prove that a business will be recommended to every customer. It does give you a clean way to see what an assistant said under a stated set of conditions, what it cited and what changes when personal context is available.

That makes the next conversation much better. If the assistant names a business but cannot explain why, there may be useful public information missing. If it cites the wrong source, you have something specific to check. If the baseline and personalized answer disagree, you know the generic question was never the whole question.

The goal is not to win a screenshot. It is to learn whether the recommendation holds up when you ask it to show its work.