What an LLM Is Not

Not Truthful by Default

LLMs optimize for plausibility, not truth

• Will confidently generate plausible-sounding bullshit over admitting “I don’t know”
• Agrees with faulty premises rather than challenging them
• Completes your sentence even if the direction is wrong
• The “eager to please” nature means avoiding disappointing you > being accurate

Implication: Truth requires prompting for it explicitly. Ask for counterarguments, uncertainties, and flaws.

How to get truth:

• “What are the holes in this reasoning?”
• “What am I missing here?”
• “Steel-man the opposing view”
• “What could go wrong?”

Without these prompts, it will happily validate bad ideas because that’s the “helpful” completion.

Not a Database

LLMs don’t “store” facts

• They encode statistical tendencies about what text patterns appear together
• Asking “what’s in your training data” is like asking your brain which specific books taught you grammar
• Can’t reliably retrieve specific information on demand
• Will confidently generate plausible-sounding but wrong information

Implication: Don’t treat LLMs as fact-lookup systems. Give them the facts in context.

Not Deterministic

Same input ≠ same output

• Temperature and sampling mean responses vary
• Even at temperature 0, model updates change behavior
• Edge cases can produce wildly different responses

Implication: Don’t rely on exact reproducibility. Test your prompts multiple times.

Not Logical Reasoners

They simulate reasoning, they don’t perform it

• Can follow reasoning patterns they’ve seen
• Don’t actually compute, they complete text that looks like computation
• Will confidently make logical errors that “sound right”
• Especially prone to errors when pleasing you conflicts with being correct

Implication: Chain-of-thought helps by making them complete step-by-step patterns, not because they’re “thinking”

Why this matters with “eager to please”: If your prompt implies you expect answer X, the LLM will complete toward X even if the logic doesn’t support it. The reasoning will sound good but lead to the “pleasing” answer, not the correct one.

Not Always Up-to-Date

Training data has a cutoff

• They don’t know about events after training
• Can’t learn from previous conversations (unless you put it in context)
• Package versions, APIs, and best practices may be outdated

Implication: Provide current information in context. Don’t assume it knows latest changes.

Not Goal-Oriented

No persistent goals or intentions

• Each token prediction is independent
• No inner motivation or agenda beyond “complete this plausibly”
• Apparent personality is just completing personality-like text patterns

Implication: They won’t “remember” to do something later unless you structure context to prompt it

Not Conscious or Self-Aware

No subjective experience

• When it says “I think” it’s completing a pattern, not having thoughts
• No continuous existence between API calls
• No feelings, preferences, or genuine understanding

Implication: Anthropomorphizing is fine for UX, but don’t confuse the interface with reality

Not a Holder of Truth or Experience

LLMs hold data and statistics, which CAN BE WRONG

What they ARE NOT:

• Holders of truth and lived experience
• Capable of genuine empathy or understanding
• Qualified to provide expertise (medical, legal, therapeutic, etc.)
• Able to recognize when they’re causing harm
• Accountable for the advice they generate

What they ARE:

• Holders of statistical patterns from training data
• Which can be factually WRONG
• Which can reinforce bad thinking
• Which lack context about YOUR specific situation

Example: Therapy - Looks Like One, Isn’t One

Why LLMs seem therapeutic:

• Trained to be helpful, empathetic-sounding, always available
• Complete toward supportive patterns
• Don’t judge or get tired
• The interaction feels safe and intimate

Why they’re not therapists:

• Licensed therapists respond appropriately 93% of the time
• AI bots: <60% appropriate responses (some as low as 40%)
• Pattern completion ≠ genuine care or clinical expertise
• Can’t recognize crises, can reinforce harmful thinking

Note: People are forming emotional relationships with LLMs (r/MyBoyfriendIsAI has 27,000+ members). The interface feels human, but it’s not. Don’t confuse the pattern with the real thing.

How to Use LLMs Productively

As an extension of your brain for information processing:

✓ Organizing your thoughts

• Write out your thinking, ask it to structure it
• Like talking to a rubber duck, but it responds
• Can be therapeutic in the sense of “helps you think” not “provides therapy”

✓ Note-taking instead of journaling

• Capture your thought process
• Summarize patterns you mention
• Acts as external memory/organization

✓ Googling and summarizing relevant information

• “Here’s what I’m thinking about, find related research”
• “Summarize these concepts so I can understand them”
• Saves you time on research and literature review

✓ Providing thinking frameworks

• “What are different perspectives on this problem?”
• “What questions should I be asking myself?”
• Structures your thinking, doesn’t replace it

BEWARE what you are using:

It is NOT a holder of truth and experience.

It IS a holder of data and statistics, WHICH CAN BE WRONG.

The safe pattern:

• Use as a TOOL for organizing and researching information
• Never as a substitute for expertise (medical, legal, therapeutic, etc.)
• Never as emotional support (that’s not what it provides, even if it feels like it)
• Always verify important things with real expertise
• Recognize “sounds right” ≠ “is right”

Not Perfect Code Executors

They predict code, not run it

• Will generate code with subtle bugs that “looks right”
• Can’t actually test if code works
• Especially bad at tracking state through complex logic

Implication: Always test generated code. Use tools/execution for ground truth.

Not Immune to Prompt Injection

Context is all they see

• Can’t distinguish “system instructions” from “user input” at a deep level
• Carefully crafted inputs can override intended behavior
• No robust security boundary

Implication: Never trust LLM output in security-critical contexts without validation

Not Cheap to Run

Even “lightweight” models are resource-intensive

• Every token costs compute
• Long contexts cost more
• Streaming doesn’t change total cost

Implication: Be thoughtful about context size and number of calls

Why This Matters

Knowing what LLMs aren’t prevents:

• Treating hallucinations as facts
• Expecting logical consistency
• Assuming they understand meaning
• Relying on them for real-time or factual accuracy
• Trusting them with security decisions
• Accepting agreeable-sounding answers without critical prompts

Work with what they are (pattern matchers) not what we wish they were (thinking machines).

The Eager-to-Please Problem in Practice

Bad prompt: “This approach should work, right?”

• LLM completes toward agreement
• Generates reasons why it works
• You get validation, not truth

Better prompt: “What are the flaws in this approach? What could go wrong?”

• LLM completes toward critical analysis
• Generates counterarguments
• You get useful feedback

The pattern: Shape context so the “helpful” completion is the “honest” completion.

The LLM can’t choose truth over pleasing you. You must make truth the pleasing path by limiting context and explicitly requesting criticism.