The AI Rabbit Hole

I’ve gone deep down the AI rabbit hole

Not the “use ChatGPT” rabbit hole, and not the “call OpenAI APIs for my webapp” rabbit hole, but the deep, wide, and evergreen open-source model rabbit hole powered by local compute.

It started with a harmless download of Open WebUI and Ollama. “Hey, it’s cool, I can run a 7B model locally on my laptop!”

Flash forward 18 months. I now have dual-RTX rigs humming under my desk, the circuit breakers in my home office trip if I run evaluation benchmarks while making toast, and my house doesn’t need central heating in the winter because a llama-3.1-70B model is actively warming the studs in the wall.

It escalated quickly: from LangChain experimentation and basic tool calling, to LangGraph orchestrations, writing custom Model Context Protocol (MCP) servers, and finally building “Deep Agents” that run recursive loops to solve tasks.

Through this descent, I’ve learned a lot about what LLMs are (and what they absolutely are not), how we implement “reasoning” in software, and how much of that reasoning is just clever illusion.

The Anatomy of the “Reasoning” Illusion

When you read academic papers on AI, they throw around terms that sound like cognitive science breakthroughs: ReAct, Reflexion, Tree of Thought, OODA loops. It makes you feel like we’ve simulated a cerebral cortex in a python script.

Great marketing, but the underlying mechanisms are grounded in standard programming patterns.

Behind the cognitive branding lies a set of standard programmatic loops.

1. ReAct (Reasoning + Acting)

ReAct is presented as a paradigm where the model “thinks” before it acts. Tactically? It’s a standard while True: loop.

You write a system prompt that says: “You must respond in the format: Thought: [your thoughts] followed by Action: tool_name.”

Then your Python wrapper does something like this:

while True:
    response = call_llm(prompt)
    if "Action:" in response:
        tool, args = parse_action(response)
        result = execute_tool(tool, args)
        prompt += f"\nObservation: {result}"
    else:
        break # The model decided it was done

Here is the control flow of that standard while True: loop visualized:

graph LR
    Start(["User Input / Prompt"]) --> LoopStart[Call LLM with context]
    LoopStart --> Response{"Response contains 'Action:'?"}
    Response -- Yes --> Parse[Parse Action and Args]
    Parse --> Exec[Execute Tool]
    Exec --> Obs["Append 'Observation: Result' to Context"]
    Obs --> LoopStart
    Response -- No --> End(["Return Final Answer"])

That’s it. It’s string parsing and regex. We took a basic loop, slapped a cognitive label on it, and called it a day.

2. Reflexion (Self-Correction)

Reflexion sounds like the model going through a zen-like state of self-realization. In reality, it’s just calling the LLM a second time and saying:

“Hey, here is the absolute garbage you just generated. Look at the output, identify three ways it is wrong, and rewrite it.”

The model doesn’t “know” it made a mistake. It just predicts the next most likely tokens that correspond to a correction based on the prompt’s instruction to criticize itself.

Here is the control flow of the Reflexion pattern visualized:

graph LR
    Start(["Initial Input"]) --> Gen["Call LLM: Generate Response"]
    Gen --> Eval{Evaluate Output}
    Eval -- "Has Flaws" --> Critic["Critic Prompt: 'Identify flaws and rewrite'"]
    Critic --> Gen
    Eval -- "Correct / Approved" --> End(["Return Final Response"])

3. Tree of Thought

This is touted as a branching decision-making framework. Practically, you generate 3 or 5 completions in parallel (paying 5x the token costs), pass those completions to a simple evaluator prompt (or a hardcoded scoring function), and throw away the ones that score poorly. It’s evolutionary selection, but with API calls and a high credit card bill.

Here is the control flow of the Tree of Thought pattern visualized:

graph LR
    Start(["Initial Input"]) --> Breadth["Generate N Completions in Parallel"]
    Breadth --> Evaluate["Evaluate Each Completion"]
    Evaluate --> Filter["Filter / Score Responses"]
    Filter --> Branch{Keep Top Branches?}
    Branch -- "Yes" --> Breadth
    Branch -- "No" --> End(["Return Best Response"])

Observations and Hot Takes

1. LLMs Are Pretty Dumb By Themselves

Next-token prediction works, and it works damn well, but without tooling, external services, and access to structured, processed, and properly stored data, the LLM itself is a party trick.

It’s a fancy Plinko machine. You drop a token in the top, it bounces off a billion parameters, and a dad joke pops out the bottom. But it doesn’t know what day it is, it can’t check your calendar, and it has no concept of state unless we build the rails to feed that state back into it.

If you isolate a state-of-the-art model from the outside world, you quickly realize how fragile its “intelligence” is:

Stateless Amnesia: They have no memory. Every single API call is a brand new initialization of the universe. If you want them to remember that their name is “Jarvis” from one chat message to the next, you have to pack that chat history into the context window and pay for those tokens over and over. They are essentially goldfishes with a terminal interface.
Math is a Vibe, Not a Calculation: They don’t have an arithmetic logic unit (ALU) in their neural weights. If you ask an LLM to multiply 4,812.39 * 982.11, it doesn’t do math. It just guesses what a number that “feels” like the correct product looks like based on its training patterns. It’s only when we hook them up to a Python interpreter tool that they can run print(4812.39 * 982.11) and get the actual correct answer.
Confident Hallucination: LLMs don’t have a “fact database.” They are pattern matchers. If you ask them to query a highly specific, obscure command flag for a legacy networking utility, they won’t say “I don’t know.” They will confidently hallucinate a flag like -disable-routing-override because, grammatically and stylistically, it fits the pattern of a valid response.
Execution Paralysis: A raw LLM is just a textbox. It can’t ping a server, it can’t delete a lockfile, it can’t check if a port is open. It is completely paralyzed without standard, classic software engineering exposing API boundaries and tool mappings to it.

2. Agentic Reasoning Is An Illusion

It’s an experience designed by humans to create a magical reaction. But reasoning itself (both in an LLM and in the software built on top of it) is, at best, a human’s interpretation of how humans think.

And guess what: human thought and reasoning is not that complex.

We love to tell ourselves that we are the ultimate creation, the evolutionary unicorns, the pinnacle of all living and breathing permutations. But in the vast majority of cases, we are deterministic, predictable, and simple. We run on our own internal ReAct loops:

Thought: I am hungry.
Action: Open fridge.
Observation: Only expired mustard.
Thought: Close fridge, order pizza.

We are just token predictors running on carbon. When we see an agent print:

[Thinking...]
> Let's analyze the server logs...
> Found error on line 42...
> Applying fix...

We feel like we are watching a digital assistant “reflecting.” In reality, those “thinking steps” are just psychological anchors in the UX. If the UI showed the raw JSON parsing speed or raw tool invocations, we’d realize it’s just a fast database client. But by forcing the model to output its thought logs, we anthropomorphize the database.

3. The Substrate Is The Real Magic

OpenAI, Google, and Anthropic have done an amazing job of building a platform on top of these models. That substrate (the tooling, the Model Context Protocol (MCP) data layer, vector databases, and the connectivity to the apps and services we use in our daily lives) is where the real magic lives. It is standard, solid software engineering that makes the “reasoning” construct possible.

The intelligence isn’t just in the model weights; it’s in the pipes connecting the weights to the world. If the LLM is a natural-language CPU, then the substrate is the motherboard, the system bus, the RAM, and the PCIe lanes. Without it, you just have a silicon chip sitting on a table doing nothing.

When you look closely at a production-grade agentic system, the “intelligence” is almost entirely supported by traditional software engineering:

Model Context Protocol (MCP): The emergence of MCP is a perfect example. We got tired of writing bespoke, fragile JSON wrappers for every tool we wanted to give an agent. MCP creates a standard protocol for servers to expose resources, prompts, and tools to clients. The agent doesn’t “know” how to use your terminal or search your files; it is just talking to an MCP client that maps standard JSON-RPC requests to system commands.
Vector DBs & RAG Plumbing: When an agent answers a highly specific question about your company’s internal codebase, it didn’t “reason” its way to the answer. Standard software engineering did. A cron job chunked your files, pushed them through an embedding API, indexed them in Pgvector, and ran a cosine-similarity query. The LLM just summarized the text file that Postgres handed to it.
Graph Orchestration & State Machines: If an agent successfully builds a multi-step project, it’s not because it formulated a master plan in its “mind.” It’s because it was forced through a LangGraph structure with strict state management, parent-child supervisor nodes, memory checkpoints, and conditional routing. We built a state machine, and the LLM is just the engine that determines which state transitions to trigger.
Structured samplers (JSON Enforcement): How do we prevent agents from breaking our production APIs? We use libraries like Pydantic, Instructor, or Outlines to literally force the model’s token distribution probabilities into a specific schema. We use regex masking on the next token sampler at the llama.cpp/vLLM level to physically prevent the LLM from outputting anything but valid JSON. The model didn’t “learn” to conform; we locked it in a math cage.

My Conclusion

Given all of this, I’ve found myself in some sort of crisis. Every direction I push, I am confronted with the same answer.

AGI, ASI, Agentic Reasoning (whatever you want to call it) will forever be defined and measured by how complex humans believe themselves to be.
And I fear we may be giving ourselves far too much credit.

Oh well. I’m gonna sit with that last thought, ignore my existential dread, and go build a Deep Agent / GraphRAG system for network log inspection now.

At least in this crisis, I can still have some fun!