<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[LLMGraph Blog]]></title><description><![CDATA[LLMGraph Blog]]></description><link>https://llmgraph.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 22 Nov 2026 14:32:41 GMT</lastBuildDate><atom:link href="https://llmgraph.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Flowise is shutting down: a migration path for your visual LLM workflows]]></title><description><![CDATA[Flowise announced on July 29 that it's winding down. The GitHub repo gets archived on August 10, npm packages and Docker images are being deprecated, and…]]></description><link>https://llmgraph.hashnode.dev/flowise-is-shutting-down-a-migration-path-for-your-visual-llm-workflows</link><guid isPermaLink="true">https://llmgraph.hashnode.dev/flowise-is-shutting-down-a-migration-path-for-your-visual-llm-workflows</guid><dc:creator><![CDATA[Abaho Katabarwa]]></dc:creator><pubDate>Wed, 05 Aug 2026 00:04:57 GMT</pubDate><content:encoded><![CDATA[<p>Flowise announced on July 29 that it's winding down. The GitHub repo gets archived on August 10, npm packages and Docker images are being deprecated, and August 31 is end of life. The code stays Apache 2.0, and the team's guidance is to fork and maintain it yourself.</p>
<p>If you're running Flowise chatflows in production, that leaves you with two real options: become the maintainer of your own fork (security patches, dependency upgrades, model API churn, all of it) or move to a platform that's actively maintained.</p>
<h2>The stated reason, and why I read it differently</h2>
<p>The Flowise team's stated reason for shutting down is that "rigid workflow low code approach quickly hits the limit when it comes to complexity." I'd frame it differently: visual workflow builders aren't the problem. Abandoned ones are. Most teams using a tool like Flowise chose it precisely because they don't want to hand-maintain orchestration code. That need didn't go away last week.</p>
<h2>What migration actually looks like</h2>
<p>I build <a href="https://llmgraph.ai">LLMGraph</a>, a hosted visual LLM workflow builder, and the concepts map over directly:</p>
<ul>
<li><strong>Chatflows map to Graphs.</strong> Same node-and-edge mental model: models, prompts, logic, and data sources wired together on a canvas.</li>
<li><strong>Deployment is built in.</strong> A finished graph deploys as a public invoke API or an embeddable chat widget in one click. No Docker, no reverse proxy, no self-hosting.</li>
<li><strong>Hosted and maintained.</strong> Model providers, infra, and security patches are our problem, not yours.</li>
</ul>
<h2>Before August 31</h2>
<p>If you want to move a Flowise setup over before end of life, every LLMGraph tier starts with a 14-day trial, and I'm happy to personally help port your chatflows. Email <a href="mailto:founders@llmgraph.ai">founders@llmgraph.ai</a> or ping me in the comments.</p>
]]></content:encoded></item><item><title><![CDATA[Hexagonal architecture in Rust, and why it makes your codebase legible to coding agents]]></title><description><![CDATA[Coding agents are good at local edits and bad at holding a whole system in their head. Ask one to fix a function and it does fine. Ask it to "add Postgres…]]></description><link>https://llmgraph.hashnode.dev/hexagonal-architecture-in-rust-and-why-it-makes-your-codebase-legible-to-coding-agents</link><guid isPermaLink="true">https://llmgraph.hashnode.dev/hexagonal-architecture-in-rust-and-why-it-makes-your-codebase-legible-to-coding-agents</guid><category><![CDATA[Rust]]></category><category><![CDATA[software architecture]]></category><category><![CDATA[AI]]></category><dc:creator><![CDATA[Abaho Katabarwa]]></dc:creator><pubDate>Tue, 28 Jul 2026 21:16:36 GMT</pubDate><content:encoded><![CDATA[<p>Coding agents are good at local edits and bad at holding a whole system in their head. Ask one to fix a function and it does fine. Ask it to "add Postgres support" to a codebase where the database calls are smeared across HTTP handlers, and it starts editing in six places, guessing at each, and quietly breaking the seventh. The problem is not the model. The problem is that the codebase never told anyone, human or machine, where things are allowed to live.</p>
<p>Hexagonal architecture is a way of telling them. It is an old idea (Alistair Cockburn named it "ports and adapters" in 2005) that turns out to be exactly the structure an agent needs: narrow contracts, bounded blast radius, and a spec it can read before it writes. This post walks through what it is, how it looks in Rust, and why the same boundaries that keep code readable for people make it tractable for agents.</p>
<p>There is a companion repo that compiles, tests, and runs: <a href="https://github.com/llmgraph-ai/hexagonal-rust-template">llmgraph-ai/hexagonal-rust-template</a>. Everything below is in there.</p>
<h2>The one rule</h2>
<p>Hexagonal architecture has a lot of vocabulary (ports, adapters, driving side, driven side) but only one rule that matters:</p>
<p><strong>Dependencies point inward. The core depends on nothing.</strong></p>
<p>Your business logic sits in the middle. It defines the interfaces it needs from the outside world (a place to store data, a way to send email) as traits. It does not know or care what implements them. Everything technological (the web framework, the database driver, the JSON) lives at the edge and depends on the core, never the other way around.</p>
<p>Draw it as a picture and it is a set of rings with the arrows all pointing to the center:</p>
<pre><code>server  -&gt;  adapters  -&gt;  application  -&gt;  domain
                \______________________________^
</code></pre>
<ul>
<li><strong>domain</strong> is the core: business types and the ports (traits).</li>
<li><strong>application</strong> is the use cases, written against those ports.</li>
<li><strong>adapters</strong> are the edge: an HTTP handler, a Postgres client, an in-memory fake.</li>
<li><strong>server</strong> is the composition root: the one place that wires a specific adapter to the core.</li>
</ul>
<h2>In Rust, the compiler enforces it</h2>
<p>Most languages document the dependency rule and hope. Rust lets you make it a compile error. Put each ring in its own crate, and give each crate a dependencies list that only points inward. Now the core cannot import an adapter, because the adapter's crate is not in its dependency graph. The rule stops being a convention that erodes over time and becomes a fact the compiler checks on every build.</p>
<p>Here is the core. Note what is not imported: no <code>axum</code>, no <code>sqlx</code>, no <code>serde</code>. Just the standard library and two small utilities.</p>
<pre><code class="language-rust">// crates/domain/src/lib.rs
pub struct ShortCode(String);

impl ShortCode {
    // The inner String is private, so the only way to get a ShortCode is to
    // parse one. An invalid code cannot exist anywhere in the system.
    pub fn parse(raw: &amp;str) -&gt; Result&lt;Self, DomainError&gt; {
        let ok = (3..=32).contains(&amp;raw.len())
            &amp;&amp; raw.chars().all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-');
        if ok { Ok(Self(raw.to_string())) } else { Err(DomainError::InvalidCode) }
    }
}
</code></pre>
<p>The core also defines the ports: the interfaces it needs from the world, as traits.</p>
<pre><code class="language-rust">// crates/domain/src/ports.rs

// A driven (outbound) port: somewhere to store links. The core does not know
// if this is Postgres, Redis, or a HashMap.
#[async_trait]
pub trait LinkRepository: Send + Sync {
    async fn save(&amp;self, link: &amp;ShortLink) -&gt; Result&lt;(), DomainError&gt;;
    async fn find(&amp;self, code: &amp;ShortCode) -&gt; Result&lt;Option&lt;ShortLink&gt;, DomainError&gt;;
}

// A driving (inbound) port: the use cases the application offers to the world.
#[async_trait]
pub trait Shortener: Send + Sync {
    async fn shorten(&amp;self, target: &amp;str, code: Option&lt;&amp;str&gt;) -&gt; Result&lt;ShortLink, DomainError&gt;;
    async fn resolve(&amp;self, code: &amp;str) -&gt; Result&lt;ShortLink, DomainError&gt;;
}
</code></pre>
<p>The application layer implements the driving port by orchestrating the driven one. This file is the behavior of the system with none of the plumbing:</p>
<pre><code class="language-rust">// crates/application/src/lib.rs
pub struct ShortenerService {
    repo: Arc&lt;dyn LinkRepository&gt;, // depends on the port, not a concrete DB
}

#[async_trait]
impl Shortener for ShortenerService {
    async fn shorten(&amp;self, target: &amp;str, code: Option&lt;&amp;str&gt;) -&gt; Result&lt;ShortLink, DomainError&gt; {
        Self::validate_target(target)?;
        let code = match code {
            Some(raw) =&gt; ShortCode::parse(raw)?,
            None =&gt; ShortCode::parse(&amp;generate_code())?,
        };
        if self.repo.find(&amp;code).await?.is_some() {
            return Err(DomainError::CodeTaken);
        }
        let link = ShortLink { code, target: target.to_string() };
        self.repo.save(&amp;link).await?;
        Ok(link)
    }
    // ...
}
</code></pre>
<p>The HTTP adapter depends on the <code>Shortener</code> port, not the concrete service. Its only job is to translate HTTP into a call on the port and translate the result back. There is no business logic here, which is exactly why you can throw it away and replace it with a CLI without touching the core:</p>
<pre><code class="language-rust">// crates/adapters/src/http.rs
async fn create_link(
    State(state): State&lt;AppState&gt;,
    Json(req): Json&lt;CreateLinkRequest&gt;,
) -&gt; Result&lt;(StatusCode, Json&lt;LinkResponse&gt;), ApiError&gt; {
    let link = state.shortener.shorten(&amp;req.target, req.code.as_deref()).await?;
    Ok((StatusCode::CREATED, Json(link.into())))
}
</code></pre>
<p>And the composition root, the single file that knows every concrete type:</p>
<pre><code class="language-rust">// crates/server/src/main.rs
let repo: Arc&lt;dyn LinkRepository&gt; = Arc::new(InMemoryLinkRepository::default());
let shortener: Arc&lt;dyn Shortener&gt; = Arc::new(ShortenerService::new(repo));
let app = http::router(shortener);
axum::serve(listener, app).await.unwrap();
</code></pre>
<p>Want Postgres instead of memory? Write a <code>PostgresLinkRepository</code> that implements <code>LinkRepository</code>, and change line one. Want a CLI instead of HTTP? Change line three. Nothing upstream notices, because nothing upstream ever knew.</p>
<h2>Why agents thrive on this</h2>
<p>Now the payoff, and the reason this old pattern is worth revisiting in 2026. Every property that makes hexagonal architecture pleasant for a human maps directly onto something an agent needs.</p>
<p><strong>Tasks map to a single file with a precise contract.</strong> "Add a Postgres adapter" is not a vague request that could touch anything. It is: implement this one trait in this one new file. The agent has the exact signature to satisfy and physically cannot reach into business logic while doing it, because the adapter crate does not depend on private application internals. The task is scoped by the architecture, not by your prompt.</p>
<p><strong>The ports are the spec.</strong> An agent can write a correct use-case test from the trait definition alone, before a single adapter exists. In the template, the use-case tests run against a five-line fake repository:</p>
<pre><code class="language-rust">#[tokio::test]
async fn shorten_then_resolve_roundtrips() {
    let service = ShortenerService::new(Arc::new(FakeRepo::default()));
    let created = service.shorten("https://example.com", Some("my-code")).await.unwrap();
    let resolved = service.resolve("my-code").await.unwrap();
    assert_eq!(created, resolved);
}
</code></pre>
<p>No database, no HTTP server, no fixtures. An agent asked to raise test coverage on the core has everything it needs from the port and nothing it needs to mock away.</p>
<p><strong>Blast radius is bounded by construction.</strong> This is the big one. When an agent makes a wrong turn in a well-layered codebase, the mistake stays inside the layer it was working in. A bad decision in the HTTP adapter cannot corrupt the domain, because the compiler will not let the domain see the adapter. You get to be wrong locally, which is the only kind of wrong that is cheap to fix. In a codebase where a database call can appear inside a request handler, there is no such floor. A confused edit propagates.</p>
<p><strong>The structure survives context limits.</strong> An agent does not need to load the whole repository to work on one adapter. The relevant surface is the port it implements plus the crate it lives in. The architecture pre-chunks the codebase into units small enough to reason about in isolation, which is the same reason it was pleasant for humans in the first place. Good structure is good structure; agents just raise the stakes on having it.</p>
<h2>Where to start</h2>
<p>Do not start by rewriting everything into four crates. Start with the one rule and one seam. Pick the dependency that hurts most (usually the database) and define it as a trait in your core. Move the concrete client behind an adapter that implements the trait. Wire it up in one place. You will feel the core get quieter almost immediately, because it stops importing things that have nothing to do with your business.</p>
<p>Then, if it earns its keep, split the rings into crates so the compiler enforces the rule you have been maintaining by hand. Rust makes that enforcement free, and free enforcement is the difference between an architecture you have and an architecture you had six months ago.</p>
<p>The full template compiles, tests, and runs: <a href="https://github.com/llmgraph-ai/hexagonal-rust-template">github.com/llmgraph-ai/hexagonal-rust-template</a>. Clone it, run <code>cargo test</code>, delete the URL shortener, and keep the shape.</p>
<hr />
<p>This is the kind of structured backend we care about at <a href="https://llmgraph.ai">LLMGraph</a>, where we build LLM and AI-agent workflows that people deploy as APIs. Legible structure is what lets both our team and our tools move fast without breaking the parts that matter.</p>
<p><em>Originally published on <a href="https://dev.to/llmgraph/hexagonal-architecture-in-rust-and-why-it-makes-your-codebase-legible-to-coding-agents-544a">DEV</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[What it actually costs to run a RAG chatbot]]></title><description><![CDATA["How much will this cost to run?" is the question that kills more AI side projects than any technical problem. People build a working RAG chatbot in a…]]></description><link>https://llmgraph.hashnode.dev/what-it-actually-costs-to-run-a-rag-chatbot</link><guid isPermaLink="true">https://llmgraph.hashnode.dev/what-it-actually-costs-to-run-a-rag-chatbot</guid><category><![CDATA[RAG ]]></category><category><![CDATA[Artificial Intelligence]]></category><category><![CDATA[Machine Learning]]></category><dc:creator><![CDATA[Abaho Katabarwa]]></dc:creator><pubDate>Tue, 28 Jul 2026 12:09:11 GMT</pubDate><content:encoded><![CDATA[<p>"How much will this cost to run?" is the question that kills more AI side projects than any technical problem. People build a working RAG chatbot in a weekend, then freeze because they have no idea whether serving it costs 5 dollars a month or 5,000. This post breaks the bill into its real parts so you can estimate before you commit.</p>
<p>I'll use round numbers and current-ish prices. Yours will differ. The point is the shape of the bill, not the exact digits.</p>
<h2>The bill has four line items</h2>
<p>A retrieval-augmented chatbot spends money in four places:</p>
<ol>
<li>Embedding your documents (one time, plus updates)</li>
<li>Storing the vectors</li>
<li>Embedding each incoming question</li>
<li>Generating each answer</li>
</ol>
<p>Three of those four are close to free. One of them is the whole bill. Guess which before you read on.</p>
<h2>1. Embedding your documents</h2>
<p>You embed each chunk of your documents once. Embedding is cheap. Most providers charge a few cents per million tokens.</p>
<p>Say you have 200 documents averaging 3,000 words each. That's roughly 800,000 tokens. At current embedding prices that is a fraction of a cent to a few cents, total, one time. Even 10,000 documents lands in the low single-dollar range.</p>
<p>If your documents never change, you pay this once and forget it. If they change daily, you re-embed the changed ones, which is still pennies.</p>
<p>Line item: negligible.</p>
<h2>2. Storing the vectors</h2>
<p>Each chunk becomes a vector, usually 768 or 1536 numbers. A few thousand chunks is a few megabytes. You can hold it in memory, in a local file, or in a hosted vector database.</p>
<p>For a small project, a local store (Chroma, SQLite with a vector extension, FAISS on disk) costs nothing beyond the machine you already run. Hosted vector databases start free and stay cheap until you're storing millions of vectors.</p>
<p>Line item: zero to a few dollars a month until you're at real scale.</p>
<h2>3. Embedding each question</h2>
<p>Every question a user asks gets embedded so you can search. A question is short, maybe 20 tokens. At embedding prices that is far less than a hundredth of a cent per question.</p>
<p>A thousand questions a day is still cents per month.</p>
<p>Line item: negligible.</p>
<h2>4. Generating the answer</h2>
<p>Here's the bill.</p>
<p>Every answer is a call to a language model, and the input to that call is not just the question. It's the question plus the chunks you retrieved plus your instructions. That's the expensive part, because generation models charge per token and you're feeding them a few thousand tokens of context every single time.</p>
<p>Rough math. Say each answer sends 2,000 tokens of context and gets back 400 tokens. On a mid-tier model at a few dollars per million input tokens and a bit more for output, one answer costs somewhere around half a cent to two cents depending on the model.</p>
<p>Now multiply by traffic:</p>
<ul>
<li>100 answers a day: roughly 1 to 6 dollars a month</li>
<li>1,000 answers a day: roughly 15 to 60 dollars a month</li>
<li>10,000 answers a day: roughly 150 to 600 dollars a month</li>
</ul>
<p>That range is wide because model choice swings it more than anything else. A frontier model can cost 10 to 20 times what a smaller model costs for the same job. For a lot of support and lookup use cases, a smaller model with good retrieval beats a frontier model with sloppy retrieval, and costs a tenth as much.</p>
<h2>What actually moves the number</h2>
<p>Once you see that generation is the bill, the levers are obvious:</p>
<ul>
<li><strong>Model choice.</strong> The single biggest lever. Try the cheapest model that passes your quality bar before reaching for the expensive one.</li>
<li><strong>Context size.</strong> Retrieving 15 chunks instead of 5 triples your input tokens for often worse answers. Fewer, better chunks cost less and read better.</li>
<li><strong>Caching.</strong> If the same questions come up again and again, and in support they do, cache the answers. A cache hit costs nothing.</li>
<li><strong>Trimming instructions.</strong> A 600-token system prompt sent on every call adds up. Tighten it.</li>
</ul>
<h2>A quick estimate you can do right now</h2>
<p>Take your expected answers per day, multiply by 30 for the month, multiply by roughly 1 cent. That's your rough monthly generation bill on a mid-tier model. Everything else on the bill is rounding error at small scale.</p>
<p>So a chatbot answering 300 questions a day costs on the order of 10 dollars a month to run. That's the number that unfreezes the project.</p>
<h2>Where this fits</h2>
<p>If you'd rather not stand up the embedding, storage, and generation plumbing yourself and meter each piece, that's the kind of thing <a href="https://llmgraph.ai">LLMGraph</a> handles. You build the workflow, it runs the retrieval and the model calls, and you get a deployed API and chat widget. The cost shape above is still what's happening underneath, so it's worth understanding either way.</p>
<p>The takeaway: a RAG chatbot is cheap to run at small scale, generation is the only line item that matters, and model choice is the dial that turns a 10-dollar bill into a 200-dollar one. Estimate before you build, and the "what will this cost" freeze goes away.</p>
<p><em>Originally published on <a href="https://dev.to/llmgraph/what-it-actually-costs-to-run-a-rag-chatbot-2hoe">DEV</a>.</em></p>
]]></content:encoded></item><item><title><![CDATA[Turn a folder of PDFs into a question-answering API in an afternoon]]></title><description><![CDATA[Someone hands you 200 PDFs and asks for a "chatbot that knows this stuff." You could read all 200. Or you could build a small API that reads them for you…]]></description><link>https://llmgraph.hashnode.dev/turn-a-folder-of-pdfs-into-a-question-answering-api-in-an-afternoon</link><guid isPermaLink="true">https://llmgraph.hashnode.dev/turn-a-folder-of-pdfs-into-a-question-answering-api-in-an-afternoon</guid><category><![CDATA[RAG ]]></category><category><![CDATA[Artificial Intelligence]]></category><category><![CDATA[Python]]></category><dc:creator><![CDATA[Abaho Katabarwa]]></dc:creator><pubDate>Tue, 28 Jul 2026 04:01:35 GMT</pubDate><content:encoded><![CDATA[<p>Someone hands you 200 PDFs and asks for a "chatbot that knows this stuff." You could read all 200. Or you could build a small API that reads them for you and answers questions with citations. This post is the second option, start to finish, with the parts that usually break called out.</p>
<p>I'll show the mechanics first so you understand what's happening, then a shorter path if you don't want to run the plumbing yourself.</p>
<h2>What you're actually building</h2>
<p>A question-answering API over your own documents is four moving parts:</p>
<ol>
<li><strong>Ingest.</strong> Read each PDF, pull out the text.</li>
<li><strong>Chunk.</strong> Split that text into pieces small enough to search.</li>
<li><strong>Embed and store.</strong> Turn each chunk into a vector, keep it in a store you can query.</li>
<li><strong>Answer.</strong> At question time, find the closest chunks and hand them to a model with the question.</li>
</ol>
<p>People call the whole thing RAG (retrieval-augmented generation). The name makes it sound heavier than it is. It's search, then a prompt.</p>
<h2>Step 1: get the text out</h2>
<p>PDFs are a mess. Some are clean text, some are scanned images, some have two columns that extract in the wrong order. Start simple with <code>pypdf</code>:</p>
<pre><code class="language-python">from pypdf import PdfReader

def read_pdf(path: str) -&gt; str:
    reader = PdfReader(path)
    return "\n".join(page.extract_text() or "" for page in reader.pages)
</code></pre>
<p>Where it breaks: scanned PDFs return empty strings because there's no text layer, only pixels. If <code>extract_text()</code> gives you nothing, those files need OCR (Tesseract, or a hosted vision model). Check for empty output early so you don't silently index blank pages.</p>
<h2>Step 2: chunk the text</h2>
<p>You can't embed a whole document as one vector and expect good answers. Retrieval works on pieces. The size of those pieces matters more than most tutorials admit.</p>
<p>Too small (a sentence) and each chunk loses its context. Too big (a whole page) and the model gets a wall of mostly-irrelevant text and the answer drifts. A reasonable starting point is 500 to 800 tokens per chunk with a small overlap so a sentence split across a boundary isn't lost.</p>
<pre><code class="language-python">def chunk(text: str, size: int = 600, overlap: int = 80) -&gt; list[str]:
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)]
</code></pre>
<p>This splits on whitespace, which is crude but fine to start. If your documents have clear structure (headings, sections), splitting on those boundaries beats a fixed window because a chunk that matches a section is a chunk that makes sense on its own.</p>
<h2>Step 3: embed and store</h2>
<p>An embedding turns a chunk into a list of numbers that captures its meaning. Similar text lands near similar text. You query by embedding the question the same way and finding the nearest chunks.</p>
<p>For a folder of 200 PDFs you do not need a managed vector database. A local store is plenty:</p>
<pre><code class="language-python">import chromadb
from chromadb.utils import embedding_functions

client = chromadb.PersistentClient(path="./store")
ef = embedding_functions.DefaultEmbeddingFunction()
col = client.get_or_create_collection("docs", embedding_function=ef)

for doc_id, text in enumerate(all_texts):
    chunks = chunk(text)
    col.add(
        ids=[f"{doc_id}-{i}" for i in range(len(chunks))],
        documents=chunks,
        metadatas=[{"source": sources[doc_id]} for _ in chunks],
    )
</code></pre>
<p>Keep the source filename in the metadata. That's what lets you cite where an answer came from, which is the difference between a demo and something people trust.</p>
<h2>Step 4: answer with citations</h2>
<p>At question time: embed the question, pull the top handful of chunks, put them in the prompt, ask the model to answer only from what it was given.</p>
<pre><code class="language-python">def answer(question: str) -&gt; str:
    hits = col.query(query_texts=[question], n_results=5)
    context = "\n\n".join(hits["documents"][0])
    sources = {m["source"] for m in hits["metadatas"][0]}

    prompt = (
        "Answer the question using only the context below. "
        "If the context does not contain the answer, say so.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    reply = call_your_model(prompt)  # Claude, GPT, a local model, your choice
    return f"{reply}\n\nSources: {', '.join(sources)}"
</code></pre>
<p>Two things that matter here:</p>
<ul>
<li><strong><code>n_results</code>.</strong> Five is a good default. More context is not better. Past a point you're paying for tokens and diluting the signal.</li>
<li><strong>The "say so" instruction.</strong> Without it the model fills gaps with plausible fiction. With it you get "the documents don't cover this," which is the honest answer and the one that keeps people trusting the system.</li>
</ul>
<p>Wrap <code>answer()</code> in any web framework and you have an API:</p>
<pre><code class="language-python">from fastapi import FastAPI

app = FastAPI()

@app.post("/ask")
def ask(question: str):
    return {"answer": answer(question)}
</code></pre>
<h2>Where this gets hard in production</h2>
<p>The afternoon version above works. The things that turn it into a real project:</p>
<ul>
<li><strong>Re-indexing.</strong> Documents change. You need a way to update or drop chunks without rebuilding everything.</li>
<li><strong>Bad retrieval.</strong> When answers are wrong it's usually retrieval, not the model. The right chunk never made it into the context. Logging what got retrieved for each question is the first debugging tool you'll want.</li>
<li><strong>Access control.</strong> If different people should see different documents, retrieval has to filter by permission before it ranks. Bolting this on later is painful.</li>
<li><strong>Cost.</strong> Embeddings are cheap. The generation call is not, and it scales with traffic. Caching common questions helps.</li>
</ul>
<h2>The shorter path</h2>
<p>If you'd rather not run the ingest, chunking, vector store, and prompt loop yourself, this is the exact shape of thing <a href="https://llmgraph.ai">LLMGraph</a> builds. You point it at your documents, it handles the chunking and retrieval, and you get a REST API and an embeddable chat widget out the other side. The four steps above still happen. You just don't maintain them.</p>
<p>Either way, the mental model is the same: read, chunk, embed, retrieve, answer. Once that clicks, "chatbot that knows our stuff" stops sounding like a research project and starts sounding like an afternoon.</p>
<p><em>Originally published on <a href="https://dev.to/llmgraph/turn-a-folder-of-pdfs-into-a-question-answering-api-in-an-afternoon-14kg">DEV</a>.</em></p>
]]></content:encoded></item></channel></rss>