The architecture of an open-source slop detector

I want Slopcheck to be useful without asking every reader to open a developer account and supply an API key. The hosted tier makes that possible, but each provider request costs me money. The bring-your-own-key option, usually shortened to BYOK, lets someone use their own provider account instead.

An automated script can buy more work from the provider than I want to fund. A submitted document can also contain instructions that try to redirect the analysis. Controlling volume, limiting the work purchased from a provider, and keeping the analysis on task require different checks.

The goal is to keep the tool free, private, open and sustainable. Its analysis planner, hosted controls and shared router have to serve that goal together:

Browser: assemble the analysis prompts and choose a pass strategy
    → tools router: forward the /slopcheck/ path
    → hosted Pages Function: check the request and counters
    → provider adapter: send prompts using the hosted provider key
    ← Markdown analysis, returned to the browser for display

The launch article explains the editorial purpose. The methodology article explains the growing pattern catalog.

What an analysis can tell an editor

Stylistic observations and substance answer different editorial questions. A paragraph can repeat itself whether a human wrote it or an AI helped produce it. An unfamiliar expression can deserve examination without establishing who wrote the sentence.

The substance checks ask questions an editor can act on. If I remove this paragraph, does the reader lose a claim, evidence or a necessary connection? Could the same paragraph describe almost any company after changing the name? Which claims would need evidence before publication?

The source prompts also ask for model-family attribution and confidence ratings. Those are requested model judgments. The orchestration code does not validate their accuracy, and separating them under two headings does not establish authorship.

Liang and colleagues' 2023 study examined seven detectors and found substantial misclassification of non-native English writing. It supports caution about those tools and populations. It does not validate Slopcheck's proposed exception for particular combinations of writing patterns. That exception would need its own evaluation against relevant human writing. I would use the tool to find passages worth revising, then inspect the passage and its sources.

How the planner chooses its calls

The methodology ran to roughly forty pages of patterns and example output. Its size creates a practical constraint: the catalog, submitted text, prompt instructions and response all compete for the model's context window.

The orchestrator estimates tokens by dividing character counts by four and rounding up. It adds an allowance of 5,000 tokens for prompt overhead, reserves 8,000 for the response, and adds a safety margin equal to the larger of 10 percent of the configured context window or 8,000 tokens. This is a planning heuristic, not a measurement from the selected model's tokenizer.

If that estimate fits, one provider call receives the assembled catalog and user text. Otherwise, the orchestrator runs focused analysis passes: model-family patterns, substance and depth, and the broader pattern catalog with calibration. A conditional citation-checking pass may follow. A final synthesis call combines the earlier findings. Using a single call when it fits avoids the cost of splitting every submission across several calls.

Exercise prepared September 27, 2026: the call counts and boundary tests below use fictional inputs and intercepted provider requests.

That produces one, four or five provider calls in the offline cases accompanying this article:

Fictional input configurationProvider calls
Catalog and text fit the configured window1
Oversized catalog, without a citation trigger4
Oversized catalog, with a citation trigger5

Splitting the catalog reduces how much methodology each analysis call receives. The inspected implementation still includes the complete submitted text in each call; it does not implement the document chunking mentioned in its introductory comment. A very long document therefore needs another design decision, such as explicit chunking with a tested synthesis method, or a clear refusal when the document cannot fit.

Both strategies return a string containing Markdown. The hosted endpoint wraps provider text in a JSON object with a text property, and the browser renders Markdown. Headings requested in a prompt are not schema-validated response fields. Progress callbacks expose the planner's selected strategy locally; they are not response metadata proving a structured output contract.

What the hosted endpoint checks

I chose a Cloudflare Pages Function alongside the static frontend instead of the standalone Worker in the original specification. That kept the frontend and its hosted endpoint in one deployment. The Function reads hosted provider keys from its environment; the browser's hosted request does not need those keys. BYOK follows a different path using the reader's key.

The hosted handler has a specific order. It parses required fields, performs the configured bot check, checks the model name and content length, reads the global kill flag, reads the IP counter, calls the provider, then updates the IP counter and estimated budget.

ControlWhat the inspected source or documentation establishesWhat the maintainer still has to verify
Provider spending controlsProviders document their own account and project limits.The actual account settings, billing period and enforcement delay.
Server-side proxyThe Function chooses the hosted key from its environment and calls the provider.Deployed secrets, access policy and the complete set of accepted request fields.
Per-IP allowanceA keyed IP identifier and UTC date select a count, read before the call and incremented afterward.Concurrent requests, shared addresses and distributed callers.
Community budget and kill flagAn already-set flag rejects a request; a post-call estimate can set it for later requests.Atomic accounting and the relationship between the estimate and actual charges.
Content capThe default checks 200,000 characters in the supplied userContent string, with a fallback to userPrompt.Whether that field bounds the separate prompts and output limit sent to the provider.
Model allowlistA model name absent from the configured list is rejected.Provider/model compatibility and deployed overrides.
Turnstile and internal keyA configured Turnstile secret triggers verification unless the internal key matches.Configuration, caller identity, integration behavior and key rotation.

Later addition, prepared September 28, 2026: current provider spending controls.

OpenAI added hard monthly organization and project spending limits on July 22, 2026, after this article's May publication date. Its current guide describes optional enforced limits alongside non-enforcing alerts, with some extra usage possible while limit state propagates. Anthropic documents monthly spending limits separately from request and token rate limits. Google documents monthly Gemini API controls, including experimental project caps with accounting delays that can permit overages. The daily service budget still needs its own enforcement.

The local budget is also an estimate. The provider adapter assigns a flat number of cents to each allowed model. After a successful call, the handler adds that estimate and sets the kill flag when the recorded total reaches 95 percent of the configured ceiling. It does not reserve the likely cost before the call or calculate the charge from returned token usage.

Day-based keys make new UTC days use new counters. The stored entries have a 90,000-second expiration, about 25 hours from a write. That is different from deleting every entry at midnight.

Two useful boundary tests

Consider a content cap configured to ten characters. A request supplies these fictional fields:

{
  "userContent": "x",
  "userPrompt": "xxxxxxxxxxx",
  "maxTokens": 999999
}

The handler accepts the one-character userContent value. Its provider adapter forwards the separate eleven-character userPrompt and the requested output limit. The offline provider receives those exact values. This is a data-path problem: a limit on one client-supplied field does not automatically limit another.

For an application that funds requests, I would put the trusted analysis instructions on the server, validate the actual provider-bound payload, and enforce the allowed output limit there. A model-name list cannot supply that contract. Nor does a content-length check demonstrate resistance to instructions embedded in the submitted document.

Concurrency exposes another boundary. In the exercise, two requests share one fictional IP and a one-request daily allowance. Both read the unused allowance before either provider response returns. Both calls proceed. The subsequent writes leave a count of one and one eight-cent budget increment, despite two successful provider calls.

That schedule is deliberately constructed with an in-memory store. It demonstrates a possible interleaving in the read-then-write code; it does not measure Cloudflare traffic or bills. Cloudflare's KV documentation describes eventual consistency and advises against using KV where a read and write must form one atomic transaction. A spending reservation needs an enforcement mechanism that makes the admission decision and reservation indivisible.

The existing kill flag does have a useful, narrower property in the example: an internal integration's key does not bypass it. The key skips the configured Turnstile check. A request bearing that key still stops at an already-set global flag before reaching the provider.

The email source sends the internal header, and the Slack and Discord paths add it when their internal key is configured; the inspected Google Docs path does not. A missing Turnstile secret causes the handler to skip that check. I rotate the internal key and keep it in server-side environment configuration for the integrations that use it.

Follow the data as well as the counters

The rate-limit helper computes an HMAC-SHA256 identifier from the IP address and a secret salt. Its counter key combines that identifier with a UTC date. Keeping the salt unchanged keeps the IP-derived identifier stable; changing the date suffix does not itself rotate the salt.

The handler processes the incoming IP. When Turnstile verification runs, its helper can send the raw IP in the verification request. The submitted prompts also travel through the hosting platform to the selected model provider. The shown KV writes contain counters and flags rather than the submitted document, but that is not evidence about every platform log or retention policy.

A privacy account needs to name the data, destination, retention and operator settings at each step. Hashing one field answers only part of that account.

The controls address different failures: a spending limit does not prevent prompt misuse; a document-size limit does not prevent a thousand short requests; an IP allowance does not stop callers spread across many addresses. Defense in depth is the only stance that survives contact with a production tool that is free, public, and worth abusing.

One subdomain, multiple tools

The public tool address separates a host, tools.synthesiswriting.org, from a path, /slopcheck/. I wanted additional tools to fit under that same host.

The tools router is a Cloudflare Worker that maps the first path segment to a backing service. For Slopcheck it strips /slopcheck, so the hosted request reaches the Pages project as /api/hosted/analyze. It forwards the method, query string, headers and body. A request to bare /slopcheck receives a redirect to /slopcheck/, which lets the frontend's relative URLs resolve under the tool's directory.

Adding another tool means adding a route and providing its backing service. The design reuses the subdomain's existing DNS setup. The backing service can change independently of the public path, provided it meets the routing contract. That reuse has a compounding payoff: I can keep adding tools without setting up a new public host for each one.

Run the request-path exercise

The downloadable exercise contains frozen source files, fictional inputs, literal expected results and a harness that intercepts every fetch used in these cases. It needs Node.js 22 or newer and no API key.

After extracting the archive, run:

node run.mjs
node controls.mjs ./control-results

The first command checks thirteen cases covering the hosted handler, planner and router. “Pass” means their outputs match the stated source-observation contract, including the weak cases described above. The second creates four isolated source mutations and confirms that unchanged expectations reject each one. It preserves the altered files and failure logs so you can inspect what caused each rejection.

No model judges any writing in this exercise. It checks request fields, call counts, forwarding and counter behavior. You can adapt the approach by starting with the contract you need, locating the actual consumer of each field, and writing an example that would fail if the consumer used the wrong value.

What contributors can inspect

Before making Slopcheck, tools-router and homebrew-tap public, I had their pre-public histories rewritten into a single initial commit while retaining the final files. The contributor cost is real: readers cannot reconstruct the earlier design iterations from that public history. I regard that as a worthwhile tradeoff for this launch. I do not consider the iteration history the teaching artifact, and early editorial decisions would have appeared without their context.

My policy after a repository becomes public is to fix mistakes with subsequent commits, without force-pushing away the published history. I also chose a consistent sponsorship destination through GitHub's FUNDING.yml convention. Voluntary funding lets readers support the work without making payment a condition of using the tool.

The source map is under synthesisengineering on GitHub: synthesis-slopcheck contains the app, hosted Function, integrations, CLI and browser extension; synthesis-tools-router contains the routing Worker; synthesis-skills contains the methodology; and homebrew-tap provides the macOS installer.

The contribution is the integration: analysis planning directs what work to buy, hosted controls govern access to that work, shared routing makes room for more tools, and public repositories make the design available for inspection and reuse. I want that combination to support a free, private and sustainable service.

When I inspect a service like this, I want to follow one request far enough to answer four questions: who supplied this field, which component checked it, what work could already have happened, and what would a competing request change?