DEV Community: anh

Anatomy of an OpenAI-compatible provider in Go

anh — Sun, 19 Apr 2026 03:00:43 +0000

GoAI shipped Cloudflare Workers AI and FPT Smart Cloud providers in v0.7.0, then refactored the shared plumbing in v0.7.1. Chat-only providers come in at ~84 lines. The two new ones, with embeddings and unique routing, land at 126 and 132. This post walks through the anatomy and which Go features made it small.

Starting point

OpenAI's Chat Completions and Embeddings shape is a de facto standard. Most inference vendors expose it. In GoAI, 18 of 24 providers speak this wire format. They differ only in URL, auth, and occasional routing. 14 of those share a single factory in internal/openaicompat. The other 4 are openai and vertex with custom routing, plus ollama and vllm which wrap the generic compat provider.

"How much code for a new one?" About 84 lines for a chat-only provider. Most of that is options boilerplate users see in their IDE. Providers with embeddings or custom routing land in the 120s.

The interface

A provider implements two interfaces from provider/:

type LanguageModel interface {
    ModelID() string
    DoGenerate(ctx context.Context, params GenerateParams) (*GenerateResult, error)
    DoStream(ctx context.Context, params GenerateParams) (*StreamResult, error)
}

type EmbeddingModel interface {
    ModelID() string
    DoEmbed(ctx context.Context, values []string, params EmbedParams) (*EmbedResult, error)
    MaxValuesPerCall() int
}

No base class, no registry, no lifecycle. Go's interfaces are satisfied implicitly, so adding a provider doesn't touch any other file.

What's shared

internal/openaicompat owns the wire format and the HTTP plumbing. Two factories do most of the work:

func NewChatModel(cfg ChatModelConfig) provider.LanguageModel
func NewEmbeddingModel(cfg EmbeddingModelConfig) provider.EmbeddingModel

The factory handles request building, streaming, response parsing, token resolution, error dispatch, and the embedding round-trip. Provider packages fill in a config struct and pass it.

internal/ is a Go convention: packages under it are importable only within the same module tree, not by external consumers. That lets 14 providers (plus Ollama and vLLM via the compat wrapper) share the factory without exposing a new public API surface.

Provider packages stay thin and user-facing. The factory owns the plumbing. Two concrete providers show how this works.

Cloudflare

Cloudflare Workers AI is OpenAI-compatible, with one quirk: the URL embeds the account ID.

https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1/chat/completions

The provider-specific work is URL construction. Everything else comes from the shared factory.

const defaultAPIBase = "https://api.cloudflare.com/client/v4"

func WithAccountID(id string) Option {
    return func(o *options) { o.accountID = id }
}

// In resolveOptions, after reading env vars CLOUDFLARE_API_TOKEN / CLOUDFLARE_ACCOUNT_ID:
if o.baseURL == "" && o.accountID != "" {
    o.baseURL = fmt.Sprintf("%s/accounts/%s/ai/v1", defaultAPIBase, o.accountID)
}

Usage:

model := cloudflare.Chat("@cf/meta/llama-3.1-8b-instruct",
    cloudflare.WithAccountID("your-account-id"))

Total file: 126 lines including chat, embeddings, and 6 With* options. Cloudflare provider docs.

FPT Smart Cloud

FPT Smart Cloud's AI marketplace has a different quirk: two regions, Global and Japan, each with its own model catalog.

const (
    baseURLGlobal = "https://mkp-api.fptcloud.com/v1"
    baseURLJP     = "https://mkp-api.fptcloud.jp/v1"
)

func WithRegion(region string) Option {
    return func(o *options) { o.region = region }
}

func regionBaseURL(region string) string {
    switch region {
    case "jp":
        return baseURLJP
    default:
        return baseURLGlobal
    }
}

Usage:

model := fptcloud.Chat("Qwen3-32B", fptcloud.WithRegion("jp"))

The JP region hosts Qwen3-32B, Llama-3.3-70B-Instruct, gpt-oss-120b, GLM-4.7, among others. I verified generate and stream against Qwen3-32B. Total file: 132 lines including chat, embeddings, and region routing. FPT Smart Cloud provider docs.

Both providers follow the same shape: resolveOptions reads env vars (CLOUDFLARE_API_TOKEN, FPT_API_KEY, etc.) as fallback, computes the base URL, then Chat() passes a ChatModelConfig to openaicompat.NewChatModel. Only the URL-derivation bit above is unique.

Compile-time interface checks

The factory has this block near the top of internal/openaicompat/factory.go:

var (
    _ provider.LanguageModel  = (*chatModel)(nil)
    _ provider.CapableModel   = (*chatModel)(nil)
    _ provider.EmbeddingModel = (*embeddingModel)(nil)
)

It assigns a nil pointer of each concrete type into the interface variable. Renaming an interface method breaks the build immediately, not silently at runtime.

Idiomatic Go, not a GoAI invention. One check covers all 14 providers that route through the factory.

Testing

Every provider ships a _test.go using net/http/httptest.NewServer or a custom http.RoundTripper to capture outgoing requests:

// Sketch; roundTripperFunc and okResponse are local helpers in the test file.
var gotAuth, gotURL string
tr := roundTripperFunc(func(req *http.Request) (*http.Response, error) {
    gotAuth = req.Header.Get("Authorization")
    gotURL = req.URL.String()
    return okResponse(), nil
})
t.Setenv("CLOUDFLARE_API_TOKEN", "env-tok")
t.Setenv("CLOUDFLARE_ACCOUNT_ID", "env-acc")
m := Chat("m", WithHTTPClient(&http.Client{Transport: tr}))
_, err := m.DoGenerate(t.Context(), params)
// assert gotAuth == "Bearer env-tok", gotURL contains "env-acc"

No mocking library. The test server (or round-tripper) runs the same code path as production. Streaming tests work the same way, just with Server-Sent Events chunks instead of a JSON body.

All 14 OpenAI-compatible providers reach 100% statement coverage. Factory at 99.8%.

Functional options

Every provider exposes the same small set:

WithAPIKey(key string)
WithTokenSource(ts provider.TokenSource)
WithBaseURL(url string)
WithHeaders(h map[string]string)
WithHTTPClient(c *http.Client)

Plus one or two provider-specific ones (WithAccountID, WithRegion). The signature is always func(*options), so adding a knob doesn't change any constructor.

Not novel, Dave Cheney wrote about it in 2014. It's why the 14 providers feel consistent without sharing a base type.

What Go didn't give me

No default arguments. Every option is a separate With* function. The factory's config struct has 12 fields, most are optional. Zero-value defaults work but grow fragile at 20+ fields.
No decorator pattern. Telemetry and retry wrap explicitly via hooks, not annotations. Verbose but clear.
No pattern matching. Response parsing is if/switch on JSON shapes. Rust enums would be cleaner here.

By the numbers

	LOC
Simple provider (deepinfra, groq, mistral, ...)	84
Complex (cloudflare, fptcloud with embeddings)	126-132
14 OpenAI-compat providers, total	~1,324
Shared factory in `internal/openaicompat`	334
Coverage	100% providers, 99.8% factory

Takeaway

The pattern that scales to 14 providers without bloat:

Split the public surface from the plumbing. User-facing names (cloudflare.WithAccountID, env var conventions) live in the provider package. HTTP dispatch, token resolution, error parsing live in internal/openaicompat. Changes to the shared code ripple across 14 providers at once without breaking any public API.
Variations as config, not plugins. Extra body fields, fixed headers, optional auth, account-ID URL building, each is a field on ChatModelConfig or a few lines in resolveOptions. No sub-classing, no registry.
Compile-time checks over documentation. The var _ LanguageModel = (*chatModel)(nil) assertion at the top of the factory guarantees every provider still satisfies the interface. No runtime surprises.

The factory is 334 lines. Each provider is a few dozen lines of declarations on top.

v0.7.1 is live. If an inference provider speaks OpenAI-compatible and isn't in GoAI yet, the Cloudflare and FPT diffs are reasonable templates.

Why (and How) I Built a Go AI SDK

anh — Tue, 07 Apr 2026 13:02:49 +0000

GoAI, a Go (Golang) LLM library: 22+ providers, 2 dependencies, type-safe generics. v0.6.1, Go 1.25+. I built it to learn Go by adding AI to infrastructure that already runs on Go.

The Go AI SDK landscape

Python has LangChain, LlamaIndex, LiteLLM. TypeScript has the Vercel AI SDK. Go has options, but none covered all the bases.

What I found

Library	Stars	Created	Providers	What's missing
go-openai	~10.6k	Aug 2020	1	Single provider only
Eino (ByteDance)	~10.5k	Dec 2024	10	Graph framework, different scope
LangChainGo	~9k	Feb 2023	~14	170+ deps, no MCP, no generics schema
Google ADK Go	~7.5k	May 2025	Gemini-first	Agent framework, Gemini-optimized
Genkit Go (Google)	~5.7k*	2024	~6	Google Cloud-heavy, 129 deps
openai-go	~3.1k	Jul 2024	1+Azure	Single provider by design
anthropic-sdk-go	~960	Jul 2024	1+Bedrock	Single provider by design
gollm	~570	Jul 2024	6	Limited tool calling, limited streaming
Jetify AI	~230	May 2025	2	Early stage, 2 providers
instructor-go	~200	May 2024	4	Structured output only
GoAI	new	Mar 2026	22+	2 deps, this post

*Genkit stars shared across JS, Go, and Python.

The gaps I kept running into:

No SchemaFrom[T]: generate JSON Schema from Go structs using generics. Only Genkit Go has this, but pulls in 129 dependencies
No built-in MCP: Model Context Protocol connects to external tool servers (filesystem, GitHub, databases, Kubernetes APIs). For infra and edge use-cases, this is how agents interact with surrounding systems
No provider-defined tools: OpenAI has web search, code interpreter, file search. Anthropic has computer use, bash, text editor, web fetch, code execution. Google has Google Search, URL context, code execution. None of the Go LLM libraries expose these
No prompt caching: Anthropic and OpenAI support cache control to reduce cost and latency
No streaming structured output: StreamObject[T] that progressively populates a Go struct as JSON arrives
Dependency weight: a library that handles API keys to every major AI provider is a high-value target. Fewer dependencies, smaller attack surface

Why I built GoAI

1. To learn Go. Not a Go expert. Built this to learn by solving a real problem. PRs welcome.

2. To learn AI-assisted development. Designed by me, built with Claude Code. I'll write a separate post on the workflow, what worked, and what didn't.

3. To build a foundation for agent orchestration. Lightweight AI agents in CI/CD, Kubernetes, CLI, edge. GoAI is the foundation layer.

What I learned from the Vercel AI SDK

The Vercel AI SDK is the reference. GoAI's API surface is directly inspired by it:

goai.GenerateText(ctx, model, opts...)
goai.StreamText(ctx, model, opts...)
goai.GenerateObject[T](ctx, model, opts...)
goai.StreamObject[T](ctx, model, opts...)
goai.Embed(ctx, model, value, opts...)
goai.EmbedMany(ctx, model, values, opts...)
goai.GenerateImage(ctx, model, opts...)

What I took from Vercel:

Unified API surface across all providers
Minimal provider interface, 2-3 methods per provider, SDK handles the rest
Tool loop with MaxSteps, model calls tool, execute, feed back, repeat
Streaming-first
Retry with exponential backoff and Retry-After header awareness

Go features that solved real problems

Go generics for type-safe LLM output. GenerateObject[T]() returns ObjectResult[T] where .Object is T, not interface{}. SchemaFrom[T]() walks the struct via reflect to generate JSON Schema, with cycle detection, embedded struct flattening, and nullable pointer fields. No schema files, no codegen. Docs.

type Recipe struct {
    Name        string   `json:"name"`
    Ingredients []string `json:"ingredients"`
    Steps       []string `json:"steps"`
}

result, err := goai.GenerateObject[Recipe](ctx, model,
    goai.WithPrompt("A simple pasta recipe"),
)
// result.Object is Recipe, fully typed

Functional options for composable configuration. WithX() pattern gives composability and extensibility without breaking changes. Separate option types (Option vs ImageOption) so the compiler catches misuse. Options are composable via WithOptions(). Docs.

result, err := goai.GenerateText(ctx, model,
    goai.WithSystem("You are a helpful assistant"),
    goai.WithPrompt("Hello"),
    goai.WithMaxSteps(3),
    goai.WithTools(searchTool, calculatorTool),
    goai.WithMaxRetries(2),
    goai.WithOnResponse(func(info goai.ResponseInfo) {
        log.Printf("latency=%v tokens=%d", info.Latency, info.Usage.TotalTokens)
    }),
)

Interfaces for provider abstraction. Three interfaces (LanguageModel, EmbeddingModel, ImageModel), implicitly satisfied. 22+ providers conform independently. Adding a provider never touches core. Docs.

Goroutines and channels for streaming. Background goroutine consumes the provider stream (SSE for most providers, binary EventStream for Bedrock, NDJSON for Gemini). Callers read from <-chan string. All functions are context-aware, retries respect ctx.Done(). Tool loops execute in parallel with bounded concurrency via semaphores. Docs.

stream, _ := goai.StreamText(ctx, model, goai.WithPrompt("Tell me a story"))

for text := range stream.TextStream() {
    fmt.Print(text)
}

if err := stream.Err(); err != nil {
    log.Fatal(err)
}

sync.Map and sync.Once. Schema generation cached in sync.Map by type. Stream consumption uses sync.Once to start the internal goroutine exactly once.

errors.As for cross-provider errors. APIError and ContextOverflowError defined once, every provider wraps into these. errors.As(err, &apiErr) works through any wrapping depth. Docs.

internal/ for encapsulation. The openaicompat codec lives in internal/, shared across 13+ providers but invisible to users.

Fewer dependencies, smaller attack surface

GoAI's core module has 2 dependencies:

Direct: golang.org/x/oauth2
Indirect: cloud.google.com/go/compute/metadata

No HTTP frameworks, no JSON schema libraries, no third-party provider SDKs. Raw HTTP calls, parse responses directly.

This was a design choice from day one (mid-March 2026). For context on why this matters:

LiteLLM (Python, 95M monthly downloads): compromised, malicious versions harvested API keys and cloud credentials. 40,000+ downloads in 40 minutes.
Axios (npm, 100M+ weekly downloads): compromised by a North Korean state actor, cross-platform RAT via fake dependency.

A multi-provider AI SDK concentrates API keys for every provider. Every dependency is an attack vector.

GoAI core:        2 dependencies
Eino:            37 dependencies
instructor-go:   41 dependencies
Genkit Go:      129 dependencies
LangChainGo:    170+ dependencies

Both Langfuse and OpenTelemetry live in separate go.mod submodules. Go's go.sum checksum verification and GOPROXY transparency log help too.

Architecture

Three layers:

User-facing API (generate.go, object.go, embed.go, image.go): seven functions, options parsing, retry, caching, tool loops, hooks. Context-aware. Multimodal input (PartImage, PartFile). Token usage tracking. Docs.
Provider interface (provider/provider.go): three interfaces, minimal surface, easy to mock. Docs.
Provider implementations (provider/openai/, provider/anthropic/, etc.): 22+ providers, separate packages. Import only what you use.

type LanguageModel interface {
    ModelID() string
    DoGenerate(ctx context.Context, params GenerateParams) (*GenerateResult, error)
    DoStream(ctx context.Context, params GenerateParams) (*StreamResult, error)
}

type EmbeddingModel interface {
    ModelID() string
    DoEmbed(ctx context.Context, values []string, params EmbedParams) (*EmbedResult, error)
    MaxValuesPerCall() int
}

type ImageModel interface {
    ModelID() string
    DoGenerate(ctx context.Context, params ImageParams) (*ImageResult, error)
}

Providers implement DoGenerate and DoStream. GoAI handles retries, caching, tool execution, streaming, hooks.

The shared codec

13+ providers share a single codec in internal/openaicompat/ (BuildRequest, ParseStream, ParseResponse). Providers with unique wire formats (Anthropic Messages API, Google Gemini REST, AWS Bedrock Converse, Cohere Chat v2) have their own implementations. Azure, vLLM, and MiniMax delegate through existing providers. Docs.

Tool calling

Two kinds: user-defined (you write Execute with json.RawMessage input) and provider-defined (runs on provider infrastructure). User-defined tools execute in parallel between steps. Docs.

Provider-defined tools

Providers expose built-in tools (web search, code execution, computer use). Each returns a provider.ToolDefinition that you wrap into goai.Tool. Docs.

// Get the provider tool definition
def := anthropic.Tools.WebSearch(anthropic.WithMaxUses(5))

// Wrap into goai.Tool (provider-defined tools have no Execute func)
tools := []goai.Tool{{
    Name:                   def.Name,
    ProviderDefinedType:    def.ProviderDefinedType,
    ProviderDefinedOptions: def.ProviderDefinedOptions,
}}

result, _ := goai.GenerateText(ctx, model,
    goai.WithPrompt("Search for Go AI libraries"),
    goai.WithTools(tools...),
)

Available: Anthropic (10: Computer, Bash, TextEditor, WebSearch, WebFetch, CodeExecution + versioned variants), OpenAI (4: WebSearch, CodeInterpreter, FileSearch, ImageGeneration), Google (3: GoogleSearch, URLContext, CodeExecution), xAI (2), Groq (1).

Go MCP client

Built-in MCP client. Docs.

transport := mcp.NewStdioTransport("npx", []string{"@modelcontextprotocol/server-filesystem", "/tmp"})
client := mcp.NewClient("myapp", "1.0", mcp.WithTransport(transport))
client.Connect(ctx)
defer client.Close()

mcpTools, _ := client.ListTools(ctx, nil)
tools := mcp.ConvertTools(client, mcpTools.Tools)

result, _ := goai.GenerateText(ctx, model,
    goai.WithPrompt("List files in /tmp"),
    goai.WithTools(tools...),
    goai.WithMaxSteps(3),
)

Stdio, SSE, and HTTP transports.

Observability

Langfuse: trace-based observability, token counting, error tracking (separate go.mod, contributed by @oscarbc96 in #24)
OpenTelemetry: distributed tracing and metrics (separate go.mod)

Both hook into OnRequest, OnResponse, OnToolCall, OnToolCallStart, OnStepFinish.

Also supported:

Prompt caching via WithPromptCaching(), implements arxiv 2601.06007v2: cache system prompts only (41-80% cost, 13-31% latency savings for agentic workloads). Cache token tracking normalized across Anthropic, OpenAI, Google, Bedrock
Reasoning tokens, supported for models that expose them
Citations, result.Sources for providers that return source annotations
Auto-batched embeddings, EmbedMany with bounded parallelism
WithToolChoice, WithTimeout, WithProviderOptions
ToolCallIDFromContext for execution tracing
compat provider for any OpenAI-compatible endpoint
90%+ test coverage with mock HTTP servers

Full docs.

Getting started

go get github.com/zendev-sh/goai

model := openai.Chat("gpt-4o")

result, _ := goai.GenerateText(ctx, model,
    goai.WithPrompt("Explain Go interfaces in 3 sentences"),
)
fmt.Println(result.Text)

Switch providers, same code:

model := anthropic.Chat("claude-sonnet-4-20250514")
model := google.Chat("gemini-2.0-flash")
model := bedrock.Chat("anthropic.claude-sonnet-4-20250514-v1:0")
model := ollama.Chat("llama3")
model := compat.Chat("my-model", compat.WithBaseURL("https://my-api.com/v1"))

More: structured output, streaming, tool calling, MCP, embeddings, image generation.

Benchmarks

Apple M2, 3 runs, in-process mock servers, identical SSE fixtures (50KB payload). Source.

Metric	GoAI	Vercel AI SDK	Delta
Cold start	569us	13.89ms	24x
Time to first chunk	320us	412us	1.3x
Streaming throughput	1.46ms/op	1.62ms/op	1.1x
GenerateText	55.7us/op	79.0us/op	1.4x
Memory (1 stream)	220KB	676KB	3x less
Schema generation	3.6us/op	3.5us/op	~parity

What's next

First commit was March 18. 18 releases in 3 weeks, now at v0.6.1. The SDK covers the provider layer: unified API, streaming, tool calling, structured output, MCP, observability. Next up is an agent orchestration layer on top of GoAI for CI/CD, Kubernetes, and CLI workflows.

Originally published on https://blog.anh.sh/why-and-how-i-built-a-go-ai-sdk