Major Redesign Of A Multi-Vendor Llm Abstraction Layer

Issue 95 Edition 2026-04-05 5 min read

General

Sources: 1 • Confidence: High • Updated: 2026-04-06 03:42

Key takeaways

The author is working on a major change to the LLM Python library and CLI tool.
To design a new abstraction layer, the author used Claude Code to review Python client libraries for Anthropic, OpenAI, Gemini, and Mistral and to craft curl commands that access raw JSON in streaming and non-streaming modes across scenarios.
Some vendors introduced new features over the past year, including server-side tool execution, that the current LLM abstraction layer cannot handle.
The scripts and captured outputs from the author’s API research have been published in a new repository.
LLM uses a plugin system that provides an abstraction layer over hundreds of LLMs from dozens of vendors.

The author is working on a major change to the LLM Python library and CLI tool.
LLM uses a plugin system that provides an abstraction layer over hundreds of LLMs from dozens of vendors.

To design a new abstraction layer, the author used Claude Code to review Python client libraries for Anthropic, OpenAI, Gemini, and Mistral and to craft curl commands that access raw JSON in streaming and non-streaming modes across scenarios.
The scripts and captured outputs from the author’s API research have been published in a new repository.

Some vendors introduced new features over the past year, including server-side tool execution, that the current LLM abstraction layer cannot handle.

What is the proposed new abstraction layer design (core objects, capabilities, and invariants), and how does it represent features like server-side tool execution?
What will be the migration path and compatibility strategy (breaking changes, deprecations, versioning, and upgrade tooling) for existing LLM users and plugins?
Which specific vendors and APIs currently expose server-side tool execution (and any other new features), and what are the exact semantic differences that break the current abstraction?
What is in the published repository (provider coverage, scenarios tested, automation/CI), and how frequently will it be updated as vendor APIs drift?
Is there any direct operator/product/investor decision readthrough intended (e.g., recommended integration patterns, vendor selection guidance, or tooling commitments), or is this purely exploratory research?

Rapid vendor API evolution is increasing maintenance burden for multi vendor LLM tooling, creating demand for abstraction layers that can express newer capabilities like server side tool execution.
Reproducible cross vendor HTTP and JSON traces may become a competitive feature for developer tooling, improving debugging, portability decisions, and faster adaptation to vendor drift.
Plugin based ecosystems that cover many vendors face potential breaking changes when core abstractions are redesigned, implying near term churn risk but possible long term stickiness if migration is smooth.

A published design for the new abstraction layer that clearly represents server side tool execution and other new features, with defined core objects, capabilities, and invariants.
A concrete migration plan with compatibility strategy including deprecations, versioning, and upgrade tooling, plus evidence that existing plugins can be updated with limited effort.
The repository shows broad provider coverage, automated scenario tests for streaming and non streaming, and regular updates as vendor APIs change.

No clear abstraction design emerges, or key vendor features remain unrepresentable, indicating the redesign does not resolve the identified feature gaps.
Migration appears highly disruptive with widespread plugin breakage and no credible upgrade path, leading to ecosystem fragmentation or abandonment.
The trace repository lacks automation, limited vendor coverage, or becomes stale as APIs drift, reducing its usefulness for grounding abstraction and debugging.