Measured Apple's fm serve returned 0 tool calls in 54 requests

Real Tool Calling & Agent Loops for
Apple Foundation Models

An MIT-licensed Swift package that brings per-step tool calling, agent loops, NPC dialogue with memory, and an OpenAI-compatible local server to Apple's on-device Foundation Models on iOS, iPadOS, macOS and visionOS 27. Pre-release: no tagged releases yet, and all measurements so far come from macOS 27.

$ git clone https://github.com/SpaceCorps/open-apple-models && cd open-apple-models && swift build -c release
$0.00
Cloud API Invoices
100%
On-Device Privacy
333
Swift Tests
10/10
Valid Enum Decisions (macOS 27)

The Probing Benchmark

Measured on macOS 27 before building anything: Apple's fm serve returned 0 tool calls in 54 requests, and the framework's own .required mode never stopped calling tools. Details in RESEARCH.md.

Native Apple FoundationModels vs. OpenAppleModels

Measured on macOS 27
Feature / Scenario Apple Native (fm serve / SPM) OpenAppleModels Runtime
Tool Calls Emitted (Tool Mode) āœ• 0 / 54 Emitted (Always answered directly) āœ“ Real tool_calls from oam serve and the Swift API
Enforcing tool_choice: "required" āœ• HTTP 500 Internal Server Error āœ“ Forced on the first step, then the model answers
Multi-Turn Loop Behavior āœ• .required loops (40+ calls, 3 of 3 runs) āœ“ Round, call and per-step tool limits
Direct Response Fallback āœ• .allowed often skips tools and invents facts āœ“ .explicit: a tool or respond_directly
Game Engine Bridging āœ• Not Provided (Swift framework only) āœ“ JSON-RPC 2.0 over stdio or a C ABI (C, Python, Unity bindings)
NPC Personas & Memory āœ• No High-Level Dialogue Primitive āœ“ Personas, facts, relationship score, secrets
CI/CD Mock Testing āœ• Not included āœ“ ScriptedLanguageModel (tests still need a macOS 27 host)

What Is in the Package

Six library products and the oam CLI, for game developers who want on-device NPCs and AI decisions, and for anyone who needs real tool calls from Apple's model.

šŸ”„

Per-Step Loop Control

Force a tool call on step 1 to ground queries against game inventory, SQLite, or player stats, then seamlessly transition to natural language generation or structured schemas.

šŸ›”ļø

Explicit Mode (.explicit)

The model must call a tool or a built-in respond_directly tool, which grounds lookups without forcing a pointless call on small talk. On eight tavern lines it chose the right action 8/8, against 7/8 for .auto.

āš”ļø

Game NPC Engine

OpenAppleModelsGame provides NPC personas with memory (facts, a relationship score from -100 to 100, a summary), secrets that unlock with the relationship, world state, decisions and content generation. A blocked reply is retried once as plain text, then replaced by a fallback line.

🌐

OpenAI-Compatible Server

A replacement for fm serve on http://127.0.0.1:1976/v1 (model "system") whose tool_calls work, with tool_choice, streaming and json_schema responses.

šŸŽ®

Game Engine Bridge

JSON-RPC 2.0 protocol v1.0 over oam stdio or the C ABI (OpenAppleModelsFFI), with tools executed by the host. Bindings for C, Python and Unity; integration notes for Godot and Unreal.

šŸ”’

On-Device, No API Key

By default everything runs on the device's own model, with no API key or cloud service, and the package sends no telemetry. Use of the model is subject to Apple's acceptable use requirements.

Simple, Declarative Code

Swift, the oam CLI, any OpenAI client, or a game engine over JSON-RPC. These examples are taken from the README, where they compile against the current code.

import OpenAppleModels

let inventory = try AgentTool(
    name: "check_inventory",
    description: "Look up how many of an item the blacksmith has and its price in gold.",
    parameters: .object(["item": .string(description: "Item name")])
) { call in
    let item = try call.string("item")
    return .json(["item": .string(item), "stock": 3, "price_gold": 45])
}

let gorm = try Agent(
    instructions: "You are Gorm, a grumpy blacksmith in a fantasy game. Reply in at most two sentences.",
    tools: [inventory])

// .required: the model must call a tool on the first step, then answers from its output.
let reply = try await gorm.respond(to: "Got any iron swords? How much?",
                                   policy: ToolPolicy(choice: .required))
print(reply.text, reply.toolCalls.count)
import OpenAppleModelsGame

let world = WorldState(["player": ["name": "Aria", "gold": 60], "time_of_day": "evening"])

let smith = try NPC(
    persona: Persona(
        name: "Gorm", role: "the village blacksmith",
        personality: "Gruff and proud, but fair", speakingStyle: "Short, blunt sentences. Calls people 'lad'."),
    tools: [inventory],                                     // any AgentTool, local or external
    world: world,                                           // adds a read_world_state tool
    options: NPCOptions(
        groundingTool: "check_inventory",                   // look up stock on every turn
        worldContextPaths: ["player.name", "player.gold", "time_of_day"]))

let turn = try await smith.talk("Evening! Got any iron swords? How much?")
print(turn.emotion, turn.line, turn.playerOptions)
// Live run of a similar setup (docs/GAMES.md): called check_inventory, then
// "Aria, I've got three iron swords for 45 gold. Take one if you're keen." in 4.4-5.2 s.
# Start the server first: oam serve   (127.0.0.1:1976)
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:1976/v1", api_key="unused")

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Current weather for a city.",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
    },
}]

response = client.chat.completions.create(
    model="system",
    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
    tools=tools,
    tool_choice="required",
)

for tool_call in response.choices[0].message.tool_calls:
    print(tool_call.function.name, tool_call.function.arguments)
# Force a tool call with an inline tool. A tool without a command is external:
# oam exits with code 10 and prints the pending calls and a transcript path.
oam respond 'How many potions do I have?' --tool-choice required \
  --tool-json '{"name":"get_potions","description":"Count potions in the inventory","parameters":{"type":"object","properties":{}}}'

# OpenAI-compatible local server (127.0.0.1:1976)
oam serve

# Talk to Mira, an NPC innkeeper with a menu tool and a till
oam demo tavern

# JSON-RPC bridge over stdin/stdout for game engines
oam stdio
// A client tool call over oam stdio, after initialize (id 1); some session/event notifications left out
→ {"jsonrpc":"2.0","id":2,"method":"session/create","params":{"session":"guard","instructions":"You are a castle guard. Use tools to act.","tools":[{"name":"open_gate","description":"Open a named gate.","parameters":{"type":"object","properties":{"gate":{"type":"string"}}}}],"options":{"toolChoice":"required"}}}
→ {"jsonrpc":"2.0","id":3,"method":"session/respond","params":{"session":"guard","prompt":"Please open the north gate.","stream":true}}
← {"jsonrpc":"2.0","id":"t-1","method":"tool/call","params":{"session":"guard","requestId":3,"call":{"id":"call_NYH7…","name":"open_gate","arguments":{"gate":"north"}}}}
→ {"jsonrpc":"2.0","id":"t-1","result":{"output":{"opened":false,"reason":"the portcullis chain is jammed"}}}
← {"jsonrpc":"2.0","id":3,"result":{"session":"guard","text":"The north gate remains shut because the portcullis chain is jammed.",…}}

Deep Technical Documentation

Comprehensive guides covering architecture, engine bridging, protocols, and benchmarking.

šŸ“

Architecture & Design →

Layers, a turn step by step, steering, errors and concurrency.

šŸŽ®

Game AI Layer →

WorldState, Persona, NPC, DecisionEngine, ContentGenerator, prompting tips, guardrails and measured performance.

šŸ“”

JSON-RPC 2.0 Protocol →

The JSON-RPC protocol v1.0: framing, ordering, session/respond, tool/call, npc/talk, decision/decide and the scripted model.

šŸ–„ļø

CLI Command Reference →

Command flags, environment variables, exit codes, and pipeline examples for the oam binary.

🌐

OpenAI HTTP Server Spec →

Request mapping, endpoints, errors, configuration, security, and a comparison with fm serve.

šŸ“Š

Probing Data & Research →

Measurements of fm, fm serve and the framework that shaped the design, all on macOS 27.