# OpenAppleModels: Tool Calling for Apple Foundation Models

> open-apple-models is an MIT-licensed Swift package on Apple's FoundationModels framework (iOS, iPadOS, macOS and visionOS 27). It adds per-step tool steering, runtime JSON-Schema tools (local closures or host-executed), a game layer (NPC dialogue, decisions, content generation), the `oam` CLI, an OpenAI-compatible server that returns real `tool_calls`, and a JSON-RPC 2.0 protocol (v1.0) over stdio or a C ABI. Pre-release: no tagged releases, depend on `main`. Library version 0.1.0.

## Core capabilities
- **Per-step tool steering (`SteeredLanguageModel`)**: `ToolChoice` applies to the first model step of a turn only (`.auto`, `.none`, `.required`, `.explicit`, `.tool(name)`), so a forced tool call cannot loop. Budgets: `maxToolRounds` (default 4) and `maxToolCalls` (default 12).
- **`.explicit`**: the model must call a tool or a built-in `respond_directly` tool. Default for `NPC`; `Agent` defaults to `.auto`.
- **External tools**: `AgentTool.external(...)` has no Swift handler; the turn pauses until the host submits the result. No time limit unless the tool sets `timeout`.
- **Game layer (`OpenAppleModelsGame`)**: `NPC`, `Persona`, `WorldState`, `DecisionEngine`, `ContentGenerator`. NPC memory is facts plus a relationship score (-100 to 100) and a summary; secrets unlock at relationship 50 by default.
- **OpenAI-compatible server (`OpenAppleModelsServer`, `oam serve`)**: default `http://127.0.0.1:1976/v1`, model id `"system"`. Supports `tool_choice`, parallel calls, streaming and `json_schema` responses.
- **Engine bridge**: JSON-RPC 2.0 protocol v1.0 over `oam stdio` or the C ABI `libOpenAppleModelsFFI` (`oam_bridge_create`, `oam_bridge_send`, `oam_bridge_destroy`, `oam_call_blocking`). Binding code exists for C, Python and Unity (C#); Godot and Unreal have integration notes only.
- **Testing**: `ScriptedLanguageModel` (`OpenAppleModelsTesting`) plays scripted steps, so tests and engine integrations run without Apple Intelligence (they still need a macOS 27 host).

## Measured (macOS 27 only; nothing measured on iPhone, iPad or Vision Pro)
- Apple's `fm serve` returned 0 tool calls in 54 requests; a forced tool gave HTTP 500.
- FoundationModels' own `.required` mode looped (40+ calls in one `respond`, 3 of 3 runs).
- Eight tavern lines: `.auto` chose the right action 7/8 (1.2 s average), `.explicit` 8/8 (1.5 s).
- Enum decisions were valid 10 of 10 times. See RESEARCH.md and GAMES.md for all numbers.

## Quick API reference

### Agent with a tool
```swift
import OpenAppleModels

let inventory = try AgentTool(
    name: "check_inventory",
    description: "Look up how many of an item the blacksmith has and its price in gold.",
    parameters: .object(["item": .string(description: "Item name")])
) { call in
    let item = try call.string("item")
    return .json(["item": .string(item), "stock": 3, "price_gold": 45])
}

let gorm = try Agent(
    instructions: "You are Gorm, a grumpy blacksmith in a fantasy game. Reply in at most two sentences.",
    tools: [inventory])
let reply = try await gorm.respond(to: "Got any iron swords? How much?",
                                   policy: ToolPolicy(choice: .required))
print(reply.text)
```

### NPC
```swift
import OpenAppleModelsGame

let world = WorldState(["player": ["name": "Aria", "gold": 60], "time_of_day": "evening"])
let smith = try NPC(
    persona: Persona(
        name: "Gorm", role: "the village blacksmith",
        personality: "Gruff and proud, but fair", speakingStyle: "Short, blunt sentences. Calls people 'lad'."),
    tools: [inventory],
    world: world,
    options: NPCOptions(
        groundingTool: "check_inventory",
        worldContextPaths: ["player.name", "player.gold", "time_of_day"]))
let turn = try await smith.talk("Evening! Got any iron swords? How much?")
print(turn.emotion, turn.line, turn.playerOptions)
```

### CLI (`oam`)
```bash
oam available                                   # exit 3 with a reason when the model is unavailable
oam respond --tools tools.json --tool-choice required 'What is the weather in Paris?'
oam respond --tool-json '{"name":"get_gold","description":"Player gold","parameters":{"type":"object","properties":{}}}' 'How much gold do I have?'
                                                # a tool without a command is external: exit 10 with the pending calls
oam serve                                       # OpenAI-compatible server on 127.0.0.1:1976
oam stdio                                       # JSON-RPC bridge over stdin/stdout
oam demo tavern                                 # NPC innkeeper demo
```

Exit codes: 0 success, 1 failure, 2 usage, 3 model unavailable, 4 guardrail or refusal, 5 context exceeded, 6 rate limited, 10 tool calls pending, 130 interrupted.

## Limits
- 8,192-token context shared by instructions, tool definitions and the conversation.
- Guardrails block some ordinary game content, more often in structured (JSON) output than in plain text.
- A rare upstream crash in streaming with tool calls; `AgentConfiguration(streamsResponses: false)` avoids streaming.

## Android
[open-android-models](https://github.com/SpaceCorps/open-android-models) is the pre-release Kotlin sibling on Gemini Nano (ML Kit GenAI Prompt API). It speaks the same JSON-RPC protocol v1.0 over JNI and has not yet run on a Gemini Nano device.

## Documentation links
- [README](https://github.com/SpaceCorps/open-apple-models#readme)
- [Full agent manual (llms-full.txt)](https://spacecorps.github.io/open-apple-models/llms-full.txt)
- [Architecture (ARCHITECTURE.md)](https://spacecorps.github.io/open-apple-models/ARCHITECTURE.md)
- [CLI reference (CLI.md)](https://spacecorps.github.io/open-apple-models/CLI.md)
- [Game layer (GAMES.md)](https://spacecorps.github.io/open-apple-models/GAMES.md)
- [JSON-RPC protocol (PROTOCOL.md)](https://spacecorps.github.io/open-apple-models/PROTOCOL.md)
- [Server (SERVER.md)](https://spacecorps.github.io/open-apple-models/SERVER.md)
- [Research and measurements (RESEARCH.md)](https://spacecorps.github.io/open-apple-models/RESEARCH.md)