Capabilities

Tools

Tool factories bind every tool to an explicit workspace:

import { createBuiltinTools, createReadOnlyTools } from "./src/libs/index.ts";

const codingTools = createBuiltinTools("/work/project");
const reviewTools = createReadOnlyTools("/work/project");

The read-only set leaves out file writes and the shell but keeps web research, which suits explorer and reviewer agents.

• search searches DuckDuckGo, Google or Bing on a rendered results page in the Playwright browser (so it needs the browser), collects up to 50 results, and opens with a breakdown by kind, site and recurring theme, so the model explains sources instead of pasting links.
• research(query, pages: 5) searches, then reads the top pages in parallel and returns numbered sources to cite as [n].
• fetch_url turns pages and documents (PDF, DOCX, PPTX, …) into clean Markdown with code snippets, images and media, and transcribes audio and video. Pages that come back empty are rendered in the browser.

Loading tools on demand

Every tool schema is sent with every request, so put rarely needed tools in a ToolGroup. The agent gets a load_tools tool listing each group; a group joins the run when the model loads it or calls one of its tools by name.

import { Agent, HttpSpeech, ToolGroup, createAudioTools, speechConfigFromSettings } from "./src/libs/index.ts";

const speech = new HttpSpeech(speechConfigFromSettings(), { workspace: root });
const agent = new Agent({
  name: "assistant",
  tools: coreTools,
  deferredTools: [new ToolGroup("audio", "speak aloud and save speech audio", createAudioTools(speech))],
});

The CLI keeps files, shell, web search, workspace, knowledge-graph and skill tools always on, and defers audio, images, media, subagents, schedules, git, tool_builder and each MCP server. /tools lists them.

Progressive discovery. Specialized tools and MCP servers are found only when a task needs them: discover_server matches the intent to a server or group, discover_tools finds the operation, get_tool_schema fetches just that schema, and invoke_tool calls it with the usual validation, permission checks, audit and cancellation. select trims large JSON results to the fields you need.

Authorization

authorizeTool runs before every call. Return true to allow, or false or a reason to deny; the model gets the reason and can recover.

import { Agent, createBuiltinTools, type ToolCall } from "./src/libs/index.ts";

const reviewPolicy = (call: ToolCall) =>
  ["write_file", "edit_file", "run_command"].includes(call.name) ? "review agents are read-only" : true;

const agent = new Agent({ name: "reviewer", tools: createBuiltinTools(), authorizeTool: reviewPolicy });

Git

The deferred git group gives the agent git_status, git_diff, git_log, git_show, git_branch and git_commit for the workspace repository. They run git with an argument list, not a shell, and refuse refs and paths that start with -.

git_commit stages exactly the paths it is given and commits only them, leaving other staged or unstaged work alone. Push, reset and rebase are not included. For other repositories, the bundled git MCP server adds git_add, git_reset (unstage only), git_checkout, git_create_branch and git_init as well; it works on the workspace and any folder listed in GIT_MCP_ROOTS. See Bundled MCP servers. plan and ask modes treat git_branch and git_commit as edits. In the SDK, createGitTools(workspace) returns the same tools.

Tools the agent writes itself

When no tool fits, the agent can write one with create_tool: a small TypeScript tool that runs in a separate, time-limited Bun child process in the workspace. The CLI always asks before creating or running one.

Web research & citations

search, fetch_url and research tell the model to cite what it uses, and the agent checks the final reply: if the turn read web pages but the reply names none of them, a Sources list is appended with each page's title and URL (keeping research's [n] numbers):

Bun 1.2 adds built-in S3 support and a text lockfile.

Sources:
1. Bun 1.2 release notes — https://bun.sh/blog/bun-v1.2

Images found while reading a page are saved in a folder per page, .andromeda/assets/web/<site>-<page>-<hash>/, with a manifest.json listing the source page, its title, and each image's file, URL and caption, so they can be shown again without re-scraping.

Browser

The agent drives a real browser through Bun Playwright. It reads each page as an accessibility snapshot whose elements carry refs, and acts on those refs instead of guessing selectors:

browser_open("news.ycombinator.com")
Page: Hacker News
Snapshot (act on elements by ref, e.g. browser_click(ref='e12')):
- link "new" [ref=e17] -> newest
- link "147 comments" [ref=e51] -> item?id=49823582
...
browser_click(ref="e51")

Tools: browser_open (always on), and in the on-demand browser group browser_snapshot, browser_click, browser_type, browser_select, browser_press, browser_back, browser_wait, browser_read, browser_screenshot and browser_close, plus the same wider set as the TypeScript edition: browser_search (DuckDuckGo, Google or Bing on a rendered results page that stays open), browser_tabs, browser_find, browser_scroll, browser_hover, browser_drag, browser_click_at, browser_fill_form, browser_file_upload, browser_drop, browser_resize, browser_emulate_media (dark mode, print), browser_handle_dialog, browser_evaluate (JavaScript), browser_run_code (a JavaScript Playwright script), browser_console_messages, browser_network_requests and browser_network_request. Opening a workspace .html file serves it locally so its CSS and scripts load, which is handy for checking pages the agent builds.

Compared with the Playwright MCP server, the built-in browser sends far smaller snapshots, which matters for small local models, and only browser_open is loaded until the agent needs more. browser_run_code always asks first. Clicks, typing, selection and key presses are refused in plan mode and ask first in ask mode; the agent is also told to ask before buying, posting, sending or deleting anything.

browser:
  enabled: true
  engine: chromium     # chromium | firefox | webkit
  channel: ""          # or chrome, msedge to use an installed browser
  headless: false      # the window is visible by default here
  idle_timeout: 900    # seconds before an idle browser closes

bunx playwright install chromium   # once; Playwright is an optional dependency

Saved browser logins

The encrypted vault lets Andromeda sign in to a site without revealing the password to the model. Add and manage entries from a terminal:

bun run main.ts vault add "Electric company" --url https://my.utility.example --username me@example.com
bun run main.ts vault list
bun run main.ts vault remove "Electric company"

Passwords are encrypted in .andromeda/vault.json; the key is protected by the operating system keychain where available. The agent can list login names and use browser_login on the matching site. Login always asks for approval, never returns the password, and disables script evaluation on that page until it navigates away.

Desktop control

On macOS, Windows and Linux desktops the on-demand computer group sees and drives the desktop: computer_screenshot, computer_move, computer_click, computer_type, computer_press, computer_scroll and computer_open_app. The agent takes a screenshot and looks at it with view_image before acting, and opens apps with computer_open_app rather than shell commands.

Linux and Ubuntu

Linux needs a desktop session; on a server or over SSH the computer tools are left out. The session type picks the tools:

• Apps open by their menu name ("Files", "Firefox") through gtk-launch and the .desktop entry, or by command name.
• Keys: in computer_press, cmd means Ctrl on Linux and super is the Windows key; letters, digits and F1–F12 also work.
• Wayland without ydotool: xdotool can still drive apps running under XWayland, but not native Wayland windows.
• Display scaling: with fractional scaling, screenshot pixels and pointer coordinates can differ; 100% scaling keeps them matched.

Skills

A skill is a folder with a SKILL.md: frontmatter plus task instructions. Skills live under skills/ or .agents/skills/.

---
name: release-check
description: Verify a release candidate before publishing.
---

Read the changelog, run `{baseDir}/check.sh`, and summarize failures.

Only the frontmatter is read at startup. The model finds skills with find_skills(query) and reads one with load_skill(name). Each turn, up to two clearly matching skills are suggested. A skill written for Andromeda (agent: andromeda) is included outright when it ranks highly or when the request contains one of its triggers:

triggers: [website, web page, landing page, html, css]
agent: andromeda

Skills written for other agents often name tools like Bash or WebFetch; load_skill adds a note mapping each to its equivalent here. /skills [query] searches them in the CLI, and teach mode creates new ones.

Install a skill with /skills install <source>: a GitHub repository or a folder in one (anthropics/skills/skills/pdf), a raw SKILL.md URL, or a local folder. The web Skills page can install every skill in a repository or folder at once. Every file is scanned first; prompt injection, download-and-run scripts, reverse shells, SSH key writes and symlinks block the install, and a skill whose instructions look like an injection is quarantined. When a skill turns out wrong, update_skill patches or rewrites it and delete_skill retires it; only workspace skills can change, and old versions go to .andromeda/skill-archive/. /skills disable <name> switches one off (saved in skills.disabled), and any skill works as a command: /pdf summarize report.pdf.

Voice & audio

Speech output uses Kokoro-82M (default) or Qwen3-TTS; speech input uses Whisper with voice activity detection. Everything runs locally, or TTS can stream from your own HTTP server. No Python is involved: Whisper and Kokoro run in-process on ONNX Runtime (downloaded on first use into data/models/onnx), and Qwen TTS and voice cloning run through a native qwen-tts built from qwentts.cpp into vendor/. The microphone is captured with ffmpeg.

Talk to Andromeda

• Press Ctrl+R or type /voice, or start with bun run main.ts --listen. Pause to send; replies are spoken sentence by sentence as the model writes them.
• Esc stops a spoken reply; say "stop listening" to turn it off.
• On macOS, give your terminal microphone access under System Settings → Privacy & Security → Microphone.
• Ask "use kokoro" or "use qwen" to switch voices mid-session, if that provider is installed.

Switch the speaking engine with /voice kokoro or /voice qwen [VOICE], or set voice.provider; the choice covers voice mode, voice notes and calls. Qwen has nine preset voices, an instruct style prompt and ten languages.

Choose a voice and its delivery

Here are the voice IDs you can select. The guided picker (bun run main.ts setup), /voice voices, /voice preview NAME and /voice choose NAME work with Kokoro quick picks. Qwen voices are selected with /voice qwen NAME.

Kokoro has additional model voices beyond these four quick picks. Load the audio tool group and call list_local_voices to see all installed Kokoro voices and saved Qwen clones.

Configure each voice engine

Choose one provider in andromeda.yml. Kokoro uses a voice ID and language code (a for American English, b for British English):

voice:
  provider: kokoro
  kokoro:
    voice: bf_lily
    language: b

Qwen uses a preset ID and language label. Add a delivery instruction in code per synthesis request, such as “warm and calm”; there is no saved tone setting in YAML:

voice:
  provider: qwen
  qwen:
    voice: Aiden
    language: English

To use a compatible external TTS endpoint, configure its full speech URL and API style, then start with andromeda --tts http. Use openai for an OpenAI-compatible endpoint or simple for a Kokoro-style endpoint:

voice:
  http:
    url: http://localhost:8880/v1/audio/speech
    api_style: openai
    voice: af_heart
    model: kokoro

Use the local speech SDK directly when you want to make audio in your own Bun program. Both providers return a complete WAV as Uint8Array; write it with Bun.write. The first synthesis downloads model files into voice.model_dir (by default under data/models), so allow time and disk space for that first run.

1. Set the provider and voice

In andromeda.yml, choose the default provider. These settings also apply when you create an SDK instance with loadSettings():

voice:
  provider: kokoro # or qwen
  kokoro:
    voice: af_heart
  qwen:
    voice: Ryan
    language: English

Install the project dependencies with bun install. Kokoro runs on CPU through ONNX Runtime and fetches its model on first use. Qwen uses the native qwen-tts program. If it is not already installed, Andromeda builds it into vendor/ on first use; building needs git and cmake. Qwen downloads about 1.3 GB of model and codec files the first time. You can set voice.qwen.binary to an existing compatible binary.

2. Make a Kokoro WAV

import { loadSettings, localSpeechFromSettings } from "./src/libs/index.ts";

const speech = localSpeechFromSettings(loadSettings());
const voices = await speech.voices("kokoro");
console.log(voices.slice(0, 8).map(({ voiceId, language }) => ({ voiceId, language })));

const { audio, format } = await speech.speak({
  text: "Hello! This audio was made with Kokoro.",
  provider: "kokoro", // optional when voice.provider is already kokoro
  voice: "af_heart",
  speed: 1,
});
await Bun.write(`greeting.${format}`, audio); // greeting.wav

Use bf_lily or bm_george for British English, or call speech.voices("kokoro") to inspect the model’s full voice catalog.

3. Make speech with Qwen

Qwen has preset speakers, language selection and a delivery instruction. The instruction guides how the preset voice speaks; it does not change the text:

import { loadSettings, localSpeechFromSettings } from "./src/libs/index.ts";

const speech = localSpeechFromSettings(loadSettings());
const voices = await speech.voices("qwen");
console.log(voices.map(({ voiceId, language }) => ({ voiceId, language })));

const { audio, format } = await speech.speak({
  text: "Welcome back. How can I help?",
  provider: "qwen",
  voice: "Aiden",
  language: "English",
  instruct: "Warm, calm, conversational",
});
await Bun.write(`qwen-greeting.${format}`, audio);

Supported preset IDs are Ryan, Aiden, Vivian, Serena, Uncle_Fu, Dylan, Eric, Ono_Anna and Sohee. Pass a language such as English, Chinese, Japanese or Korean, or use auto. Omit provider to use voice.provider from the YAML settings.

4. Clone a speaker for one request

Give Qwen a short, clean reference recording and its exact transcript. This uses the Qwen Base model configured by voice.qwen.clone_model, instead of the preset speaker model:

const { audio, format } = await speech.speak({
  text: "This sentence uses the reference speaker.",
  provider: "qwen",
  referenceAudio: "./reference.wav", // path to a readable local audio file
  referenceText: "The exact words spoken in reference.wav.",
  language: "English",
});
await Bun.write(`cloned-greeting.${format}`, audio);

For repeat use, the agent tools clone_local_voice, list_cloned_voices and synthesize_local_speech save a named clone under .andromeda/voices/ and select it with speaker: "name". Voice recordings are biometric data, so the clone tool asks for approval. See the configuration reference for custom model paths and cache locations.

speech.speak() returns { audio, format: "wav" }; audio is not a file path until you write it. For playback, use the synthesize_local_speech tool with play: true. Inline delivery tags such as [calm], [excited], [whispers] and [serious] can adjust local speech. This spoken delivery is separate from Andromeda's conversational writing style.

Emojis and tags like [laughs] or [whispers] are acted out rather than read aloud. Qwen can clone a voice from 5–15 seconds of clean speech; only clone voices you have permission to use.

In your own code

import { Agent, HttpSpeech, speechConfigFromSettings } from "./src/libs/index.ts";

const agent = new Agent({ name: "assistant", model: "gpt-oss:20b" });
const speech = new HttpSpeech(speechConfigFromSettings());   // voice.http in andromeda.yml

const reply = await agent.run("Say hello in one short sentence.");
await speech.play({ text: reply.output });
const [path] = await speech.synthesizeToFile({ text: reply.output }, "hello.wav");

For local speech, localSpeechFromSettings() gives a LocalSpeech (Whisper plus Kokoro or Qwen), and LocalVoicePipeline and LocalVoiceConversation chain it with an agent for hands-free talk. The agent's local_speech tools (synthesize_local_speech, transcribe_local_audio, clone_local_voice, …) use the same models.

Mobile voice

The remote voice server connects Andromeda to the Andromeda mobile app. Speech is transcribed, the agent responds, and audio streams back while the reply is still being generated.

bun run main.ts voice serve --host 0.0.0.0   # listen on your home network
bun run main.ts voice pair                    # show pairing QR code
bun run main.ts voice token                   # show the access token

The server uses a random access token and supports interruption when you speak over a reply. Keep it on 127.0.0.1 unless LAN access is needed; for access from outside your home, use Tailscale or a TLS proxy and configure voice.remote.public_url. Calls that require tool approval are declined because there is no approval prompt on the phone.

Images

Andromeda generates and edits images, and can look at them with a vision-capable model to check its own edits. The vision model is served by the configured chat engine.

• Ask in plain language ("draw a lighthouse at dusk, watercolor", then "make the sky orange"), or use /image <prompt> and /image edit <change>.
• edit_image keeps the original, then asks the vision model whether the change is really there, so the agent can try again.
• view_image describes an image or answers a question about it. With Ollama, the default is qwen2.5vl:3b (ollama pull qwen2.5vl:3b). With Docker Model Runner, select a DMR model that supports image input; the example smollm2 and Nomic embedding models are text-only.

Images are saved as <timestamp>-<prompt>-AI-generated.png in data/images/ unless you set image.output_dir. The first generation downloads the model into data/models/. On a 32 GB Apple silicon Mac, 1024×1024 at 40 steps takes about 4.5 minutes; 512×512 at 12 steps takes under one.

import { ImageGenerator, imageConfigFromSettings } from "./src/libs/index.ts";

const images = new ImageGenerator(imageConfigFromSettings(), { workspace: "." });
const result = await images.generate('A neon shop sign that reads "ANDROMEDA", rainy night');
const edited = await images.generate("Change the background to a sunset beach", { images: [result.path] });

Ollama image models run directly; the qwen provider runs Qwen-Image through stable-diffusion.cpp. Pick the engine with image.provider, /image qwen|ollama, or generate_image provider=. Weights are local paths or owner/repo/file on Hugging Face (GGUF or safetensors).

Videos

Short clips come from Wan 2.1 T2V 1.3B running locally through stable-diffusion.cpp's sd-cli, which installs on first use (or with bun run preflight). The weights download into data/models/ and clips are saved in data/videos/; video.wan.* takes local paths or Hugging Face files.

Ask for a clip ("a fox running through snow, low tracking shot"). generate_video runs as a deferred background task, so you can keep working; the agent tells you when the MP4 is ready. The default is 832×480, 33 frames at 16 fps (about two seconds); raise num_frames to 81 for about five.

video:
  output_dir: videos  # "" = data/videos
  width: 832
  height: 480
  num_frames: 33      # 4k+1; 81 is the model's maximum
  fps: 16
  steps: 30

Schedules

Ask Andromeda to do something on a schedule ("every weekday at 9, summarize my open PRs"), and it creates a durable cron schedule. Schedules live in SQLite, survive restarts, and their results appear in the conversation even when nobody is typing.

import { ScheduleService } from "./src/libs/index.ts";

const schedules = new ScheduleService(".andromeda/schedules.db", new Map([["reporter", agent]]));
await schedules.create({
  id: "daily-report",
  agentId: "reporter",
  cron: "0 9 * * 1-5",
  timezone: "Africa/Johannesburg",
  prompt: "Prepare the morning report.",
});
await schedules.start();

Five-, six- and seven-field cron expressions and @hourly, @daily, @weekly, @monthly and @midnight are supported. Schedules can be paused, resumed, run now or deleted. The CLI stores them in the app's own .andromeda/ directory whatever the workspace; set ANDROMEDA_STATE_DIR to move it.

Object detection

yolo_detect runs Ultralytics YOLO locally on a workspace image, or on one camera frame when no path is given, and saves an annotated image under .andromeda/yolo/. start_camera_monitor keeps watching at an interval (camera_monitor_status, stop_camera_monitor; or /camera start|status|stop). Andromeda asks before each camera use. Install it with bun run setup:yolo.