Skip to main content

AI Agent Developer & API Guide

This guide is the technical reference for integrating the plugNmeet AI Agent. It covers the server-to-server room creation API, the in-room protobuf endpoints used by the browser client, and the webhook flow for multi-session continuity.

Under Active Development

The AI Agent engine and its APIs are under active development. Field names, validation rules, and endpoint behavior may change before general release. One-click UI integration for Moodle, WordPress, and Joomla plugins is planned once the feature is finalized and released. Until then, integrate through the API described below.

Where the Configuration Lives​

The AI Agent is configured in the room's insights_features object, under ai_agent_features. The object sits inside metadata.room_features when you call /auth/room/create.

metadata
└── room_features
└── insights_features
├── is_allow
└── ai_agent_features ← this guide
├── auto_session_config
├── mcp_servers
└── custom_provider (BYOK)

Before any room can use the agent, your server must have an insights.services.ai_agent entry in config.yaml, unless the room provides its own custom_provider. See Server & Provider Setup.

ai_agent_features Field Reference​

The table lists all 32 fields of InsightsAIAgentFeatures in plugnmeet_create_room_insights.proto, plus the nested AIAgentAutoSessionConfig, ExternalMCPServerConfig, and InsightsAIAgentCustomProvider messages.

FieldTypeRequiredDescription
is_allowboolYesMaster switch for the AI Agent in this room.
session_modeenumYesAI_AGENT_SESSION_MODE_NORMAL_ROOM (0), AI_AGENT_SESSION_MODE_CONTROLLED_USERS (1), or AI_AGENT_SESSION_MODE_OPEN_PER_USER (2).
auto_session_configobjectDepends on modeRouting settings for automatic 1-on-1 sessions. See auto_session_config.
agent_display_namestringNoName shown for the agent. Default: AI Assistant.
system_promptstringYes, when is_allowHost instructions for the agent's persona and behavior.
previous_session_contextstringNoContext from a previous session, usually the summary JSON or its previous_session_context_prompt.
topicstringYes, when is_allowThe lesson, demo, or training topic the agent stays within.
allowed_subtopicsstring[]NoUp to 64 subtopics that are in scope.
forbidden_topicsstring[]NoUp to 32 topics the agent must decline.
strict_topic_enforcementboolNoDecline out-of-topic requests. Default: true.
topic_anchor_interval_turnsuint32NoRe-inject the topic anchor every N turns. Default 6, range 3 to 12.
spoken_langstring (BCP-47)NoThe learner's or visitor's language, for example en-US.
target_langstring (BCP-47)NoLanguage being learned. Leave empty for non-language topics.
instruction_langstring (BCP-47)NoLanguage the agent explains in. If empty, spoken_lang is used.
voicestringNoProvider voice ID. Must be on the server's allow-list.
speaking_ratefloatNoSpeech speed, 0.5 to 2.0. Default 1.0.
allow_whiteboard_writeboolNoAllow the agent to draw on the whiteboard. Default true.
allow_vision_captureboolNoAllow the agent to receive whiteboard and notepad snapshots. Default true.
vision_debounce_msuint32NoSnapshot debounce. Default 1200, maximum 5000.
vision_max_edge_pxuint32NoLongest snapshot edge. Default 1024, range 512 to 1600.
allow_pollsboolNoAllow quizzes and polls. Default true.
allow_chat_messageboolNoAllow the agent to post in the AI chat. Default true.
allow_notepad_writeboolNoAllow the agent to write to the shared notepad. Default true.
allow_web_fetchboolNoAllow the agent to fetch approved HTTPS pages. Default false.
allowed_knowledge_urlsstring[]Recommended when allow_web_fetch is trueUp to 32 HTTPS URL prefixes the agent may fetch.
knowledge_cache_ttl_secondsuint32NoCache time for fetched pages. Default 3600, range 60 to 86400.
mcp_serversmap<string, ExternalMCPServerConfig>NoUp to 2 external MCP servers. See External MCP Servers.
max_session_secondsuint32NoSession length cap, 60 to 7200. Default 1800 when unset.
max_audio_minutesuint32NoAudio budget per session, up to 240. 0 uses the server default of 120.
is_enabledboolMust be falseInternal runtime flag. The host cannot set it.
active_session_idstringMust be emptyInternal runtime field. The host cannot set it.
custom_providerobjectNoPer-room BYOK provider. See custom_provider.

auto_session_config​

FieldTypeDescription
allowed_participant_idsstring[]Up to 32 participant IDs. Used in CONTROLLED_USERS.
max_concurrent_sessionsuint32Used in OPEN_PER_USER. The server caps this at 16.
auto_start_agentboolOptional. Controls whether the agent starts automatically for routed sessions.

ExternalMCPServerConfig​

FieldTypeDescription
urlstringHTTPS endpoint, maximum 1024 characters.
headersmap<string, string>Optional request headers, up to 16 pairs. Key max 128 characters, value max 2048.
allowed_toolsstring[]Up to 8 tool names. Only these tools are offered to the agent.
scopestringOptional owner hint: shared, text_chat, or ai_agent.

custom_provider (BYOK)​

FieldTypeRequiredDescription
providerstringYesgoogle or openai.
api_keystringYesProvider API key, 1 to 512 characters. Never broadcast.
endpointstringNoHTTPS base URL for a gateway or self-hosted server, up to 512 characters.
auth_headerstringNoHeader name for the key, up to 128 characters.
realtime_modelstringNoOverrides the realtime model.
delegation_modelstringNoOverrides the delegation model.
summarize_modelstringNoOverrides the summary model.

Session Modes​

ModeWho starts a sessionRoutingWebcams & screen shareTypical use
NORMAL_ROOMA moderator, from the Breakout Rooms panelManual 1-on-1 breakout roomsUnchangedTeacher-led practice, sales-led demos
CONTROLLED_USERSAutomatic, for listed participantsEach allowed participant gets <Name> (AI Session)Disabled room-wideAssigned rosters, onboarding groups
OPEN_PER_USERAutomatic, for any visitorEach visitor gets an isolated session, up to max_concurrent_sessionsDisabled room-wide24/7 study rooms, product demo links

In the automatic modes, the main room works as a lobby. Participants who are not authorized wait in the main room.

Request Examples​

All examples post to /auth/room/create. They use the standard API authentication headers described in the Create Room API. Only the relevant parts are shown; other metadata fields are optional.

1. NORMAL_ROOM: Teacher-Led Breakouts​

{
"room_id": "class-algebra-01",
"metadata": {
"room_title": "Algebra Class",
"room_features": {
"allow_webcams": true,
"allow_screen_share": true,
"breakout_room_features": {
"is_allow": true
},
"insights_features": {
"is_allow": true,
"ai_features": {
"is_allow": true
},
"ai_agent_features": {
"is_allow": true,
"session_mode": "AI_AGENT_SESSION_MODE_NORMAL_ROOM",
"agent_display_name": "Math Tutor",
"system_prompt": "You are a patient algebra tutor. Work step by step on the whiteboard and ask the student to predict each step.",
"topic": "Linear equations",
"allowed_subtopics": ["one-variable equations", "word problems"],
"forbidden_topics": ["homework answers for graded tests"],
"strict_topic_enforcement": true,
"topic_anchor_interval_turns": 6,
"spoken_lang": "en-US",
"instruction_lang": "en-US",
"speaking_rate": 1.0,
"allow_whiteboard_write": true,
"allow_polls": true,
"allow_notepad_write": true,
"allow_web_fetch": false,
"max_session_seconds": 1800,
"max_audio_minutes": 60
}
}
}
}
}

2. CONTROLLED_USERS: Assigned Student Roster​

{
"room_id": "tutoring-group-7",
"metadata": {
"room_title": "Tutoring Group 7",
"room_features": {
"insights_features": {
"is_allow": true,
"ai_agent_features": {
"is_allow": true,
"session_mode": "AI_AGENT_SESSION_MODE_CONTROLLED_USERS",
"auto_session_config": {
"allowed_participant_ids": ["student-101", "student-102", "student-103"],
"auto_start_agent": true
},
"agent_display_name": "Study Coach",
"system_prompt": "You are a study coach for high-school chemistry. Ask one question at a time and check each answer with a short quiz.",
"topic": "Chemical bonding",
"allowed_subtopics": ["ionic bonds", "covalent bonds", "electronegativity"],
"forbidden_topics": ["exam cheating", "personal advice"],
"strict_topic_enforcement": true,
"topic_anchor_interval_turns": 6,
"allow_polls": true,
"max_session_seconds": 1800
}
}
}
}
}

allowed_participant_ids accepts up to 32 IDs. Each ID must match the user_id your application uses when it joins a participant.

3. OPEN_PER_USER: 24/7 Self-Service Study or Demo Room​

{
"room_id": "study-room-spanish",
"metadata": {
"room_title": "Spanish Practice Room",
"room_features": {
"allow_webcams": false,
"allow_screen_share": false,
"insights_features": {
"is_allow": true,
"ai_agent_features": {
"is_allow": true,
"session_mode": "AI_AGENT_SESSION_MODE_OPEN_PER_USER",
"auto_session_config": {
"max_concurrent_sessions": 8,
"auto_start_agent": true
},
"agent_display_name": "Lucía",
"system_prompt": "You are a friendly Spanish speaking coach for adult beginners. Speak mostly in simple Spanish and explain grammar in English when needed.",
"topic": "Everyday Spanish conversation",
"allowed_subtopics": ["greetings", "ordering food", "travel vocabulary"],
"forbidden_topics": ["politics", "medical advice"],
"spoken_lang": "en-US",
"target_lang": "es-ES",
"instruction_lang": "en-US",
"speaking_rate": 0.9,
"voice": "YOUR_PROVIDER_VOICE_ID",
"max_session_seconds": 1200,
"max_audio_minutes": 30
}
}
}
}
}

max_concurrent_sessions is capped at 16 by the server, even if you send a higher number. Set voice to a voice ID from your provider's allow-list, or omit it to use the server's default_voice.

4. With BYOK (custom_provider)​

This example uses the host's own OpenAI-compatible gateway for a single room. The key is removed from room metadata before broadcast and stored only in Redis.

{
"room_id": "demo-acme-visitor-room",
"metadata": {
"room_title": "Acme Product Demo",
"room_features": {
"insights_features": {
"is_allow": true,
"ai_agent_features": {
"is_allow": true,
"session_mode": "AI_AGENT_SESSION_MODE_OPEN_PER_USER",
"auto_session_config": {
"max_concurrent_sessions": 4
},
"system_prompt": "You are a visual sales engineer for Acme Analytics. Use the uploaded deck and keep answers short.",
"topic": "Acme Analytics product demo",
"forbidden_topics": ["pricing negotiation"],
"max_session_seconds": 1800,
"custom_provider": {
"provider": "openai",
"api_key": "sk-YOUR-ROOM-SCOPED-KEY",
"endpoint": "https://gateway.example.com/v1",
"auth_header": "Authorization",
"realtime_model": "gpt-realtime-2.1-mini",
"delegation_model": "gpt-5.4-mini",
"summarize_model": "gpt-5.4-mini"
}
}
}
}
}
}

If custom_provider is set, the server does not need an ai_agent service entry in config.yaml. Models you omit fall back to the server defaults for the chosen provider.

5. With External MCP Servers (mcp_servers)​

{
"room_id": "onboarding-acme-001",
"metadata": {
"room_title": "Acme API Onboarding",
"room_features": {
"insights_features": {
"is_allow": true,
"ai_agent_features": {
"is_allow": true,
"session_mode": "AI_AGENT_SESSION_MODE_NORMAL_ROOM",
"system_prompt": "You are a technical onboarding assistant for the Acme API. Check the checklist before each step.",
"topic": "Acme API authentication and webhooks",
"allow_web_fetch": true,
"allowed_knowledge_urls": ["https://docs.acme.example/api/"],
"knowledge_cache_ttl_seconds": 3600,
"mcp_servers": {
"acme-docs": {
"url": "https://mcp.acme.example/mcp",
"allowed_tools": ["search_docs", "read_doc"],
"scope": "ai_agent"
}
}
}
}
}
}
}

Breakout Room AI Sessions​

A moderator can start a 1-on-1 AI session inside a breakout room. Use /api/breakoutRoom/create with these fields from CreateBreakoutRoomsReq:

FieldTypeDescription
enable_ai_agentboolCreates the breakout rooms with an AI agent session attached.
ai_agent_contextstringOptional room-level context for the agent, up to 2000 characters.

Each breakout room also carries its own enable_ai_agent, ai_agent_context (up to 2000 characters), and ai_agent_joined flags. The ai_agent_joined flag tells the moderator's AI Sessions panel whether an agent has joined that room.

{
"room_id": "class-algebra-01",
"requested_user_id": "teacher-1",
"duration": 15,
"enable_ai_agent": true,
"ai_agent_context": "Student is practising two-step equations. Start with the warm-up from last session.",
"allow_return_to_main_room": true,
"rooms": [
{ "title": "Ana", "users": [{ "id": "student-ana" }] },
{ "title": "Ben", "users": [{ "id": "student-ben" }] }
]
}

Child breakout rooms inherit the parent room's BYOK provider configuration, so the same key and models apply to every session.

External Knowledge & External MCP Security​

External access is the highest-risk part of the agent's configuration. The rules below are enforced by the server.

External Knowledge (allow_web_fetch)​

  • Disabled by default. Set allow_web_fetch: true only when you need it.
  • HTTPS only. Plain http:// URLs are refused.
  • Allow-list required. The agent may fetch only URLs that start with an entry in allowed_knowledge_urls (up to 32 prefixes). Use a trailing path, such as https://docs.example.com/api/, to limit scope.
  • Cached. Fetched pages are cached for knowledge_cache_ttl_seconds, which limits repeated requests.
  • Content is untrusted. Fetched text is wrapped as data. It cannot change the agent's rules.

External MCP Servers (mcp_servers)​

  • Maximum 2 servers per room. The limit applies across shared, ai_agent, and text_chat scopes.
  • Maximum 8 allowed tools per server. Tools not listed in allowed_tools are never offered to the agent.
  • Read-only policy. The server enforces a strictly read-only tool policy. Tools that write, delete, or send data are not exposed, even if listed. Design your MCP server so that its read tools are the only ones you expose.
  • HTTPS on port 443 only. Other schemes and ports are refused.
  • SSRF protection. Connections are checked at dial time. Only globally routable addresses are allowed, so connections to private, loopback, link-local, or otherwise non-global addresses are refused.
  • Headers are secrets. Put API tokens in headers. They are stored server-side and not broadcast to participants.
Do Not Point MCP at Internal Services

Even with SSRF protection, connect MCP servers only to services you intend to expose to the AI agent and its visitors. Use a dedicated read-only MCP endpoint, not your internal admin API.

Multi-Session Continuity Flow​

Continuity lets a returning learner or prospect pick up where they left off. The flow has four steps:

  1. Session ends. The server writes an AI_AGENT_SESSION_SUMMARY artifact for the parent room and sends an artifact_created webhook. See Webhooks.
  2. Your backend fetches the artifact. Call /auth/artifact/fetch with type: "AI_AGENT_SESSION_SUMMARY" (or 11) and the room ID.
  3. Your backend prepares context. Either pass the summary JSON, or use its continuation.previous_session_context_prompt value, which is a short prompt written for this purpose.
  4. Next session starts. Send previous_session_context with the next /auth/room/create request, or with the /api/aiAgent/start request.
{
"room_id": "study-room-spanish",
"metadata": {
"room_features": {
"insights_features": {
"is_allow": true,
"ai_agent_features": {
"is_allow": true,
"session_mode": "AI_AGENT_SESSION_MODE_OPEN_PER_USER",
"previous_session_context": "Last session: the learner practised greetings and ordering food. Struggled with the subjunctive. Next focus: travel vocabulary.",
"topic": "Everyday Spanish conversation"
}
}
}
}
}

Per-user continuity works best with OPEN_PER_USER and CONTROLLED_USERS, where your backend knows the learner's identity. Use the user_info.ex_user_id field from the summary to match the learner to your own records.

In-Room Endpoints (/api/aiAgent/*)​

The browser client controls the agent through these endpoints. They are authenticated with the participant's room JWT, not with the server API key. Requests and responses use protobuf (application/x-protobuf), not JSON.

EndpointMethodRequest messageWho may call itPurpose
/api/aiAgent/startPOSTAIAgentStartReqRoom admin onlyStarts the agent. Optionally overrides system_prompt and previous_session_context.
/api/aiAgent/stopPOSTAIAgentStopReqAdmin, current delegate, or pre-authorized participantStops the session. Optional reason, for example host_stopped.
/api/aiAgent/statusGET or POSTAIAgentStatusReqAny participant in the roomReturns AIAgentStatusRes with the current AIAgentStateMsg.
/api/aiAgent/pausePOSTAIAgentPauseReqAdmin, delegate, or pre-authorized participantPauses the agent.
/api/aiAgent/resumePOSTAIAgentResumeReqAdmin, delegate, or pre-authorized participantResumes a paused agent.
/api/aiAgent/sendContextPOSTAIAgentSendContextReqAdmin, delegate, or pre-authorized participantInjects external context, such as a timer event or LMS data. source can be lms or timer.
/api/aiAgent/redelegatePOSTAIAgentRedelegateReqAdmin, delegate, or pre-authorized participantMoves control to another participant. An empty target_user_id lets the server choose.

Responses for start, stop, status, and the other control endpoints use the protobuf AIAgentStatusRes message, which has status, msg, and an optional state.

Runtime States​

AIAgentStateMsg.state reports one of these runtime states: CONNECTING, LISTENING, THINKING, SPEAKING, DRAWING, PAUSED, RECONNECTING, and STOPPED.

Example: Sending Context From an LMS Event​

Build an AIAgentSendContextReq with your protobuf library (for example protobufjs or plugnmeet-protocol-js):

AIAgentSendContextReq {
text: "The student has just submitted quiz 3 with a score of 4 out of 5. Review the missed question."
source: "lms"
}

Post the encoded bytes to /api/aiAgent/sendContext with the participant's room JWT in the Authorization header.