‍

SightLab’s AI Agent workflow makes it possible to add interactive, intelligent virtual characters into VR and XR simulations. These agents can use large language models such as OpenAI, Anthropic Claude, Gemini, or offline Ollama models, and can be paired with custom avatars, voices, personalities, animations, speech recognition, and even event-based behaviors.

This guide walks through the recommended workflow for setting up, customizing, and running an AI Agent in SightLab.

See this page for more information. 

Additionally, for adding instructional AI tutors and agents in the E-Learning Lab see this page.

What You Can Do with the AI Agent

The AI Agent can be used to create virtual humans, tutors, guides, interviewers, simulation partners, or interactive characters inside a SightLab project.

Key capabilities include:

Quick Start

To try the AI Agent with the default setup:

1. Open the AI Agent folder.
2. Run AI_Agent_GUI.py to use the GUI workflow, or run AI_Agent.py for the non-GUI version.
3. Select your environment, hardware, AI model, voice, and agent options.
4. Press Continue.
5. Press Spacebar to start the scene
6. Hold the c key or the right-hand grip button to speak.
7. Release the button to let the agent respond.

Tip: Run multi_agent_interaction.py to see an example with two AI agents talking to each other.

Setting API Keys

Online models require API keys. Offline Ollama models do not require keys.

The recommended method is to set API keys as Windows environment variables. Open Command Prompt and run:

setx OPENAI_API_KEY your-api-key

setx ANTHROPIC_API_KEY your-api-key

setx GEMINI_API_KEY your-api-key

setx ELEVENLABS_API_KEY your-api-key

After setting the keys, restart Vizard.

You can also input your keys in the SightLab Dashboard settings, or simply place them in the ‘keys’ folder in the AI_Intelligent_Agent main folder named as openai_api_key.txt, gemini_api_key.txt, eleven_labs_api_key.txt and gemini_api_key.txt. 

Use these services to obtain keys:

For offline models, install or launch Ollama and run a model from Command Prompt, for example:

ollama run gemma

The first launch may take longer while the model loads.

Creating a Custom AI Agent

The custom agent workflow has three main parts:

1. Create a prompt.
2. Create an avatar configuration.
3. Add the avatar to your scene.

Step 1: Create a Custom Prompt

The prompt defines the agent’s personality, knowledge, role, and behavior.

1. Open the prompts folder.
2. Make a copy of default.txt.
3. Rename it, for example:

MuseumGuide.txt

4. Edit the prompt to describe the agent’s role.

Example prompt structure:

You are a friendly museum guide inside a virtual art gallery.

You explain paintings clearly and conversationally.

You ask participants reflective questions about what they notice.

Keep responses concise unless the user asks for more detail.

‍

Step 2: Create an Avatar Config

Each avatar uses a config file to control animations, voices, language, morph targets, and other avatar-specific settings.

1. Open the configs folder.
2. Copy either:

Template_Male.py

or

Template_Female.py

3. Rename the file, for example:

MuseumGuideAvatar.py

4. Open the avatar in Inspector.
5. Check the animation list in the lower-left Animations panel.
6. Note the animation state numbers for idle and talking.
7. In your avatar config file, update:

TALK_ANIMATION = 2

IDLE_ANIMATION = 1

The exact numbers depend on the avatar.

You can also adjust the voice, language, blinking, mouth morphing, head turning, facial expressions, and other settings in the config file.

‍

Step 3: Add the Avatar to Your Scene

The easiest method is to place the avatar directly into your environment using Inspector.

GUI Workflow

1. Run AI_Agent_GUI.py.
2. Choose an included environment or drag in a new one.
3. Click Edit in the GUI to open the environment in Inspector.
4. Drag your avatar model into the scene.
5. Position and rotate the avatar using Inspector’s transform tools.
6. Make sure the avatar name ends in .cfg. (right click on the name in Inspector and choose “Rename”) 

‍

‍

For example:

CC2_m001_hipoly_A0_v2.cfg

7. Save the scene in Inspector.
8. Return to the SightLab GUI.
9. Close and restart the script and select your prompt and avatar config from the dropdowns.
10. Run the scene.

‍

‍

By default, the AI Agent will use the first avatar it detects in the scene when:

USE_SCENE_AVATAR = True

Non-GUI Workflow

For non-GUI use, add the avatar to the scene in Inspector, then set the avatar name in Config_Global.py:

SCENE_AVATAR_NAME = 'CC2_m001_hipoly_A0_v2.cfg'

Then run:

AI_Agent.py

If using a Reallusion avatar, or another avatar that does not auto-detect correctly, load it directly by path in the avatar config:

AVATAR_MODEL = 'Resources/avatars/Reallusion_ActorCore_Male.osgb'

Interaction Controls

Use these controls during runtime:

Ask “What are we looking at?”

Agent can interpret the visible scene

If HOLD_KEY_TO_SPEAK is set to False, the agent listens continuously and responds after a pause. This mode works best when the user is wearing headphones.

Enabling Passthrough / Augmented Reality

To use the AI Agent in passthrough AR mode, open Config_Global.py and set:

USE_PASSTHROUGH = True

This is supported on compatible headsets such as Meta Quest Pro, Meta Quest 3, Vive Focus Vision, Pimax and Varjo headsets.

Using the AI Agent Event System

The event system allows the AI to trigger actions during a conversation. For example, the agent can smile, look sad, nod, shake its head, or run a custom callback.

How It Works

The AI includes a hidden event line in its response:

event: smile

The system detects the event, runs the corresponding handler, and removes the event line before showing the text to the user.

Example:

User says:

I just won the lottery!

AI raw response:

event: smile

That's incredible! Congratulations! You must be so excited!

The user only sees:

That's incredible! Congratulations! You must be so excited!

Meanwhile, the avatar smiles.

Enable the Event System

In Config_Global.py:

USE_EVENT_SYSTEM = True

EVENT_KEYWORD = "event:"

Facial expression settings can also be configured:

SMILE_MORPH_ID = 3

SAD_MORPH_ID = 2

EXPRESSION_MORPH_AMOUNT = 0.7

EXPRESSION_DURATION = 1.2

Morph IDs are avatar-specific. To find them:

1. Open the avatar in Inspector.
2. Select the avatar in the scene graph.
3. View the morph IDs in the Properties panel.
4. Update the avatar config with the correct values.

Built-In Events

Creating a Custom Event

You can add your own events in AI_Agent_Avatar.py.

Step 1: Create the Event Handler

def event_my_custom_action():

   """Run a custom AI-triggered action."""

   try:

       print("Custom action executed!")

‍

       # Example actions:

       # avatar.state(MY_ANIMATION)

       # object.setPosition([0, 1.5, 1.5])

       # viz.playSound('sound.wav')

       # viz.clearcolor(viz.RED)

‍

   except Exception as e:

       print(f"Error in custom event: {e}")

Step 2: Register the Event

Add the event to register_default_events():

def register_default_events():

   """Register all built-in event handlers."""

   if USE_EVENT_SYSTEM:

       EVENT_REGISTRY.register("smile", event_smile)

       EVENT_REGISTRY.register("sad", event_sad)

       EVENT_REGISTRY.register("neutral", event_neutral)

       EVENT_REGISTRY.register("my_custom_action", event_my_custom_action)

Step 3: Add the Event to the Prompt

Tell the AI when to use the event:

Available events:

- smile: Use when responding happily or positively.

- sad: Use when responding with sympathy.

- my_custom_action: Use when the user asks you to demonstrate the custom action.

Multi-Agent Interactions

‍

‍

SightLab also includes a multi-agent example:

multi_agent_interaction.py

This allows two AI agents to speak with each other and respond to the user.

Example setup:

from AI_Agents import AIAgent

‍

agents = [

   AIAgent(

       config_path='configs/x_Multi_RocketBoxFemale.py',

       name='Agent1',

       prompt_path='prompts/Susan_Home Office.txt'

   ),

   AIAgent(

       config_path='configs/x_Multi_RocketBoxMale.py',

       name='Agent2',

       prompt_path='prompts/Tom_Home_Office.txt'

   ),

]

Controls:

Each agent can use its own config file, prompt, model, and voice.

Adding the AI Agent to an Existing SightLab Script

To integrate the AI Agent into another SightLab project:

1. Copy these folders/files into your project:

configs

keys

prompts

AI_Agent_Avatar.py

2. Import the AI Agent configuration:

from configs.AI_Agent_Config import *

import AI_Agent_Avatar

3. Add the avatar to SightLab:

avatar = AI_Agent_Avatar.avatar

sightlab.addSceneObject('avatar', avatar, avatar=True)

Or, if using an avatar already placed in the scene:

USE_SCENE_AVATAR = AI_Agent_Avatar.USE_SCENE_AVATAR

This allows the AI Agent to work alongside SightLab’s normal experiment features, including data collection, replay, visualizations, and object tracking.

Troubleshooting Tips

Best Practices

Use the GUI workflow when first setting up an AI Agent, especially when testing models, voices, avatars, and prompts.

Use the non-GUI workflow when embedding the AI Agent into a custom SightLab experiment.

Keep prompt files focused and specific. A good prompt should define the agent’s role, tone, knowledge boundaries, and interaction style.

Use events sparingly and clearly. Add event instructions to the prompt so the AI knows when to smile, nod, take a screenshot, or trigger a custom action.

Test avatar animations and morph IDs in Inspector before running a full experiment.

For offline demos, use Ollama and an offline TTS option such as Kokoro or Piper.

Summary

The SightLab AI Agent workflow provides a flexible way to add intelligent, conversational avatars to VR and XR simulations. Using prompts, avatar configs, LLM settings, speech recognition, text-to-speech, and event callbacks, you can build agents that act as instructors, guides, interviewers, collaborators, or interactive characters.

For most users, the recommended workflow is:

1. Start with AI_Agent_GUI.py.
2. Choose an environment and hardware setup.
3. Create or select a prompt.
4. Create or select an avatar config.
5. Add the avatar in Inspector.
6. Choose the model, voice, and speech options.
7. Run and test the interaction.
8. Add events or custom behaviors as needed.

For more information on SightLab VR Pro see the documentation here https://help.worldviz.com/sightlab/ 

Or click here to request a demo

Notes

Add can create custom voices elevenlabs 

GPT-Realtime voice (with emotional tone)

Scripted responses (multi-agent, how to do with single user)

E-Learning Lab agents

Point and get information

‍

AI Agent

‍

Download Latest Version

This document describes how to use the AI Agent in SightLab, an interactive, intelligent AI agent that can be connected to various large language models like Chat GPT, Claude, Gemini, Offline Ollama models and more. You can customize the agent's personality, use speech recognition, and leverage high-quality text-to-speech models.

Key Features

Quick Start

1. Run AI_Agent_GUI.py to start with the GUI or AI_Agent.py without a GUI
2. Select your environment, hardware, and options in the GUI and press Continue
3. Hold the 'c' key or RH grip button to speak; release to let the agent respond
4. Press 'h' to send a screenshot and ask about what the agent sees (or can just ask "What are we looking at?")

💡 Tip: Run multi_agent_interaction.py for a multi-agent conversation demo.

Setup & Installation

Required Libraries

Ensure you have the required libraries installed using the Vizard Package Manager:

⚠️ Important: Use piper-tts==1.3.0 — do not update to 1.4.0.

Kokoro has very high quality, low latency and runs offline, but requires downloading the models from here (download and extract into the main/root AI Agent folder)

ElevenLabs Additional Setup

• For elevenlabs you may need to Install mpv and add to environment variable path (also requires FFMPEG, but this comes with SightLab):
  
  • Download mpv
  • Unzip and move to C:\Program Files\
  • In Windows search, type powershell, run as administrator
  • Run: setx /M PATH "$($env:PATH);C:\Program Files\mpv-x86_64-20250812-git-211c9cb"
  • Restart Vizard

Offline Models (Ollama)

Ollama comes with the SightLab installer, but if you didn't install it with SightLab, Install Ollama from here, then open a command prompt and type ollama run followed by the model name (e.g., ollama run gemma). See the full model list. The first run may take longer as the model loads.

Note: Requires an active internet connection unless running offline models via Ollama.

API Keys

API keys are needed for online models (OpenAI, Anthropic, Gemini) and optional services (ElevenLabs TTS). For offline models via Ollama, no key is needed.

Setting API Keys (recommended method)

Open a command prompt (type "cmd" in Windows search) and run:

setx OPENAI_API_KEY your-api-key

setx ANTHROPIC_API_KEY your-api-key

setx GEMINI_API_KEY your-api-key

setx ELEVENLABS_API_KEY your-api-key

Restart Vizard after setting keys. With this method your API keys are accessible from any folder.

Obtaining Keys

Customizing Your Agent

Follow these steps to create a custom AI agent with your own personality, avatar, and voice.

Step 1 – Create a Prompt

1. In the prompts folder, make a copy of default.txt and rename it (e.g., MyAgent.txt)
2. Write your role-playing instructions for the agent's personality, context, and behavior
3. You can use an LLM to help create a detailed role-playing prompt

💡 Tip: You can use default.txt as-is if you don't need a custom personality.

‍

‍

Step 2 – Create an Avatar Config

1. In the configs folder, make a copy of Template_Male.py or Template_Female.py and give it a name (e.g., MyAvatar.py)
2. Find an avatar — either from the Avatars tab in the SightLab Dashboard Assets library, or drag in from any source (for auto-detection, use RocketBox or Mixamo avatars (requires adding .cfg to the name in Inspector))
3. Open the avatar in Inspector by double clicking or choosing Open with - Inspector and check the Animations panel (bottom left) — note which animation state number you want for idle and which for talking
4. In your config file, set:
5. TALK_ANIMATION — the state number for the talking animation
6. IDLE_ANIMATION — the state number for the idle animation
7. Optionally change voices, language, and other settings in the config file (see Avatar Configuration Options for the full list)

‍

‍

Step 3 – Add the Avatar to Your Scene

The simplest way to add an avatar is to drag and drop it directly into your environment in Inspector. The AI Agent will automatically detect and use the first avatar found in the scene (controlled by USE_SCENE_AVATAR = True, which is the default).

Using the GUI (simplest)

1. Run AI_Agent_GUI.py
2. Choose an included environment or drag in a new one
3. Click Edit in the GUI to open the environment in Inspector
4. Drag and drop your avatar model into the scene
5. Position the avatar where you want it using the translate and rotate tools (see Adding Avatar Agents for placement tips)
6. If the avatar's name doesn't end in .cfg, right-click it in Inspector and choose Rename to add .cfg to the end

‍

‍

1. Go to File → Save in Inspector, then close it
2. Back in the SightLab GUI, you will see the avatar you added — set the State for the starting idle animation (should match the IDLE_ANIMATION in your config file)
3. Select your prompt and avatar config from the dropdowns, then run the scene

‍

Without the GUI

1. Add the avatar to your environment in Inspector as above (steps 4–7)
2. Set the avatar name in Config_Global.py:
   
   

SCENE_AVATAR_NAME = 'CC2_m001_hipoly_A0_v2.cfg'

3. If running from the SightLab Dashboard, click the config action button to directly open the config file
4. Run AI_Agent.py

Reallusion Avatars

Reallusion avatars need to be loaded by path rather than from the scene. Make a copy of the x_ReallusionMale.py config file, set the path to the avatar directly (this can be done for any avatar that isn't automatically detected as well):

AVATAR_MODEL = 'Resources/avatars/Reallusion_ActorCore_Male.osgb'

Getting Assets

Refer to this page for instructions on obtaining environments and avatars. Drag new assets into the SightLab Dashboard asset library and then can drag into your project.

For more details on avatar placement, animations, and data collection, see Adding Avatar Agents.

Interaction Controls

• Hold 'c' key or RH grip button to speak; release to let the agent respond
• If USE_SPEECH_RECOGNITION is False, press 'c' to type a question
• To stop the conversation, type "q" and click "OK" in text chat
• If HOLD_KEY_TO_SPEAK is False, you only need to speak and when there is a pause of over 0.8 seconds, the agent will respond, then wait for you to speak again (requires wearing headphones)
• Press 'h' to send a screenshot as a prompt and ask questions about what the AI agent is seeing (in the sub-window), or can just ask "What are we looking at?"

Configuration

Most settings (AI model, voice, language, speech model, etc.) are available as dropdown selections in the GUI. For advanced customization, open Config_Global.py directly — or click the config action button from the SightLab Dashboard.

Passthrough / Augmented Reality

To enable AR passthrough mode, set this in Config_Global.py:

USE_PASSTHROUGH = True

This works with Meta Quest Pro, Meta Quest 3, and Varjo headsets.

Avatar Configuration Options

Each avatar uses a config file in the configs folder that controls animations, voice, and behavior. Key settings include TALK_ANIMATION, IDLE_ANIMATION, voice selection, mouth morphing, blinking, and bone/joint references. Make a copy of Template_Male.py or Template_Female.py and adjust as needed — see the comments in the template for all available options. If the avatar doesn't apply things like the head turning to follow you, facial expressions, etc. then you may need to edit these parameters and refer to the avatar in Inspector to see the bone names, etc.

AI Agent Event System

The AI agent supports event-driven interactions, allowing the AI to trigger custom actions during conversations for more expressive and interactive experiences.

The event system allows the AI to execute custom actions (like facial expressions, animations, or any other callback) by including special "event" keywords in its responses. These events are automatically detected, executed, and removed from the text shown to the user.

‍

‍

How It Works

1. Event Detection: The AI includes event: <event_name> on a line in its response
2. Event Execution: The system detects this line, triggers the corresponding handler
3. Text Cleaning: The event line is removed before displaying text to the user
4. Action: The custom action (e.g., facial expression, processing a screenshot, etc.) is performed

Event Configuration

Global Settings (Config_Global.py)

# Event System Settings

USE_EVENT_SYSTEM = True  # Enable/disable event system

EVENT_KEYWORD = "event:"  # Keyword that triggers events

‍

# Morph Target Indices for Facial Expressions

SMILE_MORPH_ID = 3   # Avatar-specific morph index for smile

SAD_MORPH_ID = 2     # Avatar-specific morph index for sad

EXPRESSION_MORPH_AMOUNT = 0.7  # Intensity (0.0 to 1.0)

EXPRESSION_DURATION = 1.2      # Duration in seconds

Avatar-Specific Settings (configs/RocketBoxMale.py, etc.)

Each avatar config can override the default morph indices:

# Event System - Facial Expression Morph Targets

SMILE_MORPH_ID = 3   # RocketBox smile morph

SAD_MORPH_ID = 2     # RocketBox sad morph

EXPRESSION_MORPH_AMOUNT = 0.7

EXPRESSION_DURATION = 1.2

Built-in Events

Facial Expressions

• smile: Makes avatar smile (uses SMILE_MORPH_ID)
• sad: Makes avatar look sad (uses SAD_MORPH_ID)
• neutral: Returns avatar to neutral expression (resets both morphs)

Actions

nod: Nod head in agreement

shake head: shake head side to side in disagreement

Vision

• Capture and process screenshot of scene: When asked things such as "What do you see" the agent can capture and process a screenshot to give an understanding of its surroundings.

Placeholder Events (for future implementation)

• wave: Wave animation placeholder

Finding Morph Target Indices for Your Avatar

Different avatars have different morph target indices. To find the correct indices:

1. Load your avatar in Inspector
2. Click on the avatar name in the scene graph and view the morph IDs on the right side Properties pane under "Morphs"
3. Update your avatar config file with the correct indices

Creating Custom Events

Step 1: Create the Event Handler Function

In AI_Agent_Avatar.py, after the existing event handlers:

def event_my_custom_action():

    """Description of what this event does"""

    try:

        # Your custom code here

        # Examples:

        # - Trigger animations: avatar.state(MY_ANIMATION)

        # - Move objects: object.setPosition([x, y, z])

        # - Play sounds: viz.playSound('sound.wav')

        # - Change lighting: viz.clearcolor(viz.RED)

        print("Custom action executed!")

    except Exception as e:

        print(f"Error in custom event: {e}")

Step 2: Register the Event

In the register_default_events() function:

def register_default_events():

    """Register all built-in event handlers"""

    if USE_EVENT_SYSTEM:

        EVENT_REGISTRY.register("smile", event_smile)

        EVENT_REGISTRY.register("sad", event_sad)

        EVENT_REGISTRY.register("neutral", event_neutral)

        EVENT_REGISTRY.register("my_custom_action", event_my_custom_action)  # Add this

        # ... rest of the events

Step 3: Update AI Prompt

Add your custom event to the prompt file so the AI knows about it:

Available events you can trigger:

- smile: Makes you smile

- sad: Makes you look sad  

- event: screenshot - Take and analyze a screenshot of what you're seeing

- my_custom_action: Description of what it does

Example Prompts

Choose prompts/Event_System_Demo.txt for a comprehensive example prompt that teaches the AI how to use events effectively.

‍

‍

Usage Example

User: "I just won the lottery!"

AI Response (raw):

event: smile

That's incredible! Congratulations! You must be so excited!

What happens:

1. Avatar smiles (smile morph applied)
2. User sees: "That's incredible! Congratulations! You must be so excited!"

Multi Agent Interactions

‍

‍

See the script multi_agent_interaction.py to see how multiple agents can interact and communicate with each other. You can modify the individual agents by calling the AIAgent class and setting parameters such as config_path, name, and prompt_path.

Conversation Modes

The conversation loop can operate in two modes:

• Scripted Dialogue: Pre-written lines that agents speak in sequence
• AI-Generated: Dynamic conversation based on a leading question

User Interaction

You can interrupt and speak to either agent during their conversation:

• Press 'c' to speak to Agent1 (left avatar)
• Press 'v' to speak to Agent2 (right avatar)
• Press 'r' to restart the conversation loop

When you interrupt, the agent will turn toward you and stop any current speech. The conversation loop pauses while you interact.

Note that for multi agent interactions, the AI model and voices are defined in the respective config file for each agent (see the configs folder).

Example

from AI_Agents import AIAgent

‍

agents = [

    AIAgent(config_path='configs/x_Multi_RocketBoxFemale.py', name='Agent1', prompt_path="prompts/Susan_Home Office.txt"),

    AIAgent(config_path='configs/x_Multi_RocketBoxMale.py', name='Agent2', prompt_path="prompts/Tom_Home_Office.txt"),

]

Adding to Existing Scripts

1. Copy the "configs", "keys", and "prompts" folders, as well as AI_Agent_Avatar.py.
2. Import with:
   
   

from configs.AI_Agent_Config import *

import AI_Agent_Avatar

3. Add avatar:
   
   

avatar = AI_Agent_Avatar.avatar

sightlab.addSceneObject('avatar', avatar, avatar=True)

‍

#Or if using the avatar you place in your scene

USE_SCENE_AVATAR = AI_Agent_Avatar.USE_SCENE_AVATAR

5. Add these lines to enable passthrough augmented reality:
   
   

if USE_PASSTHROUGH:

    import openxr

    xr = openxr.getClient()

‍

    if sightlab.getConfig() in ["Meta Quest Pro", "Meta Quest 3"]:

        passthrough = xr.getPassthroughFB()

    elif sightlab.getConfig() == "Varjo":

        passthrough = xr.getPassthroughVarjo()

‍

    viz.clearcolor(viz.BLACK, 0.0)

    if passthrough:

        passthrough.setEnabled(True)

Publishing as an Executable

To publish an application with the AI Agent (or to just publish the standard included template) add this code to the top of the script (before importing SightLab):

import viz

‍

# Filter out problematic packages that cause issues in published EXE

viz.res.addPublishFilter('*google_generativeai*')

viz.res.addPublishFilter('*-nspkg.pth')

viz.res.addPublishFilter('*.pth')

‍

# Add publish directories - ensures correct package versions are bundled

publish_directories = [

    'data',

    viz.res.getVizardPath() + 'bin/lib/site-packages/sightlab_utils/',

    viz.res.getVizardPath() + 'bin/lib/site-packages/deepdiff',

    viz.res.getVizardPath() + 'bin/lib/site-packages/numpy',

    viz.res.getVizardPath() + 'bin/lib/site-packages/pandas',

    viz.res.getVizardPath() + 'bin/lib/site-packages/pydantic',

    viz.res.getVizardPath() + 'bin/lib/site-packages/pydantic_core',

    viz.res.getVizardPath() + 'bin/lib/site-packages/openai',

    viz.res.getVizardPath() + 'bin/lib/site-packages/httpx',

    viz.res.getVizardPath() + 'bin/lib/site-packages/httpcore',

    viz.res.getVizardPath() + 'bin/lib/site-packages/anyio',

    viz.res.getVizardPath() + 'bin/lib/site-packages/sniffio',

    viz.res.getVizardPath() + 'bin/lib/site-packages/annotated_types',

    viz.res.getVizardPath() + 'bin/lib/site-packages/certifi',

    viz.res.getVizardPath() + 'bin/lib/site-packages/speech_recognition',

    viz.res.getVizardPath() + 'bin/lib/site-packages/jiter',

]

‍

for directory in publish_directories:

    viz.res.addPublishDirectory(directory)

‍

import sightlab_utils.sightlab as sl

from sightlab_utils.settings import *

Tips & Troubleshooting

Tips

• Environment Awareness: Press 'h' to take a screenshot that is sent to the agent, or simply ask it "What do you see" or "What are we looking at", etc.
• Prompts: For configuring the agent, use "I am..." for OpenAI and "You are..." for Anthropic without quotes.
• Event Trigger for Speech Button: Modify vizconnect to add an event for speaking button hold. Open settings.py in sightlab_utils/vizconnect_configs, and modify mappings for triggerDown and triggerUp or create new ones if needed. More Info on Vizconnect Events
• If getting an error with Gemini "out of quota", try using a model with more quota or enable billing for much higher limits.

Common Issues

‍