Yutarop/turtlesim_agent: Draw with AI in ROS2 TurtleSim

turtlesim_agent is an AI agent that transforms the classic ROS turtlesim simulator into a creative canvas driven by natural language. Powered by LangChain, this AI agent interprets text-based instructions and translates them into visual drawings, turning the simulated turtle into a digital artist. This project explores how large language models can interact with external environments to exhibit creative behavior. Users can describe shapes or drawing intentions in plain English, and the AI agent reasons through the instructions to execute them using turtlesim’s motion commands.

I want you to draw a rainbow composed of 7 semi-circular arcs, each with a different color and a radius ranging from 2.0 cm to 2.7 cm. The colors should follow the traditional rainbow order: violet, indigo, blue, green, yellow, orange, red with the pen’s width of 5. Please offset the starting position of each semi-circle by 0.1 cm to avoid overlap.

Note: This demo was generated using the gemini-2.0-flash model.
Please note that results may vary even when using the same model, due to the non-deterministic nature of language models. Outputs may differ depending on factors like prompt phrasing, timing, or model updates.

• ROS 2 Humble Hawksbill (This project has only been tested with ROS 2 Humble. Compatibility with other ROS 2 distributions is not guaranteed.)
• Python 3.10+
• Other dependencies as listed in requirements.txt

$ cd ~/{ROS_WORKSPACE}/src
$ git clone -b humble-devel https://github.com/Yutarop/turtlesim_agent.git
$ python3 -m pip install -r turtlesim_agent/requirements.txt
$ cd ~/{ROS_WORKSPACE} && colcon build

turtlesim_agent supports multiple language model providers via LangChain. You need to set API keys for the providers you intend to use.

To make your API keys available in your development environment, add them to your shell configuration file (e.g., ~/.bashrc, ~/.zshrc), then reload the file using source.

export OPENAI_API_KEY=your_openai_api_key
export ANTHROPIC_API_KEY=your_anthropic_api_key
export GOOGLE_API_KEY=your_google_api_key
export COHERE_API_KEY=your_cohere_api_key
export MISTRAL_API_KEY=your_mistral_api_key

💡 You only need to set the keys for the providers you plan to use.

If you’re running a local or remote LLM server (e.g., via Ollama), specify the server URL as follows:

export URL="https:your_ollama_server_ip:11434"

To trace and debug agent behavior using LangSmith, set the following environment variables:

Basic Tracing Configuration:

export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export LANGSMITH_TRACING=false

Full Configuration with API Key and Project Name:

export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=your_api_key_here
export LANGSMITH_PROJECT=your_project_name_here

To specify which Large Language Model (LLM) your agent should use, you need to configure the model name in two places:

• turtlesim_node.py
• turtlesim_agent.launch.xml (only if you use ROS 2 launch files)

In both turtlesim_node.py and turtlesim_agent.launch.xml, update the agent_model parameter to match the model you want to use.

💡 The default model is "gemini-2.0-flash". Replace it with your preferred model name (e.g., "gpt-4", "claude-3-opus", etc.).

If you specify a custom model name, make sure it is supported by LangChain. You can verify this by checking or updating the logic inside llm.py.

• If the model is not yet handled, add a corresponding case in llm.py to load the model correctly.
• Refer to LangChain documentation for the latest supported models and providers.

Once you have configured the variables, proceed to build and apply the changes to finalize the setup:

$ cd ~/{ROS_WORKSPACE} && colcon build
$ source ~/.bashrc  # or source ~/.zshrc

turtlesim_agent offers two modes of interaction:

• A CLI-based interface, recommended for debugging and understanding the agent’s internal reasoning.
• A GUI-based chat interface, ideal for intuitive and user-friendly interaction.

$ ros2 run turtlesim turtlesim_node
$ ros2 run turtlesim_agent turtlesim_agent_node

$ ros2 launch turtlesim_agent turtlesim_agent.launch.xml

turtlesim_agent utilizes the tools implemented in the tools/ directory as callable functions that it can invoke during the reasoning process to accomplish user-defined drawing tasks.

tools/
├── all_tools.py # Imports and exports all available tools for the agent
├── math_tools.py # Basic arithmetic and geometric calculations
├── status_tools.py # Queries the current status of the turtle (e.g., position, orientation)
├── motion_tools.py # Controls the movement of the turtle (e.g., forward, rotate)
├── pen_tools.py # Manages pen states (e.g., color, on/off, width)
└── simulation_tools.py # Resets simulation or spawns new turtles

One of the core ideas behind this project is enabling creative expression through tool augmentation. If you’d like to enhance the agent’s capabilities further, feel free to add your own tools to the tools/ directory.

To make new tools available:

1. Create a new *_tools.py file in the tools/ directory.
2. Define your custom functions using LangChain-compatible signatures.
3. Import them in all_tools.py so that the agent can access them.

We welcome any ideas that make TurtleSim Agent more creative and expressive.

Whether it’s new tools, smarter models, better prompts, experimental results, or entirely new use cases — feel free to open an issue or pull request. The results of the experiment will be posted and updated on the wiki. If you manage to draw a shape successfully, please update the wiki!

Keep your files stored safely and securely with the SanDisk 2TB Extreme Portable SSD. With over 69,505 ratings and an impressive 4.6 out of 5 stars, this product has been purchased over 8K+ times in the past month. At only $129.99, this Amazon’s Choice product is a must-have for secure file storage.

Help keep private content private with the included password protection featuring 256-bit AES hardware encryption. Order now for just $129.99 on Amazon!

Help Power Techcratic’s Future – Scan To Support

If Techcratic’s content and insights have helped you, consider giving back by supporting the platform with crypto. Every contribution makes a difference, whether it’s for high-quality content, server maintenance, or future updates. Techcratic is constantly evolving, and your support helps drive that progress.

As a solo operator who wears all the hats, creating content, managing the tech, and running the site, your support allows me to stay focused on delivering valuable resources. Your support keeps everything running smoothly and enables me to continue creating the content you love. I’m deeply grateful for your support, it truly means the world to me! Thank you!