Several of our activities are not yet fully operational within the virtual research laboratories. Although most have been containerized using Docker, they are missing intuitive interfaces that would allow for self-guided operation. Here are a few examples:
The Bremen Ambient Assisted Living Laboratory (BAALL) ontology is a repository of knowledge useful for agents providing assisted living, e.g. knowledge about food and dietary recommendations and restrictions. In this laboratory you will see how to query a fragment of the ontology, and in so doing, also get an intro to ontologies and description logic.
For more information, you can visit the webpage of BAALL to get a better idea of the complete project.
The Metacognition Lab is an interactive online platform designed as a companion to our systematic review of Computational Metacognitive Architectures (CMAs). In this lab, you can explore a range of dynamic visualizations that bring the paper’s analyses to life, offering an in-depth look at how CMAs handle metacognitive experiences. Drawing on data from 36 identified systems, the lab allows you to navigate through the diverse design choices used to model, store, and process introspective information.
Upon entering the lab, you can select specific visualizations—from interactive network graphs depicting theoretical lineages and co-authorship data to grid charts that map symbolic versus sub-symbolic representations. Detailed tooltips and filtering options enable you to tailor your exploration: for instance, you can focus on certain systems, search for specific authors. By hovering over data points, you can retrieve contextual details such as underlying data sources.
Ultimately, The Metacognition Lab fosters a deeper understanding of the review’s findings by encouraging hands-on interaction with the data. Whether you are a researcher, educator, or practitioner, this environment provides a unique gateway to uncover patterns, pose new questions, and glean insights into the rapidly evolving field of metacognitive architectures.
In this laboratory, we investigate the role of common sense in perception because we want to find out what prospective machinery efficiently drives our interpretation of the world to show how to overcome the high uncertainty resulting from the severe temporal, spatial and informational limitations in sensor data as the embodied agent interacts with its environment. Perception in such complex scenes (e.g., safety-critical, dynamic) does not only goes beyond processing sensor data to address classical tasks such as object classification, usually known as the what- and where-object-questions, but also faces the what-, how-, and why-happen-questions (e.g., task execution verification, estimation of physical parameters, quantities and detection of states such as fullness, stability). We generalize the problem of perception by placing events instead of objects at the center of scene understanding, where perception takes place as a loop which consists in predicting on the one hand the effects (anticipation) and on the other hand the causes (explanation) of events.
The Talk2YourKnowledgeBase system from the fortiss research institute research institute takes natural language input from the user, interpreting it through an interface that connects to a knowledge graph database. When a query is made, the system retrieves relevant context using this interface, ensuring that the response is informed by structured and symbolic data.
To extract precise information, a SPARQL query is generated dynamically, leveraging both a large language model and the knowledge graph database itself. This query is then executed against the database, retrieving the most relevant facts and relationships. Once the data is gathered, the language model processes it, generating a well-formed answer that is coherent, context-aware, and human-readable. If the response contains geometric properties, e.g., regarding specific surfaces of an interaction object or an entity of an environmental description, it may be visualized in a web-based graphical interface, augmenting the generated natural language answer. This combination of communication modalities provides an intuitive and interactive way for the user to engage with the retrieved knowledge.
The entire process runs seamlessly within Docker containers, ensuring consistency, scalability, and ease of deployment across different environments.
In the Dual Arm Manipulation & Decision Laboratory, you can observe the capabilities of a dual-arm robot performing table setting tasks under uncertainty. This lab focuses on enhancing the decision-making skills of the robot while it interacts with various objects. The ultimate goal is to develop a system that can intelligently adapt to changing environments and tasks, choosing its next action and which gripper to use autonomously based on several factors of its (spatial) environment.
Currently, the lab showcases two heuristics: The Opportunistic Planning Model (OPM) for action selection, and the Dual-Arm Grasp Action Planner (DAGAP) which decides which gripper to use.
The FAME (Future Action Modelling Engine) Lab stands at the cutting edge of robotics research, operating as a virtual research laboratory under the auspices of the ERC project bearing the same name. This ambitious project is dedicated to exploring how robots can conceptualize and deliberate on future actions to preemptively address and avoid execution failures. A central focus of the FAME Lab’s research is enabling robots to learn manipulation tasks by observing instructional videos. This complex process involves the robot identifying essential motion patterns within these videos, understanding the rationale behind their effectiveness without explicit knowledge of the underlying physics, and adapting these critical motions to its own operational context, which introduces a variety of uncertainties. Overcoming these challenges would mark a significant milestone, granting robots the ability to autonomously learn from instructional content, thereby acquiring a wide range of skills and competencies. A practical application of this research could enable a robot to adeptly cut any fruit, using any tool, for any purpose, in any context, showcasing the potential for robots to achieve a remarkable level of autonomous functionality and versatility.
The TraceBot Lab offers a pioneering platform for conducting research with robotic systems uniquely designed to have a profound understanding of their actions and perceptions, specifically targeting the automation of the membrane-based sterility testing process. At the heart of the TraceBot Lab’s mission is the integration of verifiable actions into robotic manipulations, facilitated by advanced reasoning over sensor-actor trails within a comprehensive traceability framework. This framework capitalizes on digital-twin technology, which serves to replicate the physical world within a virtual environment, enhancing robot motion planners with the ability to autonomously execute self-checking procedures. These innovative procedures aim to create a semantic trace of the robot’s actions, ensuring that every manipulation is not only verifiable but also meets the rigorous standards required in regulated environments like healthcare. Through this approach, the TraceBot Lab is setting new benchmarks for the reliability and accountability of robotic systems in critical sectors.
For more information, you can visit the webpage of TraceBot to get a better idea of the complete project.
In this virtual research lab, we aim to empower robots with the ability to use general action plans that can be parameterised by various sources into a variation of actionable tasks, particularly in everyday manipulations like cutting, pouring or whisking. These plans enable robots to adapt cutting techniques such as slicing, quartering, and peeling to various fruits or to infer parameters for successful pouring and whisking based on the available ingredients and objects, making abstract knowledge practically applicable in robot perception-action loops.
In this virtual research lab, we aim to empower robots with the ability to understand the ‘what’ and ‘how’ of task demonstrations so that robots are enabled to reason about logged memories and differentiate between performed tasks, particularly in everyday manipulations like cutting or pouring. By integrating actionable knowledge graphs, robots are enabled to link contained object and environment information to ation information and map the information to the parameters from their generalized action plans. These plans then enable robots to adapt cutting techniques such as slicing, quartering, and peeling as looged in the task demonstrations, allowing for more specialised task execution.
In the laboratory below, you have the opportunity to select a VR task demonstration to then explore actionable knowledge graph content tailored to specific task domains, including fruit cutting, by utilizing resources like Wikipedia, biology textbooks, nutrition information sources, and instructional websites such as WikiHow. Additionally, you’ll have access to a comprehensive robotic action plan designed specifically for fruit cutting tasks. The integration of actionable knowledge graph information with the task demonstration, such as ”quartering an apple,” can be translated into specific action parameters of the robot. The customized plan can be tested and refined within a simulated environment.
The Interactive Task Learning Lab delves into the forefront of robotics, exploring advanced learning methodologies that empower robots with the ability to not only perform specific tasks, but also to grasp the essence of the tasks themselves. This includes understanding the goals, identifying potential unwanted side effects, and determining suitable metrics for evaluating task performance. Beyond mere execution, this approach necessitates that robots comprehend the physical laws governing their environment, predict the outcomes of their actions, and align their behavior with human expectations. Central to this lab’s research is the dynamic interaction between robots and humans where the human acts as a teacher, imparting knowledge about the conceptual framework of tasks, encompassing both the underlying concepts and their interrelations. Robots are thus challenged to recognize the limits of their knowledge and actively seek assistance, integrating and acting upon advice in a manner that reflects an understanding of its purpose and implications for modifying their actions. This innovative lab not only pushes the boundaries of robot learning but also paves the way for more intuitive and collaborative human-robot interactions.
For more information, you can visit the webpage of Interactive Task Learning to get a better idea on how a robot can learn from different teaching methodologies.
URoboSim is an Unreal Engine 4/5 plugin that allows importing robots to Unreal Engine using the URDF/SDF format and control them using various ROS interfaces. With using URoboSim, it is not only possible to execute real robot plans in simulation, it is also achievable to emulate the actions of a real robot in order to build a continuous belief state of the world.
You can try out an example project here.
This virtual research lab explores reactive motion generation through reasoning. We propose image schema-based reasoning for decision-making within motion controllers. Our reasoner is tightly coupled with the controller, continuously monitoring actions and inferring motion primitives to adapt to dynamic environments. The symbolic theory and the reasoner evaluating it can be exchanged in a plug and play way. By providing real-time feedback, the reasoner enables the controller to make informed decisions and generate appropriate motion responses.
TIAGo robot in the IAI Bremen apartment laboratory.
