Researchers at Singapore Management University (SMU) are developing a computer vision system designed to analyze and understand the intricate processes involved in cooking through a new dataset called VISOR (Video Segmentations and Object Relations). Led by Assistant Professor Zhu Bin, this initiative aims to enhance how machines interpret actions and interactions in kitchen environments.
What is VISOR?
VISOR is a comprehensive dataset focused on egocentric videos, which capture actions from the perspective of the person performing them. The dataset includes:
- Object Annotations: Over 10 million dense marks across 2.8 million images, labeling 1,477 entities (e.g., knives, flour scoops, and vegetables) and categorizing them into broader macro-categories (e.g., cutlery, appliances).
- Interaction Insights: It analyzes how human hands interact with various kitchen objects and tracks transformations (e.g., flour turning into dough).
- Task Recognition: VISOR introduces tasks like "Where did this come from?" to identify the origin of ingredients in cooking processes.
Annotation Techniques
The dataset employs two main types of annotations:
- Sparse Masks: Applied to key frames, highlighting significant moments.
- Dense Masks: Continuous pixel-level annotations for every frame, filled in using computer vision algorithms. These allow for detailed tracking of object manipulation over time.
Challenges and Solutions
The nature of egocentric videos presents unique challenges, such as occlusions when hands move over objects. VISOR addresses these by:
- Fine-Grained Analysis: The detailed object masks maintain clarity even during transformations, facilitating deeper understanding of actions.
- Enhanced Interaction Modeling: Annotations help researchers study human behavior in natural settings like kitchens.
- Long-Term Object Tracking: Continuous annotations support investigations into sustained interactions and reasoning.
Future Applications
As computer vision technology advances, VISOR could be instrumental in developing assistive tools for individuals with disabilities or the elderly, enabling them to navigate tasks more independently. Additionally, it holds promise for:
- Robotics: Robots capable of understanding complex interactions could assist in cooking, cleaning, and other household tasks.
- Training and Education: The dataset can aid in creating VR or AR applications that provide step-by-step cooking guidance from a first-person perspective.
In summary, VISOR aims to bridge the gap between human action recognition and machine understanding, offering a rich resource for advancing computer vision in dynamic, everyday contexts like cooking.