Tutorial 1: Sketch Understanding

Exploring Spatial Cognition through Sketch Understanding

by Kenneth D. Forbus, Qualitative Reasoning Group, Northwestern University, USA.

Abstract

Sketching is a natural way for people to communicate spatial information with each other. This makes it a useful method for exploring spatial cognition: People use their spatial knowledge in understanding sketches, and software that communicates with people via sketching will need similar capabilities. This half-day tutorial is intended for cognitive science researchers, including AI researchers, psychologists, and learning scientists who want to use sketching in their research. It will focus on CogSketch, a publicly available sketch understanding system which is being developed to support cognitive science research and serve as a platform for sketch-based educational software. CogSketch incorporates visual processing of digital ink, qualitative spatial representations, analogical matching over integrated spatial and conceptual representations, and a large open-source knowledge base, derived from OpenCyc. CogSketch is being developed by the NSF-funded Spatial Intelligence and Learning Center (SILC), with feedback from the research and education communities.

Contents

The following topics will be covered in the tutorial:

• Basics of open-domain sketch understanding. Digital ink, conceptual labeling.
• Visual processing in CogSketch. Ink and Voronoi processing. Qualitative spatial representations and grouping operations.Decomposition and matching of shapes.
• Analogical processing in CogSketch. Basics of structure-mapping theory, structure-mapping engine (SME). Spatial considerations in mapping. Similarity-based retrieval.
• Examples of experiments already conducted using CogSketch: Raven’s Progressive Matrices, oddity tasks, learning spatial prepositions, learning to recognize complex objects.
• Incorporating conceptual knowledge in sketch understanding. Qualitative mechanics, applications to engineering design and geosciences education.
• Support for psychological experiments. Copy-and-paste of stimuli from PowerPoint. Accessing timing information. Gathering sketch data. Automatic data interpretation and scoring.
• Support for computational experiments. Exporting knowledge and ink data. Overview of CogSketch API for using its spatial reasoning facilities from other software. Extending the knowledge base.

Audience

The intended audience includes AI researchers, psychologists, and other cognitive science researchers interested in how they might use sketching to explore spatial cognition. It also includes learning scientists and AI researchers who are interested in using sketching as an interface modality in their systems. Attendees of the tutorial will gain an understanding of the state of the art in open-domain sketch understanding, and how it might be used to facilitate their research. The CogSketch team and SILC are actively seeking collaborators, whose feedback will play a guiding role in the development of CogSketch. Attendees are welcome to download CogSketch on their laptops in advance of the tutorial and follow along with some of the demonstrations. Handouts will be provided, and sample sketches and code developed for the tutorial will be available on the SILC web site.

Instructor

Kenneth Forbus is the Walter P. Murphy Professor of Computer Science and Professor of Education at Northwestern University. His research interests include qualitative reasoning, analogy and similarity, sketch understanding, spatial reasoning, cognitive simulation, reasoning system design, articulate educational software, and the use of AI in computer gaming. He received his degrees from MIT (Ph.D. in 1984). He is a Fellow of the American Association for Artificial Intelligence, the Cognitive Science Society, and the Association for Computing Machinery. He serves on the editorial boards of Cognitive Science, the AAAI Press, and on the Advisory Board of the Journal of Game Development. He is leading the CogSketch effort in SILC, creating a computational model of spatial reasoning and learning that is a cognitive science research instrument and will be a platform for sketch-based educational software.

References

CogSketch can be downloaded at http://www.spatialintelligence.org/projects/cogsketch_index.html.

Research discussed will include:

• Forbus, K., & Usher, J. (2002). Sketching for knowledge capture: A progress report. Proceedings of IUI'02, San Francisco, California, January 13-16, 2002.
• Tomai, E., Lovett, A., Forbus, K., & Usher, J. (2005). A Structure Mapping Model for Solving Geometric Analogy Problems. Proceedings of the 27th Annual Conference of the Cognitive Science Society, Stressa, Italy, 2190-2195.
• Lockwood, K., Forbus, K., Halstead, D. & Usher, J. (2006). Automatic Categorization of Spatial Prepositions. Proceedings of the 28th Annual Conference of the Cognitive Science Society. Vancouver, Canada.
• Lovett, A. Forbus, K., & Usher, J. (2007). Analogy with qualitative spatial representations can simulate solving Raven's Progressive Matrices. Proceedings of the 29th Annual Conference of the Cognitive Science Society. Nashville, TN.
• Lovett, A., Dehghani, M. and Forbus, K. (2007). Incremental Learning of Perceptual Categories for Open-Domain Sketch Recognition. International Joint Conferences on Artificial Intelligence, 447-452.