hotauthor: izadi_s* : Human-Computer Interaction Bibliography Search

[1] FlexCase: Enhancing Mobile Interaction with a Flexible Sensing and Display Cover Interaction Techniques for Mobile Interfaces / Rendl, Christian / Kim, David / Parzer, Patrick / Fanello, Sean / Zirkl, Martin / Scheipl, Gregor / Haller, Michael / Izadi, Shahram Proceedings of the ACM CHI'16 Conference on Human Factors in Computing Systems 2016-05-07 v.1 p.5138-5150

ACM Digital Library Link

Summary: FlexCase is a novel flip cover for smartphones, which brings flexible input and output capabilities to existing mobile phones. It combines an e-paper display with a pressure- and bend-sensitive input sensor to augment the capabilities of a phone. Due to the form factor, FlexCase can be easily transformed into several different configurations, each with different interaction possibilities. Users can use FlexCase to perform a variety of touch, pressure, grip and bend gestures in a natural manner, much like interacting with a sheet of paper. The secondary e-paper display can act as a mechanism for providing user feedback and persisting content from the main display. In this paper, we explore the rich design space of FlexCase and present a number of different interaction techniques. Beyond, we highlight how touch and flex sensing can be combined to support a novel type of gestures, which we call Grip & Bend gestures. We also describe the underlying technology and gesture sensing algorithms. Numerous applications apply the interaction techniques in convincing real-world examples, including enhanced e-paper reading and interaction, a new copy and paste metaphor, high degree of freedom 3D and 2D manipulation, and the ability to transfer content and support input between displays in a natural and flexible manner.

[2] The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces Augmented & Virtual Reality in the Real World / Miksik, Ondrej / Vineet, Vibhav / Lidegaard, Morten / Prasaath, Ram / Nießner, Matthias / Golodetz, Stuart / Hicks, Stephen L. / Pérez, Patrick / Izadi, Shahram / Torr, Philip H. S. Proceedings of the ACM CHI'15 Conference on Human Factors in Computing Systems 2015-04-18 v.1 p.3317-3326

ACM Digital Library Link

Summary: We present an augmented reality system for large scale 3D reconstruction and recognition in outdoor scenes. Unlike existing prior work, which tries to reconstruct scenes using active depth cameras, we use a purely passive stereo setup, allowing for outdoor use and extended sensing range. Our system not only produces a map of the 3D environment in real-time, it also allows the user to draw (or 'paint') with a laser pointer directly onto the reconstruction to segment the model into objects. Given these examples our system then learns to segment other parts of the 3D map during online acquisition. Unlike typical object recognition systems, ours therefore very much places the user 'in the loop' to segment particular objects of interest, rather than learning from predefined databases. The laser pointer additionally helps to 'clean up' the stereo reconstruction and final 3D map, interactively. Using our system, within minutes, a user can capture a full 3D map, segment it into objects of interest, and refine parts of the model during capture. We provide full technical details of our system to aid replication, as well as quantitative evaluation of system components. We demonstrate the possibility of using our system for helping the visually impaired navigate through spaces. Beyond this use, our system can be used for playing large-scale augmented reality games, shared online to augment streetview data, and used for more detailed car and person navigation.

[3] Accurate, Robust, and Flexible Real-time Hand Tracking Mid-Air Gestures and Interaction / Sharp, Toby / Keskin, Cem / Robertson, Duncan / Taylor, Jonathan / Shotton, Jamie / Kim, David / Rhemann, Christoph / Leichter, Ido / Vinnikov, Alon / Wei, Yichen / Freedman, Daniel / Kohli, Pushmeet / Krupka, Eyal / Fitzgibbon, Andrew / Izadi, Shahram Proceedings of the ACM CHI'15 Conference on Human Factors in Computing Systems 2015-04-18 v.1 p.3633-3642

ACM Digital Library Link

Summary: We present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. Most uniquely, our tracker is highly flexible, dramatically improving upon previous approaches which have focused on front-facing close-range scenarios. This flexibility opens up new possibilities for human-computer interaction with examples including tracking at distances from tens of centimeters through to several meters (for controlling the TV at a distance), supporting tracking using a moving depth camera (for mobile scenarios), and arbitrary camera placements (for VR headsets). These features are achieved through a new pipeline that combines a multi-layered discriminative reinitialization strategy for per-frame pose estimation, followed by a generative model-fitting stage. We provide extensive technical details and a detailed qualitative and quantitative analysis.

[4] FlexSense: a transparent self-sensing deformable surface Novel hardware I / Rendl, Christian / Kim, David / Fanello, Sean / Parzer, Patrick / Rhemann, Christoph / Taylor, Jonathan / Zirkl, Martin / Scheipl, Gregor / Rothländer, Thomas / Haller, Michael / Izadi, Shahram Proceedings of the 2014 ACM Symposium on User Interface Software and Technology 2014-10-05 v.1 p.129-138

ACM Digital Library Link

Summary: We present FlexSense, a new thin-film, transparent sensing surface based on printed piezoelectric sensors, which can reconstruct complex deformations without the need for any external sensing, such as cameras. FlexSense provides a fully self-contained setup which improves mobility and is not affected from occlusions. Using only a sparse set of sensors, printed on the periphery of the surface substrate, we devise two new algorithms to fully reconstruct the complex deformations of the sheet, using only these sparse sensor measurements. An evaluation shows that both proposed algorithms are capable of reconstructing complex deformations accurately. We demonstrate how FlexSense can be used for a variety of 2.5D interactions, including as a transparent cover for tablets where bending can be performed alongside touch to enable magic lens style effects, layered input, and mode switching, as well as the ability to use our device as a high degree-of-freedom input controller for gaming and beyond.

[5] In-air gestures around unmodified mobile devices Augmented reality I / Song, Jie / Sörös, Gábor / Pece, Fabrizio / Fanello, Sean Ryan / Izadi, Shahram / Keskin, Cem / Hilliges, Otmar Proceedings of the 2014 ACM Symposium on User Interface Software and Technology 2014-10-05 v.1 p.319-329

ACM Digital Library Link

Summary: We present a novel machine learning based algorithm extending the interaction space around mobile devices. The technique uses only the RGB camera now commonplace on off-the-shelf mobile devices. Our algorithm robustly recognizes a wide range of in-air gestures, supporting user variation, and varying lighting conditions. We demonstrate that our algorithm runs in real-time on unmodified mobile devices, including resource-constrained smartphones and smartwatches. Our goal is not to replace the touchscreen as primary input device, but rather to augment and enrich the existing interaction vocabulary using gestures. While touch input works well for many scenarios, we demonstrate numerous interaction tasks such as mode switches, application and task management, menu selection and certain types of navigation, where such input can be either complemented or better served by in-air gestures. This removes screen real-estate issues on small touchscreens, and allows input to be expanded to the 3D space around the device. We present results for recognition accuracy (93% test and 98% train), impact of memory footprint and other model parameters. Finally, we report results from preliminary user evaluations, discuss advantages and limitations and conclude with directions for future work.

[6] 3D-board: a whole-body remote collaborative whiteboard Collaboration / Zillner, Jakob / Rhemann, Christoph / Izadi, Shahram / Haller, Michael Proceedings of the 2014 ACM Symposium on User Interface Software and Technology 2014-10-05 v.1 p.471-479

ACM Digital Library Link

Summary: This paper presents 3D-Board, a digital whiteboard capable of capturing life-sized virtual embodiments of geographically distributed users. When using large-scale screens for remote collaboration, awareness for the distributed users' gestures and actions is of particular importance. Our work adds to the literature on remote collaborative workspaces, it facilitates intuitive remote collaboration on large scale interactive whiteboards by preserving awareness of the full-body pose and gestures of the remote collaborator. By blending the front-facing 3D embodiment of a remote collaborator with the shared workspace, an illusion is created as if the observer was looking through the transparent whiteboard into the remote user's room. The system was tested and verified in a usability assessment, showing that 3D-Board significantly improves the effectiveness of remote collaboration on a large interactive surface.

[7] RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces On and above the surface / Kim, David / Izadi, Shahram / Dostal, Jakub / Rhemann, Christoph / Keskin, Cem / Zach, Christopher / Shotton, Jamie / Large, Timothy / Bathiche, Steven / Nießner, Matthias / Butler, D. Alex / Fanello, Sean / Pradeep, Vivek Proceedings of ACM CHI 2014 Conference on Human Factors in Computing Systems 2014-04-26 v.1 p.1377-1386

ACM Digital Library Link

Summary: We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces. Our setup is simple, cheap, and easily reproducible, comprising of two infrared cameras, diffuse infrared LEDs, and any off-the-shelf retro-reflective material. The retro-reflector aids image segmentation, creating a strong contrast between the surface and any object in proximity. A new highly efficient stereo matching algorithm precisely estimates the 3D contours of interacting objects and the retro-reflective surfaces. A novel pipeline enables 3D finger, hand and object tracking, as well as gesture recognition, purely using these 3D contours. We demonstrate high-precision sensing, allowing robust disambiguation between a finger or stylus touching, pressing or interacting above the surface. This allows many interactive scenarios that seamlessly mix together freehand 3D interactions with touch, pressure and stylus input. As shown, these rich modalities of input are enabled on and above any retro-reflective surface, including custom "physical widgets" fabricated by users. We compare our system with Kinect and Leap Motion, and conclude with limitations and future work.

[8] Type-hover-swipe in 96 bytes: a motion sensing mechanical keyboard Novel keyboards / Taylor, Stuart / Keskin, Cem / Hilliges, Otmar / Izadi, Shahram / Helmes, John Proceedings of ACM CHI 2014 Conference on Human Factors in Computing Systems 2014-04-26 v.1 p.1695-1704

ACM Digital Library Link

Summary: We present a new type of augmented mechanical keyboard, capable of sensing rich and expressive motion gestures performed both on and directly above the device. Our hardware comprises of low-resolution matrix of infrared (IR) proximity sensors interspersed between the keys of a regular mechanical keyboard. This results in coarse but high frame-rate motion data. We extend a machine learning algorithm, traditionally used for static classification only, to robustly support dynamic, temporal gestures. We propose the use of motion signatures a technique that utilizes pairs of motion history images and a random forest based classifier to robustly recognize a large set of motion gestures on and directly above the keyboard. Our technique achieves a mean per-frame classification accuracy of 75.6% in leave-one-subject-out and 89.9% in half-test/half-training cross-validation. We detail our hardware and gesture recognition algorithm, provide performance and accuracy numbers, and demonstrate a large set of gestures designed to be performed with our device. We conclude with qualitative feedback from users, discussion of limitations and areas for future work.

[9] A low-cost transparent electric field sensor for 3d interaction on mobile devices Tangible interactions and technologies / Goc, Mathieu Le / Taylor, Stuart / Izadi, Shahram / Keskin, Cem Proceedings of ACM CHI 2014 Conference on Human Factors in Computing Systems 2014-04-26 v.1 p.3167-3170

ACM Digital Library Link

Summary: We contribute a thin, transparent, and low-cost design for electric field sensing, allowing for 3D finger and hand tracking and gestures on mobile devices. Our approach requires no direct instrumentation of the hand or body, and is non-optical, allowing for a compact form-factor that is resilient to ambient illumination. Our simple driver electronics are based on an off-the-shelf chip that removes the need for building custom analog electronics. We describe the design of our transparent electrode array, and present a machine learning algorithm for mapping from signal measurements at the receivers to 3D positions. We demonstrate non-contact motion gestures, and precise 3D hand and finger localization. We conclude by discussing limitations and future work.

[10] Quick and dirty: streamlined 3D scanning in archaeology Multiple dimensions and displays / Knibbe, Jarrod / O'Hara, Kenton P. / Chrysanthi, Angeliki / Marshall, Mark T. / Bennett, Peter D. / Earl, Graeme / Izadi, Shahram / Fraser, Mike Proceedings of ACM CSCW 2014 Conference on Computer-Supported Cooperative Work and Social Computing 2014-02-15 v.1 p.1366-1376

ACM Digital Library Link

Summary: Capturing data is a key part of archaeological practice, whether for preserving records or to aid interpretation. But the technologies used are complex and expensive, resulting in time-consuming processes associated with their use. These processes force a separation between ongoing interpretive work and capture. Through two field studies we elicit more detail as to what is important about this interpretive work and what might be gained through a closer integration of capture technology with these practices. Drawing on these insights, we go on to present a novel, portable, wireless 3D modeling system that emphasizes "quick and dirty" capture. We discuss its design rational in relation to our field observations and evaluate this rationale further by giving the system to archaeological experts to explore in a variety of settings. While our device compromises on the resolution of traditional 3D scanners, its support of interpretation through emphasis on real-time capture, review and manipulability suggests it could be a valuable tool for the future of archaeology.

[11] Video collections in panoramic contexts Visualization & video / Tompkin, James / Pece, Fabrizio / Shah, Rajvi / Izadi, Shahram / Kautz, Jan / Theobalt, Christian Proceedings of the 2013 ACM Symposium on User Interface Software and Technology 2013-10-08 v.1 p.131-140

ACM Digital Library Link

Summary: Video collections of places show contrasts and changes in our world, but current interfaces to video collections make it hard for users to explore these changes. Recent state-of-the-art interfaces attempt to solve this problem for 'outside->in' collections, but cannot connect 'inside->out' collections of the same place which do not visually overlap. We extend the focus+context paradigm to create a video-collections+context interface by embedding videos into a panorama. We build a spatio-temporal index and tools for fast exploration of the space and time of the video collection. We demonstrate the flexibility of our representation with interfaces for desktop and mobile flat displays, and for a spherical display with joypad and tablet controllers. We study with users the effect of our video-collection+context system to spatio-temporal localization tasks, and find significant improvements to accuracy and completion time in visual search tasks compared to existing systems. We measure the usability of our interface with System Usability Scale (SUS) and task-specific questionnaires, and find our system scores higher.

[12] Demo hour Demo hour / Takahashi, Yoichi / Matoba, Yasushi / Koike, Hideki / Cassinelli, Alvaro / Angesleva, Jussi / Watanabe, Yoshihiro / Frasca, Gonzalo / Ishikawa, Masatoshi / Hirsch, Matthew / Izadi, Shahram / Holtzman, Henry / Raskar, Ramesh / Bonnard, Quentin / Legge, Amanda / Geiduschek, Anna / Kaplan, Frédéric / Dillenbourg, Pierre interactions 2013-05 v.20 n.3 p.8-9

ACM Digital Library Link

[13] 8D: interacting with a relightable glasses-free 3D display Papers: performing interaction / Hirsch, Matthew / Izadi, Shahram / Holtzman, Henry / Raskar, Ramesh Proceedings of ACM CHI 2013 Conference on Human Factors in Computing Systems 2013-04-27 v.1 p.2209-2212

ACM Digital Library Link

Summary: We present an 8-dimensional (8D) display that allows glasses-free viewing of 3D imagery, whist capturing and reacting to incident environmental and user controlled light sources. We demonstrate two interactive possibilities enabled by our lens-array-based hardware prototype, and realtime GPU-accelerated software pipeline. Additionally, we describe a path to deploying such displays in the future, using current Sensor-in-Pixel (SIP) LCD panels, which physically collocate sensing and display elements.

[14] 8D display: a relightable glasses-free 3D display Demo session / Hirsch, Matthew / Izadi, Shahram / Holtzman, Henry / Raskar, Ramesh Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces 2012-11-11 p.319-322

ACM Digital Library Link

Summary: magine a display that behaves like a window. Glancing through it, viewers perceive a virtual 3D scene with correct parallax, without the need to wear glasses or track the user. Light that passes through the display correctly illuminates the virtual scene. While researchers have considered such displays, or prototyped subsets of these capabilities, we contribute a new, interactive, relightable, glasses-free 3D display. By simultaneously capturing a 4D light field, and displaying a 4D light field, we are able to realistically modulate the incident light on rendered content. We present our optical design, and GPU pipeline. Beyond mimicking the physical appearance of objects under natural lighting, an 8D display can create arbitrary directional illumination patterns and record their interaction with physical objects. Our hardware points the way towards novel 3D interfaces, in which users interact with digital content using light widgets, physical objects, and gesture.

[15] Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor Hands & fingers / Kim, David / Hilliges, Otmar / Izadi, Shahram / Butler, Alex D. / Chen, Jiawen / Oikonomidis, Iason / Olivier, Patrick Proceedings of the 2012 ACM Symposium on User Interface Software and Technology 2012-10-07 v.1 p.167-176

ACM Digital Library Link

Summary: Digits is a wrist-worn sensor that recovers the full 3D pose of the user's hand. This enables a variety of freehand interactions on the move. The system targets mobile settings, and is specifically designed to be low-power and easily reproducible using only off-the-shelf hardware. The electronics are self-contained on the user's wrist, but optically image the entirety of the user's hand. This data is processed using a new pipeline that robustly samples key parts of the hand, such as the tips and lower regions of each finger. These sparse samples are fed into new kinematic models that leverage the biomechanical constraints of the hand to recover the 3D pose of the user's hand. The proposed system works without the need for full instrumentation of the hand (for example using data gloves), additional sensors in the environment, or depth cameras which are currently prohibitive for mobile scenarios due to power and form-factor considerations. We demonstrate the utility of Digits for a variety of application scenarios, including 3D spatial interaction with mobile devices, eyes-free interaction on-the-move, and gaming. We conclude with a quantitative and qualitative evaluation of our system, and discussion of strengths, limitations and future work.

[16] Steerable augmented reality with the beamatron Augmented reality / Wilson, Andrew / Benko, Hrvoje / Izadi, Shahram / Hilliges, Otmar Proceedings of the 2012 ACM Symposium on User Interface Software and Technology 2012-10-07 v.1 p.413-422

ACM Digital Library Link

Summary: Steerable displays use a motorized platform to orient a projector to display graphics at any point in the room. Often a camera is included to recognize markers and other objects, as well as user gestures in the display volume. Such systems can be used to superimpose graphics onto the real world, and so are useful in a number of augmented reality and ubiquitous computing scenarios. We contribute the Beamatron, which advances steerable displays by drawing on recent progress in depth camera-based interactions. The Beamatron consists of a computer-controlled pan and tilt platform on which is mounted a projector and Microsoft Kinect sensor. While much previous work with steerable displays deals primarily with projecting corrected graphics onto a discrete set of static planes, we describe computational techniques that enable reasoning in 3D using live depth data. We show two example applications that are enabled by the unique capabilities of the Beamatron: an augmented reality game in which a player can drive a virtual toy car around a room, and a ubiquitous computing demo that uses speech and gesture to move projected graphics throughout the room.

[17] KinÊtre: animating the world with the human body Augmented reality / Chen, Jiawen / Izadi, Shahram / Fitzgibbon, Andrew Proceedings of the 2012 ACM Symposium on User Interface Software and Technology 2012-10-07 v.1 p.435-444

ACM Digital Library Link

Summary: KinÊtre allows novice users to scan arbitrary physical objects and bring them to life in seconds. The fully interactive system allows diverse static meshes to be animated using the entire human body. Traditionally, the process of mesh animation is laborious and requires domain expertise, with rigging specified manually by an artist when designing the character. KinÊtre makes creating animations a more playful activity, conducted by novice users interactively "at runtime". This paper describes the KinÊtre system in full, highlighting key technical contributions and demonstrating many examples of users animating meshes of varying shapes and sizes. These include non-humanoid meshes and incomplete surfaces produced by 3D scanning -- two challenging scenarios for existing mesh animation systems. Rather than targeting professional CG animators, KinÊtre is intended to bring mesh animation to a new audience of novice users. We demonstrate potential uses of our system for interactive storytelling and new forms of physical gaming.

[18] Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces HCI / Molyneaux, David / Izadi, Shahram / Kim, David / Hilliges, Otmar / Hodges, Steve / Cao, Xiang / Butler, Alex / Gellersen, Hans Proceedings of Pervasive 2012: International Conference on Pervasive Computing 2012-06-18 p.197-215

Keywords: Handheld projection; geometry and spatial awareness; interaction

Link to Digital Content at Springer

Summary: This paper presents two novel handheld projector systems for indoor pervasive computing spaces. These projection-based devices are "aware" of their environment in ways not demonstrated previously. They offer both spatial awareness, where the system infers location and orientation of the device in 3D space, and geometry awareness, where the system constructs the 3D structure of the world around it, which can encompass the user as well as other physical objects, such as furniture and walls. Previous work in this area has predominantly focused on infrastructure-based spatial-aware handheld projection and interaction. Our prototypes offer greater levels of environment awareness, but achieve this using two opposing approaches; the first infrastructure-based and the other infrastructure-less sensing. We highlight a series of interactions including direct touch, as well as in-air gestures, which leverage the shadow of the user for interaction. We describe the technical challenges in realizing these novel systems; and compare them directly by quantifying their location tracking and input sensing capabilities.

[19] The role of physical controllers in motion video gaming Game design / Freeman, Dustin / Hilliges, Otmar / Sellen, Abigail / O'Hara, Kenton / Izadi, Shahram / Wood, Kenneth Proceedings of DIS'12: Designing Interactive Systems 2012-06-11 p.701-710

ACM Digital Library Link

Summary: Systems that detect the unaugmented human body allow players to interact without using a physical controller. But how is interaction altered by the absence of a physical input device? What is the impact on game performance, on a player's expectation of their ability to control the game, and on their game experience? In this study, we investigate these issues in the context of a table tennis video game. The results show that the impact of holding a physical controller, or indeed of the fidelity of that controller, does not appear in simple measures of performance. Rather, the difference between controllers is a function of the responsiveness of the game being controlled, as well as other factors to do with expectations, real world game experience and social context.

[20] At home with surface computing? Touch in context / Kirk, David / Izadi, Shahram / Hilliges, Otmar / Banks, Richard / Taylor, Stuart / Sellen, Abigail Proceedings of ACM CHI 2012 Conference on Human Factors in Computing Systems 2012-05-05 v.1 p.159-168

ACM Digital Library Link

Summary: This paper describes a field study of an interactive surface deployed in three family homes. The tabletop technology provides a central place where digital content, such as photos, can be easily archived, managed and viewed. The tabletop affords multi-touch input, allowing digital content to be sorted, triaged and interacted with using one or two-handed interactions. A physics-based simulation adds dynamics to digital content, providing users with rich ways of interacting that borrows from the real-world. The field study is one of the first of a surface computer within a domestic environment. Our goal is to uncover people's inter-actions, appropriations, perceptions and experiences with such technologies, exploring the potential barriers to use. Given these devices provide such a revolutionary shift in interaction, will people be able to engage with them in everyday life in the ways we intend? In answering this question, we hope to deepen our understanding of the design of such systems for home and consumer domains.

[21] Shake'n'sense: reducing interference for overlapping structured light depth cameras Sensory interaction modalities / Butler, D. Alex / Izadi, Shahram / Hilliges, Otmar / Molyneaux, David / Hodges, Steve / Kim, David Proceedings of ACM CHI 2012 Conference on Human Factors in Computing Systems 2012-05-05 v.1 p.1933-1936

ACM Digital Library Link

Summary: We present a novel yet simple technique that mitigates the interference caused when multiple structured light depth cameras point at the same part of a scene. The technique is particularly useful for Kinect, where the structured light source is not modulated. Our technique requires only mechanical augmentation of the Kinect, without any need to modify the internal electronics, firmware or associated host software. It is therefore simple to replicate. We show qualitative and quantitative results highlighting the improvements made to interfering Kinect depth signals. The camera frame rate is not compromised, which is a problem in approaches that modulate the structured light source. Our technique is non-destructive and does not impact depth values or geometry. We discuss uses for our technique, in particular within instrumented rooms that require simultaneous use of multiple overlapping fixed Kinect cameras to support whole room interactions.

[22] HoloDesk: direct 3d interactions with a situated see-through display Morphing & tracking & stacking: 3D interaction / Hilliges, Otmar / Kim, David / Izadi, Shahram / Weiss, Malte / Wilson, Andrew Proceedings of ACM CHI 2012 Conference on Human Factors in Computing Systems 2012-05-05 v.1 p.2421-2430

ACM Digital Library Link

Summary: HoloDesk is an interactive system combining an optical see through display and Kinect camera to create the illusion that users are directly interacting with 3D graphics. A virtual image of a 3D scene is rendered through a half silvered mirror and spatially aligned with the real-world for the viewer. Users easily reach into an interaction volume displaying the virtual image. This allows the user to literally get their hands into the virtual display and to directly interact with an spatially aligned 3D virtual world, without the need for any specialized head-worn hardware or input device. We introduce a new technique for interpreting raw Kinect data to approximate and track rigid (e.g., books, cups) and non-rigid (e.g., hands, paper) physical objects and support a variety of physics-inspired interactions between virtual and real. In particular the algorithm models natural human grasping of virtual objects with more fidelity than previously demonstrated. A qualitative study highlights rich emergent 3D interactions, using hands and real-world objects. The implementation of HoloDesk is described in full, and example application scenarios explored. Finally, HoloDesk is quantitatively evaluated in a 3D target acquisition task, comparing the system with indirect and glasses-based variants.

[23] Enhancing naturalness of pen-and-tablet drawing through context sensing Graspable interfaces / Sun, Minghui / Cao, Xiang / Song, Hyunyoung / Izadi, Shahram / Benko, Hrvoje / Guimbretiere, Francois / Ren, Xiangshi / Hinckley, Ken Proceedings of the 2011 ACM International Conference on Interactive Tabletops and Surfaces 2011-11-13 p.83-86

ACM Digital Library Link

Summary: Among artists and designers, the pen-and-tablet combination is widely used for creating digital drawings, as digital pens outperform other input devices in replicating the experience of physical drawing tools. In this paper, we explore how contextual information such as the relationship between the hand, the pen, and the tablet can be leveraged in the digital drawing experience to further enhance its naturalness. By embedding sensors in the pen and the tablet to sense and interpret these contexts, we demonstrate how several physical drawing practices can be reflected and assisted in digital interaction scenarios.

[24] Portico: tangible interaction on and around a tablet Tangible / Avrahami, Daniel / Wobbrock, Jacob O. / Izadi, Shahram Proceedings of the 201 ACM Symposium on User Interface Software and Technology1 2011-10-16 v.1 p.347-356

ACM Digital Library Link

Summary: We present Portico, a portable system for enabling tangible interaction on and around tablet computers. Two cameras on small foldable arms are positioned above the display to recognize a variety of physical objects placed on or around the tablet. These cameras have a larger field-of-view than the screen, allowing Portico to extend interaction significantly beyond the tablet itself. Our prototype, which uses a 12" tablet, delivers an interaction space six times the size of the tablet screen. Portico thus allows tablets to extend both their sensing capabilities and interaction space without sacrificing portability. We describe the design of our system and present a number of applications that demonstrate Portico's unique capability to track objects. We focus on a number of fun applications that demonstrate how such a device can be used as a low-cost way to create personal surface computing experiences. Finally, we discuss the challenges in supporting tangible interaction beyond the screen and describe possible mechanisms for overcoming them.

[25] KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera 3D / Izadi, Shahram / Kim, David / Hilliges, Otmar / Molyneaux, David / Newcombe, Richard / Kohli, Pushmeet / Shotton, Jamie / Hodges, Steve / Freeman, Dustin / Davison, Andrew / Fitzgibbon, Andrew Proceedings of the 201 ACM Symposium on User Interface Software and Technology1 2011-10-16 v.1 p.559-568

ACM Digital Library Link

Summary: KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.