HCI Bibliography Home | HCI Conferences | UIST Archive | Detailed Records | RefWorks | EndNote | Hide Abstracts
UIST Tables of Contents: 030405060708091011-111-212-112-213-113-214-114-215-115-2

Proceedings of the 2012 ACM Symposium on User Interface Software and Technology

Fullname:Proceedings of the 25th annual ACM symposium on User interface software and technology
Editors:Rob Miller; Hrvoje Benko; Celine Latulipe
Location:Cambridge, Massachusetts
Dates:2012-Oct-07 to 2012-Oct-10
Standard No:ISBN: 978-1-4503-1580-7; ACM DL: Table of Contents; hcibib: UIST12-1
Links:Conference Website
Summary:UIST (ACM Symposium on User Interface Software and Technology) is the premier forum for innovations in the software and technology of human-computer interfaces. Sponsored by ACM's special interest groups on computer-human interaction (SIGCHI) and computer graphics (SIGGRAPH), UIST brings together researchers and practitioners from diverse areas that include traditional graphical & web user interfaces, tangible & ubiquitous computing, virtual & augmented reality, multimedia, new input & output devices, and CSCW. The intimate size, single track, and comfortable surroundings make this symposium an ideal opportunity to exchange research results and implementation experiences.
  1. UIST 2012-10-07 Volume 1
    1. Keynote address
    2. Groups & crowds
    3. Tutorials & learning
    4. Hands & fingers
    5. Toolkits
    6. Interactions 1
    7. Pen
    8. Interactions II
    9. Augmented reality
    10. Multi-touch
    11. Tactile & grip
    12. Fabrication & hardware

UIST 2012-10-07 Volume 1

Keynote address

What art can tell us about the brain BIBAFull-Text 1-2
  Margaret Livingstone
Artists have been doing experiments on vision longer than neurobiologists. Some major works of art have provided insights as to how we see; some of these insights are so fundamental that they can be understood in terms of the underlying neurobiology. For example, artists have long realized that color and luminance can play independent roles in visual perception. Picasso said, "Colors are only symbols. Reality is to be found in luminance alone." This observation has a parallel in the functional subdivision of our visual systems, where color and luminance are processed by the newer, primate-specific What system, and the older, colorblind, Where (or How) system. Many techniques developed over the centuries by artists can be understood in terms of the parallel organization of our visual systems. I will explore how the segregation of color and luminance processing are the basis for why some Impressionist paintings seem to shimmer, why some op art paintings seem to move, some principles of Matisse's use of color, and how the Impressionists painted "air". Central and peripheral vision are distinct, and I will show how the differences in resolution across our visual field make the Mona Lisa's smile elusive, and produce a dynamic illusion in Pointillist paintings, Chuck Close paintings, and photomosaics. I will explore how artists have intuited important features about how our brains extract relevant information about faces and objects, and I will discuss why learning disabilities may be associated with artistic talent.

Groups & crowds

RevMiner: an extractive interface for navigating reviews on a smartphone BIBAFull-Text 3-12
  Jeff Huang; Oren Etzioni; Luke Zettlemoyer; Kevin Clark; Christian Lee
Smartphones are convenient, but their small screens make searching, clicking, and reading awkward. Thus, perusing product reviews on a smartphone is difficult. In response, we introduce RevMiner -- a novel smartphone interface that utilizes Natural Language Processing techniques to analyze and navigate reviews. RevMiner was run over 300K Yelp restaurant reviews extracting attribute-value pairs, where attributes represent restaurant attributes such as sushi and service, and values represent opinions about the attributes such as fresh or fast. These pairs were aggregated and used to: 1) answer queries such as "cheap Indian food", 2) concisely present information about each restaurant, and 3) identify similar restaurants. Our user studies demonstrate that on a smartphone, participants preferred RevMiner's interface to tag clouds and color bars, and that they preferred RevMiner's results to Yelp's, particularly for conjunctive queries (e.g., "great food and huge portions"). Demonstrations of RevMiner are available at revminer.com.
Cross-device interaction via micro-mobility and f-formations BIBAFull-Text 13-22
  Nicolai Marquardt; Ken Hinckley; Saul Greenberg
GroupTogether is a system that explores cross-device interaction using two sociological constructs. First, F-formations concern the distance and relative body orientation among multiple users, which indicate when and how people position themselves as a group. Second, micro-mobility describes how people orient and tilt devices towards one another to promote fine-grained sharing during co-present collaboration. We sense these constructs using: (a) a pair of overhead Kinect depth cameras to sense small groups of people, (b) low-power 8GHz band radio modules to establish the identity, presence, and coarse-grained relative locations of devices, and (c) accelerometers to detect tilting of slate devices. The resulting system supports fluid, minimally disruptive techniques for co-located collaboration by leveraging the proxemics of people as well as the proxemics of devices.
Real-time captioning by groups of non-experts BIBAFull-Text 23-34
  Walter Lasecki; Christopher Miller; Adam Sadilek; Andrew Abumoussa; Donato Borrello; Raja Kushalnagar; Jeffrey Bigham
Real-time captioning provides deaf and hard of hearing people immediate access to spoken language and enables participation in dialogue with others. Low latency is critical because it allows speech to be paired with relevant visual cues. Currently, the only reliable source of real-time captions are expensive stenographers who must be recruited in advance and who are trained to use specialized keyboards. Automatic speech recognition (ASR) is less expensive and available on-demand, but its low accuracy, high noise sensitivity, and need for training beforehand render it unusable in real-world situations. In this paper, we introduce a new approach in which groups of non-expert captionists (people who can hear and type) collectively caption speech in real-time on-demand. We present Legion:Scribe, an end-to-end system that allows deaf people to request captions at any time. We introduce an algorithm for merging partial captions into a single output stream in real-time, and a captioning interface designed to encourage coverage of the entire audio stream. Evaluation with 20 local participants and 18 crowd workers shows that non-experts can provide an effective solution for captioning, accurately covering an average of 93.2% of an audio stream with only 10 workers and an average per-word latency of 2.9 seconds. More generally, our model in which multiple workers contribute partial inputs that are automatically merged in real-time may be extended to allow dynamic groups to surpass constituent individuals (even experts) on a variety of human performance tasks.
Carpus: a non-intrusive user identification technique for interactive surfaces BIBAFull-Text 35-44
  Raf Ramakers; Davy Vanacken; Kris Luyten; Karin Coninx; Johannes Schöning
Interactive surfaces have great potential for co-located collaboration because of their ability to track multiple inputs simultaneously. However, the multi-user experience on these devices could be enriched significantly if touch points could be associated with a particular user. Existing approaches to user identification are intrusive, require users to stay in a fixed position, or suffer from poor accuracy. We present a non-intrusive, high-accuracy technique for mapping touches to their corresponding user in a collaborative environment. By mounting a high-resolution camera above the interactive surface, we are able to identify touches reliably without any extra instrumentation, and users are able to move around the surface at will. Our technique, which leverages the back of users' hands as identifiers, supports walk-up-and-use situations in which multiple people interact on a shared surface.
Low-cost audience polling using computer vision BIBAFull-Text 45-54
  Andrew Cross; Edward Cutrell; William Thies
Electronic response systems known as "clickers" have demonstrated educational benefits in well-resourced classrooms, but remain out-of-reach for most schools due to their prohibitive cost. We propose a new, low-cost technique that utilizes computer vision for real-time polling of a classroom. Our approach allows teachers to ask a multiple-choice question. Students respond by holding up a qCard: a sheet of paper that contains a printed code, similar to a QR code, encoding their student IDs. Students indicate their answers (A, B, C or D) by holding the card in one of four orientations. Using a laptop and an off-the-shelf webcam, our software automatically recognizes and aggregates the students' responses and displays them to the teacher. We built this system and performed initial trials in secondary schools in Bangalore, India. In a 25-student classroom, our system offers 99.8% recognition accuracy, captures 97% of responses within 10 seconds, and costs 15 times less than existing electronic solutions.
CrowdScape: interactively visualizing user behavior and output BIBAFull-Text 55-62
  Jeffrey Rzeszotarski; Aniket Kittur
Crowdsourcing has become a powerful paradigm for accomplishing work quickly and at scale, but involves significant challenges in quality control. Researchers have developed algorithmic quality control approaches based on either worker outputs (such as gold standards or worker agreement) or worker behavior (such as task fingerprinting), but each approach has serious limitations, especially for complex or creative work. Human evaluation addresses these limitations but does not scale well with increasing numbers of workers. We present CrowdScape, a system that supports the human evaluation of complex crowd work through interactive visualization and mixed initiative machine learning. The system combines information about worker behavior with worker outputs, helping users to better understand and harness the crowd. We describe the system and discuss its utility through grounded case studies. We explore other contexts where CrowdScape's visualizations might be useful, such as in user studies.

Tutorials & learning

Learning design patterns with Bayesian grammar induction BIBAFull-Text 63-74
  Jerry Talton; Lingfeng Yang; Ranjitha Kumar; Maxine Lim; Noah Goodman; Radomír Mech
Design patterns have proven useful in many creative fields, providing content creators with archetypal, reusable guidelines to leverage in projects. Creating such patterns, however, is a time-consuming, manual process, typically relegated to a few experts in any given domain. In this paper, we describe an algorithmic method for learning design patterns directly from data using techniques from natural language processing and structured concept learning. Given a set of labeled, hierarchical designs as input, we induce a probabilistic formal grammar over these exemplars. Once learned, this grammar encodes a set of generative rules for the class of designs, which can be sampled to synthesize novel artifacts. We demonstrate the method on geometric models and Web pages, and discuss how the learned patterns can drive new interaction mechanisms for content creators.
Adding structured data in unstructured web chat conversation BIBAFull-Text 75-82
  Min Wu; Arin Bhowmick; Joseph Goldberg
Web chat is becoming the primary customer contact channel in customer relationship management (CRM), and Question/Answer/Lookup (QAL) is the dominant communication pattern in CRM agent-to-customer chat. Text-based web chat for QAL has two main usability problems. Chat transcripts between agents and customers are not tightly integrated into agent-side applications, requiring customer service agents to re-enter customer typed data. Also, sensitive information posted in chat sessions in plain text raises security concerns. The addition of web form widgets to web chat not only solves both of these problems but also adds new usability benefits to QAL. Forms can be defined beforehand or, more flexibly, dynamically composed. Two preliminary user studies were conducted. An agent-side study showed that adding inline forms to web chat decreased overall QAL completion time by 47 percent and increased QAL accuracy by removing all potential human errors. A customer-side study showed that web chat with inline forms is intuitive to customers.
Waken: reverse engineering usage information and interface structure from software videos BIBAFull-Text 83-92
  Nikola Banovic; Tovi Grossman; Justin Matejka; George Fitzmaurice
We present Waken, an application-independent system that recognizes UI components and activities from screen captured videos, without any prior knowledge of that application. Waken can identify the cursors, icons, menus, and tooltips that an application contains, and when those items are used. Waken uses frame differencing to identify occurrences of behaviors that are common across graphical user interfaces. Candidate templates are built, and then other occurrences of those templates are identified using a multi-phase algorithm. An evaluation demonstrates that the system can successfully reconstruct many aspects of a UI without any prior application-dependant knowledge. To showcase the design opportunities that are introduced by having this additional meta-data, we present the Waken Video Player, which allows users to directly interact with UI components that are displayed in the video.
MixT: automatic generation of step-by-step mixed media tutorials BIBAFull-Text 93-102
  Pei-Yu Chi; Sally Ahn; Amanda Ren; Mira Dontcheva; Wilmot Li; Björn Hartmann
Users of complex software applications often learn concepts and skills through step-by-step tutorials. Today, these tutorials are published in two dominant forms: static tutorials composed of images and text that are easy to scan, but cannot effectively describe dynamic interactions; and video tutorials that show all manipulations in detail, but are hard to navigate. We hypothesize that a mixed tutorial with static instructions and per-step videos can combine the benefits of both formats. We describe a comparative study of static, video, and mixed image manipulation tutorials with 12 participants and distill design guidelines for mixed tutorials. We present MixT, a system that automatically generates step-by-step mixed media tutorials from user demonstrations. MixT segments screencapture video into steps using logs of application commands and input events, applies video compositing techniques to focus on salient information, and highlights interactions through mouse trails. Informal evaluation suggests that automatically generated mixed media tutorials were as effective in helping users complete tasks as tutorials that were created manually.
GamiCAD: a gamified tutorial system for first time AutoCad users BIBAFull-Text 103-112
  Wei Li; Tovi Grossman; George Fitzmaurice
We present GamiCAD, a gamified in-product, interactive tutorial system for first time AutoCAD users. We introduce a software event driven finite state machine to model a user's progress through a tutorial, which allows the system to provide real-time feedback and recognize success and failures. GamiCAD provides extensive real-time visual and audio feedback that has not been explored before in the context of software tutorials. We perform an empirical evaluation of GamiCAD, comparing it to an equivalent in-product tutorial system without the gamified components. In an evaluation, users using the gamified system reported higher subjective engagement levels and performed a set of testing tasks faster with a higher completion ratio.
Tutorial-based interfaces for cloud-enabled applications BIBAFull-Text 113-122
  Gierad Laput; Eytan Adar; Mira Dontcheva; Wilmot Li
Powerful image editing software like Adobe Photoshop and GIMP have complex interfaces that can be hard to master. To help users perform image editing tasks, we introduce tutorial-based applications (tapps) that retain the step-by-step structure and descriptive text of tutorials but can also automatically apply tutorial steps to new images. Thus, tapps can be used to batch process many images automatically, similar to traditional macros. Tapps also support interactive exploration of parameters, automatic variations, and direct manipulation (e.g., selection, brushing). Another key feature of tapps is that they execute on remote instances of Photoshop, which allows users to edit their images on any Web-enabled device. We demonstrate a working prototype system called TappCloud for creating, managing and using tapps. Initial user feedback indicates support for both the interactive features of tapps and their ability to automate image editing. We conclude with a discussion of approaches and challenges of pushing monolithic direct-manipulation GUIs to the cloud.

Hands & fingers

Facet: a multi-segment wrist worn system BIBAFull-Text 123-130
  Kent Lyons; David Nguyen; Daniel Ashbrook; Sean White
We present Facet, a multi-display wrist worn system consisting of multiple independent touch-sensitive segments joined into a bracelet. Facet automatically determines the pose of the system as a whole and of each segment individually. It further supports multi-segment touch, yielding a rich set of touch input techniques. Our work builds on these two primitives to allow the user to control how applications use segments alone and in coordination. Applications can expand to use more segments, collapses to encompass fewer, and be swapped with other segments. We also explore how the concepts from Facet could apply to other devices in this design space.
iRing: intelligent ring using infrared reflection BIBAFull-Text 131-136
  Masa Ogata; Yuta Sugiura; Hirotaka Osawa; Michita Imai
We present the iRing, an intelligent input ring device developed for measuring finger gestures and external input. iRing recognizes rotation, finger bending, and external force via an infrared (IR) reflection sensor that leverages skin characteristics such as reflectance and softness. Furthermore, iRing allows using a push and stroke input method, which is popular in touch displays. The ring design has potential to be used as a wearable controller because its accessory shape is socially acceptable, easy to install, and safe, and iRing does not require extra devices. We present examples of iRing applications and discuss its validity as an inexpensive wearable interface and as a human sensing device.
Bimanual gesture keyboard BIBAFull-Text 137-146
  Xiaojun Bi; Ciprian Chelba; Tom Ouyang; Kurt Partridge; Shumin Zhai
Gesture keyboards represent an increasingly popular way to input text on mobile devices today. However, current gesture keyboards are exclusively unimanual. To take advantage of the capability of modern multi-touch screens, we created a novel bimanual gesture text entry system, extending the gesture keyboard paradigm from one finger to multiple fingers. To address the complexity of recognizing bimanual gesture, we designed and implemented two related interaction methods, finger-release and space-required, both based on a new multi-stroke gesture recognition algorithm. A formal experiment showed that bimanual gesture behaviors were easy to learn. They improved comfort and reduced the physical demand relative to unimanual gestures on tablets. The results indicated that these new gesture keyboards were valuable complements to unimanual gesture and regular typing keyboards.
Magic finger: always-available input through finger instrumentation BIBAFull-Text 147-156
  Xing-Dong Yang; Tovi Grossman; Daniel Wigdor; George Fitzmaurice
We present Magic Finger, a small device worn on the fingertip, which supports always-available input. Magic Finger inverts the typical relationship between the finger and an interactive surface: with Magic Finger, we instrument the user's finger itself, rather than the surface it is touching. Magic Finger senses touch through an optical mouse sensor, enabling any surface to act as a touch screen. Magic Finger also senses texture through a micro RGB camera, allowing contextual actions to be carried out based on the particular surface being touched. A technical evaluation shows that Magic Finger can accurately sense 22 textures with an accuracy of 98.9%. We explore the interaction design space enabled by Magic Finger, and implement a number of novel interaction techniques that leverage its unique capabilities.
YourGloves, hothands and hotmits: devices to hold hands at a distance BIBAFull-Text 157-166
  Daniel Gooch; Leon Watts
There is a growing body of work in HCI on the design of communication technologies to help support lovers in long distance relationships. We build upon this work by presenting an exploratory study of hand-holding prototypes. Our work distinguishes itself by basing distance communication metaphors on elements of familiar, simple co-located behaviours. We argue that the combined evocative power of unique co-created physical representations of the absent other can be used by separated lovers to generate powerful and positive experiences, in turn sustaining romantic connections at a distance.
Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor BIBAFull-Text 167-176
  David Kim; Otmar Hilliges; Shahram Izadi; Alex D. Butler; Jiawen Chen; Iason Oikonomidis; Patrick Olivier
Digits is a wrist-worn sensor that recovers the full 3D pose of the user's hand. This enables a variety of freehand interactions on the move. The system targets mobile settings, and is specifically designed to be low-power and easily reproducible using only off-the-shelf hardware. The electronics are self-contained on the user's wrist, but optically image the entirety of the user's hand. This data is processed using a new pipeline that robustly samples key parts of the hand, such as the tips and lower regions of each finger. These sparse samples are fed into new kinematic models that leverage the biomechanical constraints of the hand to recover the 3D pose of the user's hand. The proposed system works without the need for full instrumentation of the hand (for example using data gloves), additional sensors in the environment, or depth cameras which are currently prohibitive for mobile scenarios due to power and form-factor considerations. We demonstrate the utility of Digits for a variety of application scenarios, including 3D spatial interaction with mobile devices, eyes-free interaction on-the-move, and gaming. We conclude with a quantitative and qualitative evaluation of our system, and discussion of strengths, limitations and future work.


Clui: a platform for handles to rich objects BIBAFull-Text 177-188
  Hubert Pham; Justin Mazzola Paluska; Rob Miller; Steve Ward
On the desktop, users are accustomed to having visible handles to objects that they want to organize, share, or manipulate. Web applications today feature many classes of such objects, like flight itineraries, products for sale, people, recipes, and businesses, but there are no interoperable handles for high-level semantic objects that users can grab. This paper proposes Clui, a platform for exploring a new data type, called a Webit, that provides uniform handles to rich objects. Clui uses plugins to 1) create Webits on existing pages by extracting semantic data from those pages, and 2) augmenting existing sites with drag and drop targets that accept and interpret Webits. Users drag and drop Webits between sites to transfer data, auto-fill search forms, map associated locations, or share Webits with others. Clui enables experimentation with handles to semantic objects and the standards that underlie them.
DejaVu: integrated support for developing interactive camera-based programs BIBAFull-Text 189-196
  Jun Kato; Sean McDirmid; Xiang Cao
The increasing popularity of interactive camera-based programs highlights the inadequacies of conventional IDEs in developing these programs given their distinctive attributes and workflows. We present DejaVu, an IDE enhancement that eases the development of these programs by enabling programmers to visually and continuously monitor program data in consistency with the frame-based pipeline of computer-vision programs; and to easily record, review, and reprocess temporal data to iteratively improve the processing of non-reproducible camera input. DejaVu was positively received by three experienced programmers of interactive camera-based programs in our preliminary user trial.
Homework: putting interaction into the infrastructure BIBAFull-Text 197-206
  Richard Mortier; Tom Rodden; Peter Tolmie; Tom Lodge; Robert Spencer; Andy Crabtree; Joe Sventek; Alexandros Koliousis
This paper presents a user driven redesign of the domestic network infrastructure that draws upon a series of ethnographic studies of home networks. We present an infrastructure based around a purpose built access point that has modified the handling of protocols and services to reflect the interactive needs of the home. The developed infrastructure offers a novel measurement framework that allows a broad range of infrastructure information to be easily captured and made available to interactive applications. This is complemented by a diverse set of novel interactive control mechanisms and interfaces for the underlying infrastructure. We also briefly reflect on the technical and user issues arising from deployments.
DataPlay: interactive tweaking and example-driven correction of graphical database queries BIBAFull-Text 207-218
  Azza Abouzied; Joseph Hellerstein; Avi Silberschatz
Writing complex queries in SQL is a challenge for users. Prior work has developed several techniques to ease query specification but none of these techniques are applicable to a particularly difficult class of queries: quantified queries. Our hypothesis is that users prefer to specify quantified queries interactively by trial-and-error. We identify two impediments to this form of interactive trial-and-error query specification in SQL: (i) changing quantifiers often requires global syntactical query restructuring, and (ii) the absence of non-answers from SQL's results makes verifying query correctness difficult. We remedy these issues with DataPlay, a query tool with an underlying graphical query language, a unique data model and a graphical interface. DataPlay provides two interaction features that support trial-and-error query specification. First, DataPlay allows users to directly manipulate a graphical query by changing quantifiers and modifying dependencies between constraints. Users receive real-time feedback in the form of updated answers and non-answers. Second, DataPlay can auto-correct a user's query, based on user feedback about which tuples to keep or drop from the answers and non-answers. We evaluated the effectiveness of each interaction feature with a user study and we found that direct query manipulation is more effective than auto-correction for simple queries but auto-correction is more effective than direct query manipulation for more complex queries.
SnipMatch: using source code context to enhance snippet retrieval and parameterization BIBAFull-Text 219-228
  Doug Wightman; Zi Ye; Joel Brandt; Roel Vertegaal
Programmers routinely use source code snippets to increase their productivity. However, locating and adapting code snippets to the current context still takes time: for example, variables must be renamed, and dependencies included. We believe that when programmers decide to invest time in creating a new code snippet from scratch, they would also be willing to spend additional effort to make that code snippet configurable and easy to integrate. To explore this insight, we built SnipMatch, a plug-in for the Eclipse IDE. SnipMatch introduces a simple markup that allows snippet authors to specify search patterns and integration instructions. SnipMatch leverages this information, in conjunction with current code context, to improve snippet search and parameterization. For example, when a search query includes local variables, SnipMatch suggests compatible snippets, and automatically adapts them by substituting in these variables. In the lab, we observed that participants integrated snippets faster when using SnipMatch than when using standard Eclipse. Findings from a public deployment to 93 programmers suggest that SnipMatch has become integrated into the work practices of real users.
ConstraintJS: programming interactive behaviors for the web by integrating constraints and states BIBAFull-Text 229-238
  Stephen Oney; Brad Myers; Joel Brandt
Interactive behaviors in GUIs are often described in terms of states, transitions, and constraints, where the constraints only hold in certain states. These constraints maintain relationships among objects, control the graphical layout, and link the user interface to an underlying data model. However, no existing Web implementation technology provides direct support for all of these, so the code for maintaining constraints and tracking state may end up spread across multiple languages and libraries. In this paper we describe ConstraintJS, a system that integrates constraints and finite-state machines (FSMs) with Web languages. A key role for the FSMs is to enable and disable constraints based on the interface's current mode, making it possible to write constraints that sometimes hold. We illustrate that constraints combined with FSMs can be a clearer way of defining many interactive behaviors with a series of examples.
User interface toolkit mechanisms for securing interface elements BIBAFull-Text 239-250
  Franziska Roesner; James Fogarty; Tadayoshi Kohno
User interface toolkit research has traditionally assumed that developers have full control of an interface. This assumption is challenged by the mashup nature of many modern interfaces, in which different portions of a single interface are implemented by multiple, potentially mutually distrusting developers (e.g., an Android application embedding a third-party advertisement). We propose considering security as a primary goal for user interface toolkits. We motivate the need for security at this level by examining today's mashup scenarios, in which security and interface flexibility are not simultaneously achieved. We describe a security-aware user interface toolkit architecture that secures interface elements while providing developers with the flexibility and expressivity traditionally desired in a user interface toolkit. By challenging trust assumptions inherent in existing approaches, this architecture effectively addresses important interface-level security concerns.

Interactions 1

Cliplets: juxtaposing still and dynamic imagery BIBAFull-Text 251-260
  Neel Joshi; Sisil Mehta; Steven Drucker; Eric Stollnitz; Hugues Hoppe; Matt Uyttendaele; Michael Cohen
We explore creating ""cliplets"", a form of visual media that juxtaposes still image and video segments, both spatially and temporally, to expressively abstract a moment. Much as in ""cinemagraphs"", the tension between static and dynamic elements in a cliplet reinforces both aspects, strongly focusing the viewer's attention. Creating this type of imagery is challenging without professional tools and training. We develop a set of idioms, essentially spatiotemporal mappings, that characterize cliplet elements, and use these idioms in an interactive system to quickly compose a cliplet from ordinary handheld video. One difficulty is to avoid artifacts in the cliplet composition without resorting to extensive manual input. We address this with automatic alignment, looping optimization and feathering, simultaneous matting and compositing, and Laplacian blending. A key user-interface challenge is to provide affordances to define the parameters of the mappings from input time to output time while maintaining a focus on the cliplet being created. We demonstrate the creation of a variety of cliplet types. We also report on informal feedback as well as a more structured survey of users.
JellyLens: content-aware adaptive lenses BIBAFull-Text 261-270
  Cyprien Pindat; Emmanuel Pietriga; Olivier Chapuis; Claude Puech
Focus+context lens-based techniques smoothly integrate two levels of detail using spatial distortion to connect the magnified region and the context. Distortion guarantees visual continuity, but causes problems of interpretation and focus targeting, partly due to the fact that most techniques are based on statically-defined, regular lens shapes, that result in far-from-optimal magnification and distortion. JellyLenses dynamically adapt to the shape of the objects of interest, providing detail-in-context visualizations of higher relevance by optimizing what regions fall into the focus, context and spatially-distorted transition regions. This both improves the visibility of content in the focus region and preserves a larger part of the context region. We describe the approach and its implementation, and report on a controlled experiment that evaluates the usability of JellyLenses compared to regular fisheye lenses, showing clear performance improvements with the new technique for a multi-scale visual search task.
PiVOT: personalized view-overlays for tabletops BIBAFull-Text 271-280
  Abhijit Karnik; Diego Martinez Plasencia; Walterio Mayol-Cuevas; Sriram Subramanian
We present PiVOT, a tabletop system aimed at supporting mixed-focus collaborative tasks. Through two view-zones, PiVOT provides personalized views to individual users while presenting an unaffected and unobstructed shared view to all users. The system supports multiple personalized views which can be present at the same spatial location and yet be only visible to the users it belongs to. The system also allows the creation of personal views that can be either 2D or (auto-stereoscopic) 3D images. We first discuss the motivation and the different implementation principles required for realizing such a system, before exploring different designs able to address the seemingly opposing challenges of shared and personalized views. We then implement and evaluate a sample prototype to validate our design ideas and present a set of sample applications to demonstrate the utility of the system.
Histomages: fully synchronized views for image editing BIBAFull-Text 281-286
  Fanny Chevalier; Pierre Dragicevic; Christophe Hurter
We present Histomages, a new interaction model for image editing that considers color histograms as spatial rearrangements of image pixels. Users can select pixels on image histograms as they would select image regions and directly manipulate them to adjust their colors. Histomages are also affected by other image tools such as paintbrushes. We explore some possibilities offered by this interaction model, and discuss the four key principles behind it as well as their implications for the design of feature-rich software in general.


Sketch-editing games: human-machine communication, game theory and applications BIBAFull-Text 287-298
  Andre Ribeiro; Takeo Igarashi
We study uncertainty in graphical-based interaction (with special attention to sketches). We argue that a comprehensive model for the problem must include the interaction participants (and their current beliefs), their possible actions and their past sketches. It's yet unclear how to frame and solve the former problem, considering all the latter elements. We suggest framing the problem as a game and solving it with a game-theoretical solution, which leads to a framework for the design of new two-way, sketch-based user interfaces. In special, we use the framework to design a game that can progressively learn visual models of objects from user sketches, and use the models in real-world interactions. Instead of an abstract visual criterion, players in this game learn models to optimize interaction (the game's duration). This two-way sketching game addresses problems essential in emerging interfaces (such as learning and how to deal with interpretation errors). We review possible applications in robotic sketch-to-command, hand gesture recognition, media authoring and visual search, and evaluate two. Evaluations demonstrate how players improve performance with repeated play, and the influence of interaction aspects on learning.
Interpreting strokes on paper with a mobile assistant BIBAFull-Text 299-308
  Theophanis Tsandilas
Digital pen technology has allowed for the easy transfer of pen data from paper to the computer. However, linking handwritten content with the digital world remains a hard problem as it requires the translation of unstructured and highly personal vocabularies into structured ones that computers can easily understand and process. Automatic recognition can help to this direction, but as it is not always reliable, solutions require the active cooperation between users and recognition algorithms. This work examines the use of portable touch-screen devices in connection with pen and paper to help users direct and refine the interpretation of their strokes on paper. We explore four techniques of bi-manual interaction that combine touch and pen-writing, where user attention is divided between the original strokes on paper and their interpretation by the electronic device. We demonstrate the techniques through a mobile interface for writing music that complements the automatic recognition with interactive user-driven interpretation. An experiment evaluates the four techniques and provides insights about their strengths and limitations.
High-performance pen + touch modality interactions: a real-time strategy game eSports context BIBAFull-Text 309-318
  William Hamilton; Andruid Kerne; Tom Robbins
We used the situated context of real-time strategy (RTS) games to address the design and evaluation of new pen + touch interaction techniques. RTS play is a popular genre of Electronic Sports (eSports), games played and spectated at an extremely high level. Interaction techniques are critical for eSports players, because they so directly impact performance.
   Through this process, new techniques and implications for pen + touch and bi-manual interaction emerged. We enhance non-dominant hand (NDH) interaction with edge-constrained affordances, anchored to physical features of interactive surfaces, effectively increasing target width. We develop bi-manual overloading, an approach to reduce the total number of occurrences of NDH retargeting. The novel isosceles lasso select technique facilitates selection of complex object subsets. Pen-in-hand interaction, dominant hand touch interaction performed with the pen stowed in the palm, also emerged as an efficient and expressive interaction paradigm.
GaussSense: attachable stylus sensing using magnetic sensor grid BIBAFull-Text 319-326
  Rong-Hao Liang; Kai-Yin Cheng; Chao-Huai Su; Chien-Ting Weng; Bing-Yu Chen; De-Nian Yang
This work presents GaussSense, which is a back-of-device sensing technique for enabling input on an arbitrary surface using stylus by exploiting magnetism. A 2mm-thick Hall sensor grid is developed to sense magnets that are embedded in the stylus. Our system can sense the magnetic field that is emitted from the stylus when it is within 2cm of any non-ferromagnetic surface. Attaching the sensor behind an arbitrary thin surface enables the stylus input to be recognized by analyzing the distribution of the applied magnetic field. Attaching the sensor grid to the back of a touchscreen device and incorporating magnets into the corresponding stylus enable the system 1) to distinguish touch events that are caused by a finger from those caused by the stylus, 2) to sense the tilt angle of the stylus and the pressure with which it is applied, and 3) to detect where the stylus hovers over the screen. A pilot study reveals that people were satisfied with the novel sketching experiences based on this system.
FlexAura: a flexible near-surface range sensor BIBAFull-Text 327-330
  Shenwei Liu; François Guimbretière
The availability of flexible capacitive sensors that can be fitted around mice, smartphones, and pens carries great potential in leveraging grasp as a new interaction modality. Unfortunately, most capacitive sensors only track interaction directly on the surface, making it harder to differentiate among grips and constraining user movements. We present a new optical range sensor design based on high power infrared LEDs and photo-transistors, which can be fabricated on a flexible PCB and wrapped around a wide variety of graspable objects including pens, mice, smartphones, and slates. Our sensor offers a native resolution of 10 dpi with a sensing range of up to 30mm (1.2") and sampling speed of 50Hz. Based on our prototype wrapped around the barrel of a pen, we present a summary of the characteristics of the sensor and describe the sensor output in several typical pen grips. Our design is versatile enough to apply not only to pens but to a wide variety of graspable objects including smartphones and slates.
PhantomPen: virtualization of pen head for digital drawing free from pen occlusion & visual parallax BIBAFull-Text 331-340
  David Lee; KyoungHee Son; Joon Hyub Lee; Seok-Hyung Bae
We present PhantomPen, a direct pen input device whose pen head is virtualized onto the tablet display surface and visually connected to a graspable pen barrel in order to achieve digital drawing free from pen occlusion and visual parallax. As the pen barrel approaches the display, the virtual pen head smoothly appears as if the rendered virtual pen head and the physical pen barrel are in unity. The virtual pen head provides visual feedback by changing its virtual form according to pen type, color, and thickness while the physical pen tip, hidden in the user's sight, provides tactile feedback. Three experiments were carefully designed based on an analysis of drawings by design professionals and observations of design drawing classes. With these experiments that simulate natural drawing we proved significant performance advantages of PhantomPen. PhantomPen was at least as usable as the normal stylus in basic line drawing, and was 17% faster in focus region drawing (26% faster in extreme focus region drawing). PhantomPen also reduced error rate by 40% in a typical drawing setup where users have to manage a complex combination of pen and stroke properties.

Interactions II

Exposing and understanding scrolling transfer functions BIBAFull-Text 341-350
  Philip Quinn; Andy Cockburn; Géry Casiez; Nicolas Roussel; Carl Gutwin
Scrolling is controlled through many forms of input devices, such as mouse wheels, trackpad gestures, arrow keys, and joysticks. Performance with these devices can be adjusted by introducing variable transfer functions to alter the range of expressible speed, precision, and sensitivity. However, existing transfer functions are typically "black boxes" bundled into proprietary operating systems and drivers. This presents three problems for researchers: (1) a lack of knowledge about the current state of the field; (2) a difficulty in replicating research that uses scrolling devices; and (3) a potential experimental confound when evaluating scrolling devices and techniques. These three problems are caused by gaps in researchers' knowledge about what device and movement factors are important for scrolling transfer functions, and about how existing devices and drivers use these factors. We fill these knowledge gaps with a framework of transfer function factors for scrolling, and a method for analysing proprietary transfer functions -- demonstrating how state of the art commercial devices accommodate some of the human control phenomena observed in prior studies.
Giving a hand to the eyes: leveraging input accuracy for subpixel interaction BIBAFull-Text 351-358
  Nicolas Roussel; Géry Casiez; Jonathan Aceituno; Daniel Vogel
We argue that the current practice of using integer positions for pointing events artificially constrains human precision capabilities. The high sensitivity of current input devices can be harnessed to enable precise direct manipulation ""in between"" pixels, called subpixel interaction. We provide detailed analysis of subpixel theory and implementation, including the critical component of revised control-display gain transfer functions. A prototype implementation is described with several illustrative examples. Guidelines for subpixel domain applicability are provided and an overview of required changes to operating systems and graphical user interface frameworks are discussed.
UnderScore: musical underlays for audio stories BIBAFull-Text 359-366
  Steve Rubin; Floraine Berthouzoz; Gautham Mysore; Wilmot Li; Maneesh Agrawala
Audio producers often use musical underlays to emphasize key moments in spoken content and give listeners time to reflect on what was said. Yet, creating such underlays is time-consuming as producers must carefully (1) mark an emphasis point in the speech (2) select music with the appropriate style, (3) align the music with the emphasis point, and (4) adjust dynamics to produce a harmonious composition. We present UnderScore, a set of semi-automated tools designed to facilitate the creation of such underlays. The producer simply marks an emphasis point in the speech and selects a music track. UnderScore automatically refines, aligns and adjusts the speech and music to generate a high-quality underlay. UnderScore allows producers to focus on the high-level design of the underlay; they can quickly try out a variety of music and test different points of emphasis in the story. Amateur producers, who may lack the time or skills necessary to author underlays, can quickly add music to their stories. An informal evaluation of UnderScore suggests that it can produce high-quality underlays for a variety of examples while significantly reducing the time and effort required of radio producers.
Accessible skimming: faster screen reading of web pages BIBAFull-Text 367-378
  Faisal Ahmed; Yevgen Borodin; Andrii Soviak; Muhammad Islam; I. V. Ramakrishnan; Terri Hedgpeth
In our information-driven web-based society, we are all gradually falling ""victims"" to information overload [5]. However, while sighted people are finding ways to sift through information faster, Internet users who are blind are experiencing an even greater information overload. These people access computers and Internet using screen-reader software, which reads the information on a computer screen sequentially using computer-generated speech. While sighted people can learn how to quickly glance over the headlines and news articles online to get the gist of information, people who are blind have to use keyboard shortcuts to listen through the content narrated by a serial audio interface. This interface does not give them an opportunity to know what content to skip and what to listen to. So, they either listen to all of the content or listen to the first part of each sentence or paragraph before they skip to the next one. In this paper, we propose an automated approach to facilitate non-visual skimming of web pages. We describe the underlying algorithm, outline a non-visual skimming interface, and report on the results of automated experiments, as well as on our user study with 23 screen-reader users. The results of the experiments suggest that we have been moderately successful in designing a viable algorithm for automatic summarization that could be used for non-visual skimming. In our user studies, we confirmed that people who are blind could read and search through online articles faster and were able to understand and remember most of what they have read with our skimming system. Finally, all 23 participants expressed genuine interest in using non-visual skimming in the future.

Augmented reality

PICOntrol: using a handheld projector for direct control of physical devices through visible light BIBAFull-Text 379-388
  Dominik Schmidt; David Molyneaux; Xiang Cao
Today's environments are populated with a growing number of electric devices which come in diverse form factors and provide a plethora of functions. However, rich interaction with these devices can become challenging if they need be controlled from a distance, or are too small to accommodate user interfaces on their own. In this work, we explore PICOntrol, a new approach using an off-the-shelf handheld pico projector for direct control of physical devices through visible light. The projected image serves a dual purpose by simultaneously presenting a visible interface to the user, and transmitting embedded control information to inexpensive sensor units integrated with the devices. To use PICOntrol, the user points the handheld projector at a target device, overlays a projected user interface on its sensor unit, and performs various GUI-style or gestural interactions. PICOntrol enables direct, visible, and rich interactions with various physical devices without requiring central infrastructure. We present our prototype implementation as well as explorations of its interaction space through various application examples.
DuploTrack: a real-time system for authoring and guiding Duplo block assembly BIBAFull-Text 389-402
  Ankit Gupta; Dieter Fox; Brian Curless; Michael Cohen
We demonstrate a realtime system which infers and tracks the assembly process of a snap-together block model using a Kinect® sensor. The inference enables us to build a virtual replica of the model at every step. Tracking enables us to provide context specific visual feedback on a screen by augmenting the rendered virtual model aligned with the physical model. The system allows users to author a new model and uses the inferred assembly process to guide its recreation by others. We propose a novel way of assembly guidance where the next block to be added is rendered in blinking mode with the tracked virtual model on screen. The system is also able to detect any mistakes made and helps correct them by providing appropriate feedback. We focus on assemblies of Duplo® blocks.
   We discuss the shortcomings of existing methods of guidance -- static figures or recorded videos -- and demonstrate how our method avoids those shortcomings. We also report on a user study to compare our system with standard figure-based guidance methods found in user manuals. The results of the user study suggest that our method is able to aid users' structural perception of the model better, leads to fewer assembly errors, and reduces model construction time.
Capturing indoor scenes with smartphones BIBAFull-Text 403-412
  Aditya Sankar; Steven Seitz
In this paper, we present a novel smartphone application designed to easily capture, visualize and reconstruct homes, offices and other indoor scenes. Our application leverages data from smartphone sensors such as the camera, accelerometer, gyroscope and magnetometer to help model the indoor scene. The output of the system is two-fold; first, an interactive visual tour of the scene is generated in real time that allows the user to explore each room and transition between connected rooms. Second, with some basic interactive photogrammetric modeling the system generates a 2D floor plan and accompanying 3D model of the scene, under a Manhattan-world assumption. The approach does not require any specialized equipment or training and is able to produce accurate floor plans.
Steerable augmented reality with the beamatron BIBAFull-Text 413-422
  Andrew Wilson; Hrvoje Benko; Shahram Izadi; Otmar Hilliges
Steerable displays use a motorized platform to orient a projector to display graphics at any point in the room. Often a camera is included to recognize markers and other objects, as well as user gestures in the display volume. Such systems can be used to superimpose graphics onto the real world, and so are useful in a number of augmented reality and ubiquitous computing scenarios. We contribute the Beamatron, which advances steerable displays by drawing on recent progress in depth camera-based interactions. The Beamatron consists of a computer-controlled pan and tilt platform on which is mounted a projector and Microsoft Kinect sensor. While much previous work with steerable displays deals primarily with projecting corrected graphics onto a discrete set of static planes, we describe computational techniques that enable reasoning in 3D using live depth data. We show two example applications that are enabled by the unique capabilities of the Beamatron: an augmented reality game in which a player can drive a virtual toy car around a room, and a ubiquitous computing demo that uses speech and gesture to move projected graphics throughout the room.
3D puppetry: a Kinect-based interface for 3D animation BIBAFull-Text 423-434
  Robert Held; Ankit Gupta; Brian Curless; Maneesh Agrawala
We present a system for producing 3D animations using physical objects (i.e., puppets) as input. Puppeteers can load 3D models of familiar rigid objects, including toys, into our system and use them as puppets for an animation. During a performance, the puppeteer physically manipulates these puppets in front of a Kinect depth sensor. Our system uses a combination of image-feature matching and 3D shape matching to identify and track the physical puppets. It then renders the corresponding 3D models into a virtual set. Our system operates in real time so that the puppeteer can immediately see the resulting animation and make adjustments on the fly. It also provides 6D virtual camera \\rev{and lighting} controls, which the puppeteer can adjust before, during, or after a performance. Finally our system supports layered animations to help puppeteers produce animations in which several characters move at the same time. We demonstrate the accessibility of our system with a variety of animations created by puppeteers with no prior animation experience.
KinÊtre: animating the world with the human body BIBAFull-Text 435-444
  Jiawen Chen; Shahram Izadi; Andrew Fitzgibbon
KinÊtre allows novice users to scan arbitrary physical objects and bring them to life in seconds. The fully interactive system allows diverse static meshes to be animated using the entire human body. Traditionally, the process of mesh animation is laborious and requires domain expertise, with rigging specified manually by an artist when designing the character. KinÊtre makes creating animations a more playful activity, conducted by novice users interactively "at runtime". This paper describes the KinÊtre system in full, highlighting key technical contributions and demonstrating many examples of users animating meshes of varying shapes and sizes. These include non-humanoid meshes and incomplete surfaces produced by 3D scanning -- two challenging scenarios for existing mesh animation systems. Rather than targeting professional CG animators, KinÊtre is intended to bring mesh animation to a new audience of novice users. We demonstrate potential uses of our system for interactive storytelling and new forms of physical gaming.


Dynamic tactile guidance for visual search tasks BIBAFull-Text 445-452
  Ville Lehtinen; Antti Oulasvirta; Antti Salovaara; Petteri Nurmi
Visual search in large real-world scenes is both time consuming and frustrating, because the search becomes serial when items are visually similar. Tactile guidance techniques can facilitate search by allowing visual attention to focus on a subregion of the scene. We present a technique for dynamic tactile cueing that couples hand position with a scene position and uses tactile feedback to guide the hand actively toward the target. We demonstrate substantial improvements in task performance over a baseline of visual search only, when the scene's complexity increases. Analyzing task performance, we demonstrate that the effect of visual complexity can be practically eliminated through improved spatial precision of the guidance.
Designing for low-latency direct-touch input BIBAFull-Text 453-464
  Albert Ng; Julian Lepinski; Daniel Wigdor; Steven Sanders; Paul Dietz
Software designed for direct-touch interfaces often utilize a metaphor of direct physical manipulation of pseudo "real-world" objects. However, current touch systems typically take 50-200ms to update the display in response to a physical touch action. Utilizing a high performance touch demonstrator, subjects were able to experience touch latencies ranging from current levels down to about 1ms. Our tests show that users greatly prefer lower latencies, and noticeable improvement continued well below 10ms. This level of performance is difficult to achieve in commercial computing systems using current technologies. As an alternative, we propose a hybrid system that provides low-fidelity visu-al feedback immediately, followed by high-fidelity visuals at standard levels of latency.
A user-specific machine learning approach for improving touch accuracy on mobile devices BIBAFull-Text 465-476
  Daryl Weir; Simon Rogers; Roderick Murray-Smith; Markus Löchtefeld
We present a flexible Machine Learning approach for learning user-specific touch input models to increase touch accuracy on mobile devices. The model is based on flexible, non-parametric Gaussian Process regression and is learned using recorded touch inputs. We demonstrate that significant touch accuracy improvements can be obtained when either raw sensor data is used as an input or when the device's reported touch location is used as an input, with the latter marginally outperforming the former. We show that learned offset functions are highly nonlinear and user-specific and that user-specific models outperform models trained on data pooled from several users. Crucially, significant performance improvements can be obtained with a small (≈200) number of training examples, easily obtained for a particular user through a calibration game or from keyboard entry data.
Proton++: a customizable declarative multitouch framework BIBAFull-Text 477-486
  Kenrick Kin; Björn Hartmann; Tony DeRose; Maneesh Agrawala
Proton++ is a declarative multitouch framework that allows developers to describe multitouch gestures as regular expressions of touch event symbols. It builds on the Proton framework by allowing developers to incorporate custom touch attributes directly into the gesture description. These custom attributes increase the expressivity of the gestures, while preserving the benefits of Proton: automatic gesture matching, static analysis of conflict detection, and graphical gesture creation. We demonstrate Proton++'s flexibility with several examples: a direction attribute for describing trajectory, a pinch attribute for detecting when touches move towards one another, a touch area attribute for simulating pressure, an orientation attribute for selecting menu items, and a screen location attribute for simulating hand ID. We also use screen location to simulate user ID and enable simultaneous recognition of gestures by multiple users. In addition, we show how to incorporate timing into Proton++ gestures by reporting touch events at a regular time interval. Finally, we present a user study that suggests that users are roughly four times faster at interpreting gestures written using Proton++ than those written in procedural event-handling code commonly used today.
Extended multitouch: recovering touch posture and differentiating users using a depth camera BIBAFull-Text 487-496
  Sundar Murugappan; A Vinayak; Niklas Elmqvist; Karthik Ramani
Multitouch surfaces are becoming prevalent, but most existing technologies are only capable of detecting the user's actual points of contact on the surface and not the identity, posture, and handedness of the user. In this paper, we define the concept of extended multitouch interaction as a richer input modality that includes all of this information. We further present a practical solution to achieve this on tabletop displays based on mounting a single commodity depth camera above a horizontal surface. This will enable us to not only detect when the surface is being touched, but also recover the user's exact finger and hand posture, as well as distinguish between different users and their handedness. We validate our approach using two user studies, and deploy the technique in a scratchpad tool and in a pen + touch sketch tool.
FlowBlocks: a multi-touch ui for crowd interaction BIBAFull-Text 497-508
  Florian Block; Daniel Wigdor; Brenda Caldwell Phillips; Michael S. Horn; Chia Shen
Multi-touch technology lends itself to collaborative crowd interaction (CI). However, common tap-operated widgets are impractical for CI, since they are susceptible to accidental touches and interference from other users. We present a novel multi-touch interface called FlowBlocks in which every UI action is invoked through a small sequence of user actions: dragging parametric UI-Blocks, and dropping them over operational UI-Docks. The FlowBlocks approach is advantageous for CI because it a) makes accidental touches inconsequential; and b) introduces design parameters for mutual awareness, concurrent input, and conflict management. FlowBlocks was successfully used on the floor of a busy natural history museum. We present the complete design space and describe a year-long iterative design and evaluation process which employed the Rapid Iterative Test and Evaluation (RITE) method in a museum setting.

Tactile & grip

PyzoFlex: printed piezoelectric pressure sensing foil BIBAFull-Text 509-518
  Christian Rendl; Patrick Greindl; Michael Haller; Martin Zirkl; Barbara Stadlober; Paul Hartmann
Ferroelectric material supports both pyro- and piezoelectric effects that can be used for sensing pressures on large, bended surfaces. We present PyzoFlex, a pressure-sensing input device that is based on a ferroelectric material. It is constructed with a sandwich structure of four layers that can be printed easily on any material. We use this material in combination with a high-resolution Anoto-sensing foil to support both hand and pen input tracking. The foil is bendable, energy-efficient, and it can be produced in a printing process. Even a hovering mode is feasible due to its pyroelectric effect. In this paper, we introduce this novel input technology and discuss its benefits and limitations.
Jamming user interfaces: programmable particle stiffness and sensing for malleable and shape-changing devices BIBAFull-Text 519-528
  Sean Follmer; Daniel Leithinger; Alex Olwal; Nadia Cheng; Hiroshi Ishii
Malleable and organic user interfaces have the potential to enable radically new forms of interactions and expressiveness through flexible, free-form and computationally controlled shapes and displays. This work, specifically focuses on particle jamming as a simple, effective method for flexible, shape-changing user interfaces where programmatic control of material stiffness enables haptic feedback, deformation, tunable affordances and control gain. We introduce a compact, low-power pneumatic jamming system suitable for mobile devices, and a new hydraulic-based technique with fast, silent actuation and optical shape sensing. We enable jamming structures to sense input and function as interaction devices through two contributed methods for high-resolution shape sensing using: 1) index-matched particles and fluids, and 2) capacitive and electric field sensing. We explore the design space of malleable and organic user interfaces enabled by jamming through four motivational prototypes that highlight jamming's potential in HCI, including applications for tabletops, tablets and for portable shape-changing mobile devices.
A thin stretchable interface for tangential force measurement BIBAFull-Text 529-536
  Yuta Sugiura; Masahiko Inami; Takeo Igarashi
We have developed a simple skin-like user interface that can be easily attached to curved as well as flat surfaces and used to measure tangential force generated by pinching and dragging interactions. The interface consists of several photoreflectors that consist of an IR LED and a phototransistor and elastic fabric such as stocking and rubber membrane. The sensing method used is based on our observation that photoreflectors can be used to measure the ratio of expansion and contraction of a stocking using the changes in transmissivity of IR light passing through the stocking. Since a stocking is thin, stretchable, and nearly transparent, it can be easily attached to various types of objects such as mobile devices, robots, and different parts of the body as well as to various types of conventional pressure sensors without altering the original shape of the object. It can also present natural haptic feedback in accordance with the amount of force exerted. A system using several such sensors can determine the direction of a two-dimensional force. A variety of example applications illustrated the utility of this sensing system.
Capacitive fingerprinting: exploring user differentiation by sensing electrical properties of the human body BIBAFull-Text 537-544
  Chris Harrison; Munehiko Sato; Ivan Poupyrev
At present, touchscreens can differentiate multiple points of contact, but not who is touching the device. In this work, we consider how the electrical properties of humans and their attire can be used to support user differentiation on touchscreens. We propose a novel sensing approach based on Swept Frequency Capacitive Sensing, which measures the impedance of a user to the environment (i.e., ground) across a range of AC frequencies. Different people have different bone densities and muscle mass, wear different footwear, and so on. This, in turn, yields different impedance profiles, which allows for touch events, including multitouch gestures, to be attributed to a particular user. This has many interesting implications for interactive design. We describe and evaluate our sensing approach, demonstrating that the technique has considerable promise. We also discuss limitations, how these might be overcome, and next steps.
GripSense: using built-in sensors to detect hand posture and pressure on commodity mobile phones BIBAFull-Text 545-554
  Mayank Goel; Jacob Wobbrock; Shwetak Patel
We introduce GripSense, a system that leverages mobile device touchscreens and their built-in inertial sensors and vibration motor to infer hand postures including one- or two-handed interaction, use of thumb or index finger, or use on a table. GripSense also senses the amount of pres-sure a user exerts on the touchscreen despite a lack of direct pressure sensors by inferring from gyroscope readings when the vibration motor is "pulsed." In a controlled study with 10 participants, GripSense accurately differentiated device usage on a table vs. in hand with 99.67% accuracy and when in hand, it inferred hand postures with 84.26% accuracy. In addition, GripSense distinguished three levels of pressure with 95.1% accuracy. A usability analysis of GripSense was conducted in three custom applications and showed that pressure input and hand-posture sensing can be useful in a number of scenarios.
Pressages: augmenting phone calls with non-verbal messages BIBAFull-Text 555-562
  Eve Hoggan; Craig Stewart; Laura Haverinen; Giulio Jacucci; Vuokko Lantz
ForcePhone is a mobile synchronous haptic communication system. During phone calls, users can squeeze the side of the device and the pressure level is mapped to vibrations on the recipient's device. The pressure/vibrotactile messages supported by ForcePhone are called pressages. Using a lab-based study and a small field study, this paper addresses the following questions: how can haptic interpersonal communication be integrated into a standard mobile device? What is the most appropriate feedback design for pressages? What types of non-verbal cues can be represented by pressages? Do users make use of pressages during their conversations? The results of this research indicate that such a system has value as a communication channel in real-world settings with users expressing greetings, presence and emotions through pressages.

Fabrication & hardware

Acoustic barcodes: passive, durable and inexpensive notched identification tags BIBAFull-Text 563-568
  Chris Harrison; Robert Xiao; Scott Hudson
We present acoustic barcodes, structured patterns of physical notches that, when swiped with e.g., a fingernail, produce a complex sound that can be resolved to a binary ID. A single, inexpensive contact microphone attached to a surface or object is used to capture the waveform. We present our method for decoding sounds into IDs, which handles variations in swipe velocity and other factors. Acoustic barcodes could be used for information retrieval or to triggering interactive functions. They are passive, durable and inexpensive to produce. Further, they can be applied to a wide range of materials and objects, including plastic, wood, glass and stone. We conclude with several example applications that highlight the utility of our approach, and a user study that explores its feasibility.
PICL: portable in-circuit learner BIBAFull-Text 569-578
  Adam Fourney; Michael Terry
This paper introduces the PICL, the portable in-circuit learner. The PICL explores the possibility of providing standalone, low-cost, programming-by-demonstration machine learning capabilities to circuit prototyping. To train the PICL, users attach a sensor to the PICL, demonstrate example input, then specify the desired output (expressed as a voltage) for the given input. The current version of the PICL provides two learning modes, binary classification and linear regression. To streamline training and also make it possible to train on highly transient signals (such as those produced by a camera flash or a hand clap), the PICL includes a number of input inferencing techniques. These techniques make it possible for the PICL to learn with as few as one example. The PICL's behavioural repertoire can be expanded by means of various output adapters, which serve to transform the output in useful ways when prototyping. Collectively, the PICL's capabilities allow users of systems such as the Arduino or littleBits electronics kit to quickly add basic sensor-based behaviour, with little or no programming required.
Midas: fabricating custom capacitive touch sensors to prototype interactive objects BIBAFull-Text 579-588
  Valkyrie Savage; Xiaohan Zhang; Björn Hartmann
An increasing number of consumer products include user interfaces that rely on touch input. While digital fabrication techniques such as 3D printing make it easier to prototype the shape of custom devices, adding interactivity to such prototypes remains a challenge for many designers. We introduce Midas, a software and hardware toolkit to support the design, fabrication, and programming of flexible capacitive touch sensors for interactive objects. With Midas, designers first define the desired shape, layout, and type of touch sensitive areas, as well as routing obstacles, in a sensor editor. From this high-level specification, Midas automatically generates layout files with appropriate sensor pads and routed connections. These files are then used to fabricate sensors using digital fabrication processes, e.g., vinyl cutters and conductive ink printers. Using step-by-step assembly instructions generated by Midas, designers connect these sensors to the Midas microcontroller, which detects touch events. Once the prototype is assembled, designers can define interactivity for their sensors: Midas supports both record-and-replay actions for controlling existing local applications and WebSocket-based event output for controlling novel or remote applications. In a first-use study with three participants, users successfully prototyped media players. We also demonstrate how Midas can be used to create a number of touch-sensitive interfaces.
Printed optics: 3D printing of embedded optical elements for interactive devices BIBAFull-Text 589-598
  Karl Willis; Eric Brockmeyer; Scott Hudson; Ivan Poupyrev
We present an approach to 3D printing custom optical elements for interactive devices labelled Printed Optics. Printed Optics enable sensing, display, and illumination elements to be directly embedded in the casing or mechanical structure of an interactive device. Using these elements, unique display surfaces, novel illumination techniques, custom optical sensors, and embedded optoelectronic components can be digitally fabricated for rapid, high fidelity, highly customized interactive devices. Printed Optics is part of our long term vision for interactive devices that are 3D printed in their entirety. In this paper we explore the possibilities for this vision afforded by fabrication of custom optical elements using today's 3D printing technology.
Interactive construction: interactive fabrication of functional mechanical devices BIBAFull-Text 599-606
  Stefanie Mueller; Pedro Lopes; Patrick Baudisch
Personal fabrication tools, such as laser cutters and 3D printers allow users to create precise objects quickly. However, working through a CAD system removes users from the workpiece. Recent interactive fabrication tools reintroduce this directness, but at the expense of precision.
   In this paper, we introduce constructable, an interactive drafting table that produces precise physical output in every step. Users interact by drafting directly on the workpiece using a hand-held laser pointer. The system tracks the pointer, beautifies its path, and implements its effect by cutting the workpiece using a fast high-powered laser cutter.
   Constructable achieves precision through tool-specific constraints, user-defined sketch lines, and by using the laser cutter itself for all visual feedback, rather than using a screen or projection. We demonstrate how constructable allows creating simple but functional devices, including a simple gearbox, that cannot be created with traditional interactive fabrication tools.