My investigation into robot dexterity has focused on seeking a coordinated understanding of computational and control issues in manipulation tasks. One objective of this study is dynamic retrieval of geometric information such as shape and pose, and of mechanical information such as motion and force. Another objective is careful engineering of the above information to make the robot exhibit skills during its execution of physical tasks. Through efforts balanced between theoretical inquiry and experimental demonstration, I hope to gain in-depth knowledge about action and intelligence as they interact with each other. From an application point of view, such knowledge will have the promise of making impact on not only industrial automation but also personal robots in the future.

For the robot to eventually demonstrate skills approaching the human level, close integration of sensing into action must be achieved to pave the way for a framework under which skills of purposeful object handling can be formulated, analyzed, and automated. To demonstrate the above philosophy, I have made primary use of one source of information which is ubiquitous in the physical world --- contact between two or more bodies. This can be either the location or force of contact or both. I am particularly interested in contact between a robotic hand (the term ``robotic hand'' refers to a manipulator in general, whether or not shaped similar to the human hand) and an object being manipulated by the hand.

Under the support from an NSF CAREER Award (2002-2007), my investigation has so far centered on the localization, recognition, reconstruction, as well as grasping of shapes (in particular, curved shapes) which I think are fundamental for achieving skillful and intelligent object handling.

In the near future (until 2007), my main focus will be on deepening the above investigation, with a shift of attention to 3D curved shapes. More details down this line of research are described in the end of Parts 1 and 2 below.

In the near future, my research will branch out to haptics, dexterous manipulation, and human-robot interaction. A plan for these efforts will follow in Part 5.

**Tactile shape recognition using differential invariants.**Ph.D. student Rinat Ibrayev and I have been studying how to recognize shapes bounded by low-degree polynomial curves using minimal tactile data. The problem generalizes traditional model-based recognition in the sense that each model now is not just a specific shape but rather a family of a continuum of parametric shapes.

Differential and semi-differential invariants have the advantage of requiring local data (which are yielded by touch sensors) only. The invariants we have derived are independent of not only translation and rotation (as invariants used in computer vision) but also boundary points at which they are evaluated. In theory, at most three such points are needed for quadratic curves and cubic spline curves.

These invariants allow us to discriminate one family of curves from another, and determine the real shape out of the recognized family. Furthermore, contact locations at which the tactile data were obtained can also be estimated, hence the relative placement of the finger on the shape becomes known. Therefore the invariant-based approach has the potential of unifying shape recognition, recovery, and localization nicely as the human hand does subconsciously everyday.

We have conducted preliminary experiments with real tactile data to support the validity of this approach. A stable method is devised for estimating curvature and its derivative.

Preliminary work on this topic was reported at the 2004 IEEE International Conference on Robotics and Automation (ICRA). A more complete version (with some experimental results) was presented at the 2004 International Workshop on Algorithmic Foundations of Robotics (WAFR), and has been selected for a special issue of the International Journal of Robotics Research (IJRR) in 2005.

*Future extensions*: We will extend the design of invariants to more general curves used in applications, and construct a recognition tree for 2D curved shapes that can be searched with a query generated over tactile data. Then we will move on to investigate invariant-based recognition of curved shapes in 3D (which is a potential Ph.D. thesis topic for Rinat).**Parts localization and reconstruction with touch sensors**Inspired by the human hand's ability to determine its placement on a familiar object through finger tracing and fumbling, I have demonstrated how to localize a 2D curved shape by the rolling of a jaw equipped with a touch sensor. The measurable information includes how much the jaw has turned and how far it has moved on the shape.

I designed a numerical algorithm that computes the configuration of the jaw after the rolling motion. The algorithm is complete and the amount of numerical computation is asymptotically optimal. Localization was demonstrated through experiments on a robot. Instead of using industrial force/torque sensors (which are expensive and sometimes overkills for lab experiments), I implemented a 3-axis force/torque sensor that can detect contact location. Progressive results on the localization work were presented at ICRA 2000 and the 2001 and 2003 IEEE/RSJ International Conferences on Intelligent Robots and Systems (IROS). A summary has been conditionally accepted to IEEE Transactions on Robotics.

Meanwhile, Ph.D. student Liangchuan Mi has shown how a simple touch sensor like a joystick can be used to reconstruct planar curved shapes with hardly any loss of shape accuracy. Despite the joystick's limited force sensing, it can generate precise contact measurements by taking advantage of an Adept robot's high positional accuracy. Liangchuan came up with a very effective position control strategy that predicts the movement direction in the next step based on the current force reading and a polynomial fit to local tracking history. This work will be presented at IROS 2004 in Japan.

*Future extensions*: Localization of an object relative to the hand by touch will change the paradigm of grasping in robotics where currently sensing is often conducted in advance by a vision system or by sensors detached from the hand. Further investigation on the interpretation of "feeling" by touch, I believe, will make possible close integration between sensing and control. The next step will be to move on to study grasping strategies that are feedback driven and more robust to errors and uncertainties. Meanwhile, the shape tracking work will be extended to the reconstruction of 3D surfaces, which I hope Liangchuan will investigate with me in his Ph.D. thesis work.**Curve Computation and Finding Antipodal Grasps.**Finding geometric substructures related to curves (such as common tangents and antipodal points) may be formulated as traditional nonlinear programming. But such a solution would be neither complete nor efficient due to the inherent local nature of nonlinear programming methods. I have demonstrated that, in two dimensions, computational efficiency and (almost) completeness can be achieved by exploiting both global and differential geometry.

I introduce a curve processing scheme that dissects a curve into monotone segments (based on some task-specific criteria) and then couples marching with numerical bisection on these segments to search for the desired substructures. To demonstrate this idea, I present an efficient algorithm that computes, up to numerical resolution, all pairs of antipodal points on a planar curved shape. These points are used for achieving stable grasps of the shape. The algorithm makes use of new insights into the differential geometry at two antipodal points, and employs a subroutine to construct all common tangents of two curve segments. The numerical convergence rate and running time of the algorithm have been determined.

The work represents an advance on computation involving parametric curves, and also gives a very satisfactory solution to one of the well-known problems in robot grasping. It is described in a 31-page IJRR paper in 2004.

In addition, I also applied the above curve computation scheme to construct convex hulls for closed parametric plane curves. This work is summarized in a 51-page submission to Computational Geometry: Theory and Applications.

**Geometric and Dynamic Robot Sensing (Ph.D. thesis)**My thesis work investigated geometric and mechanical sensing strategies for objects of known shapes (which include industrial parts and everyday desktop items). I first introduced two strategies that use simple geometric constraints to either immobilize the object or to distinguish its real pose from a finite number of apparent poses. Computational complexity issues are examined. Then I developed a sensing strategy called "pose and motion from contact." By applying nonlinear observability theory, I demonstrated that essential task information is often hidden in mechanical interactions, and showed how this information can be properly revealed. Nonlinear observers were designed for the purpose.

The thesis work was performed at Carnegie Mellon University. Most of the results were published in two IJRR papers (28 pages and 25 pages, respectively) in 1996 and 1999. An independent result (on the local observability of a rolling 3D object on a palm imbued with tactile sensors) was presented at WAFR in 1998.

**Future Research Plan**Besides continuing efforts on tactile shape recognition and reconstruction as outlined in Parts 1 and 2, my future research agenda will also include haptics, dexterous manipulation, and human-robot interaction.

One of the open issues in haptics is to deal with response delays which often cause the instability of a haptic system. This is primarily due to the lack of a reliable model of the physical environment. Meanwhile, much needs to be investigated in haptic rendering on the use of measurements on real environments to construct models of physical shapes, stiffness, texture, etc. To speed up the response, tactile measurements may be combined with a priori knowledge to construct a rough model of the (local) environment. Such a model provides information like shape, stiffness, etc. It will then get refined during the interactions between the servant and the environment under guidance of the user. With multiple users interacting with the same environment, fusion of tactile data and path planning are also among interesting research topics.

In dexterous manipulation, I would like to tackle the problem of manipulating an object while exploring its geometric and physical properties, with or without vision. Applications include tasks with high degrees of complexity, such as robotic and robot-assisted surgeries, space exploration, and household robotics (tasks from replacing a lightbulb to cleaning up the dishes on the dinner table). The problem iteself is a representative domain where several research areas in robotics come together: dynamics and control of the hand; observability of the object; grasping under dynamical constraints; planning of manipulation trajectory; recovery/estimation of shape, textures, stiffness, etc.; and sensor fusion (e.g., in the presence of multiple touch sensors or a vision system). Existing work has mostly concentrated on one of the above areas, sometimes under restricted assumptions, so that the results are not general enough to be applicable in real tasks with reasonable degrees of complexity. Clearly, efforts are needed to understand their roles and interactions and to characterize the results in a unified framework in order for there to be a major breakthrough in the field. Such understanding and its experimental verification will be necessary before the advent of general purpose robots. I am aware that this research may turn out to be a long term scientific endeavor but devoted efforts will definitely be worthwhile.

Another interesting area I hope to explore in the future is human-robot interaction. How to make the robot learn manipulation skills when coached by a human? Can a robot and a human physically participate in a game such as poker the way two humans do (dealing cards, mixing cards, etc.)? Learning is expected to play an important role except the skills to be enhanced are in the form of control algorithms dealing primarily with mechanics. Based on the interactions, the human coach should always be able to accelerate the "learning" process by reprogramming refined knowledge into the robot.

I envision my lab to grow into a research group with at least five graduate students and a couple of undergraduate research assistants, in the next few years. Three or four years later, the group is expected to yield at least one Ph.D. every other year. I will work hard to maintain the flow of graduate students and the intellectual output of the lab. Sufficient grant support and an aggressive student recruiting strategy are both vital for achieving the above goals. With these efforts, I will strive to build the lab into a visible robotics place heavily oriented to geometry, touch, and robot dexterity.