Human motion capture through a novel wearable multi-linkage sensory system

Jen-Yuan (James) Chang, Distinguished Professor at the Department of Power Mechanical Engineering, National Tsing Hua University, Taiwan, lifts the lid on human motion capture through a novel wearable multi-linkage sensory system

With the increasing demand for human-computer interaction in recent years, such as VR/AR, the medical industry, sport science measurement and film/entertainment industry, for instance, the development of motion tracking of the multi-linkage system has gradually attracted the attention not only from academic community but also industry sectors. The reason behind the increased attention is that the multi-linkage sensory system applies to posture measurement of the human body, as well as robotic arm applications. This so-called multi-linkage sensory system generally refers to a system consisting of multiple rigid bodies which are connected or related to each other. Each linkage is treated as a rigid body that can move in a 3-dimensional space in a non-detached manner in connection with other linkages, where each linkage maintains the property of rigidity. For example, the index finger is composed of three linkages connected through the finger joints to the palm.

Traditionally, devices to capture the object’s posture mainly rely on visually-based recognition. This measurement method uses the data captured from multiple calibrated cameras to obtain overlapping images, and the images are used to calculate the position and the posture of objects through the triangulation methods. Although the technology is mature and reliable, the cost of the devices is not economical and its application is limited by the measurement site and often encounters light-shielding problems.

Another inevitable problem is that the optical measurement method has the problem of line-of-sight shielding, that is being blocked by other objects. Once such a phenomenon occurs, the measured result will be inaccurate and most likely impossible to measure. This phenomenon usually appears while measuring complex and compact objects, such as the motion of the hand. For this specific reason, how to design wearable multi-linkage measurement system which can avoid the light-shielding, line-of-sight shielding and ambient light interference has also become an important issue in practical industrial applications.

Novel wearable multi-linkage measurement system

This article discusses the patented work (U.S. Patent No. 10,299,731; Taiwan Patent No. I612276) of a novel wearable multi-linkage measurement system developed by Professor Chang’s group in National Tsing Hua University, of which work was funded by Ministry of Science and Technology, Taiwan. This object pose measurement system was based on inertial measurement units (IMU).

In each IMU, three types of sensors, such as an accelerometer, a magnetometer, and a gyroscope, are included and employed to measure the gravity, geomagnetic direction and the rotational speed of the object. With the development of microelectromechanical systems (MEMS) manufacture technique, the IMU sensor ship can be downsized as small as fingernails, suitable for creating a low-cost, highly wearable and even a portable motion capture system!

IMU sensors

As illustrated in Figure 1(c), to capture all the motion of the human hand, 16 IMU sensors are required to record all the finger linkage’s motion of the hand. To be able to use so many sensors in the limited and crowded space on the hand to capture hand movements without affecting the operator’s hand movements, all of the components should be designed as small as they can.

As shown in Figure 1(a) when compared with the little finger and a penny, the printed circuit board with an IMU sensor chip is made only 1.2 mm thick and 8 mm wide. Each IMU sensor chipboard is secured in a 3D printed finger sleeve as shown in Figure 1(b) which is later worn on each finger linkage. Each IMU sensor board on the finger linkage is connected by cables to adjacent one via the finger joint. Based on adjacent IMU sensor’s acceleration, velocity and geomagnetic direction readouts, the shared finger joint’s angular displacement, velocity and acceleration can then be calculated in real-time.

To demonstrate the system capability in obtaining dexterous motion, the IMU sensor array is worn by a human hand and angles for the finger’s metacarpophalangeal joints (MCP) and distal/ proximal interphalangeal (DIP/PIP) joints are obtained in real-time as shown in Figure 2(b). At the five depicted time instances, the hand’s gestures as shown in Figure 2(a) can be real-time captured by the system. The measured information by the sensor array system can then be used to render a virtual or digital representation of the corresponding hand gesture with a high level of accuracy.

As opposed to the conventional approach, the system validated by calibrated experiments indicates the inertial measurement system and this method can be used in monitoring hand and multi-linkage motions, calculating the angles of the joints, and even identifying gestures or motion with a satisfactory level of accuracy. The captured and processed information from the system can be further used to create the corresponding digital twin in the cyber-physical system. Not only human skills can be recorded by the system and digitally transformed when applied with artificial intelligence (AI), but the digitised skills can also be further evolved in the digital twin for endless applications.

Please note: This is a commercial profile

Contributor Profile

Read Profile

Jen-Yuan (James) Chang

Distinguished Professor
National Tsing Hua University, Taiwan
Phone: +886 3 574 2498
Email: jychang@pme.nthu.edu.tw
Website: Visit Website