Research Scientist

Soyong Shin

Founding Research Scientist @ Nuance Labs
Previously Ph.D., Carnegie Mellon University · Visiting Researcher, Meta FAIR

I build human-like AI — systems that perceive, reason, and behave the way people do. My work centers on multimodal models that read human nuance and emotion, and generative models that let AI avatars react, speak, and express themselves like people — in real time. I earned my Ph.D. at Carnegie Mellon, where my dissertation on markerless 3D human motion tracking was recognized with the Best Dissertation Award.

Multimodal LLMs Video Generation Generative AI Human Motion & 3D Vision
Google Scholar GitHub LinkedIn Curriculum Vitae Email
Soyong Shin

Experience

Professional

2026.01 — Present
Nuance Labs
Founding Research Scientist
2024.09 — 2025.12
Meta FAIR
Visiting Researcher · Perception Team
2024.05 — 2024.08
Meta Reality Labs
Research Scientist Intern
2023.05 — 2025.11
Max Planck Institute for Intelligent Systems
Guest Scientist

Education

2026
Ph.D., Mechanical Engineering
Carnegie Mellon University
Dissertation: Markerless 3D Human Motion Tracking
2021
M.S., Mechanical Engineering
Carnegie Mellon University
2019
B.S., Mechanical Engineering
Seoul National University

News

May 2026
Graduated with a Ph.D. from Carnegie Mellon University; awarded the Best Dissertation Award.
2026
Four papers accepted to CVPR 2026 (SAM 3D Body, DuoMo, MAMMA, GRIP) and two to 3DV 2026 (BodyContact4D, EgoMDM).
Jan 2026
Joined Nuance Labs as a Founding Research Scientist, working on human-like and generative AI.
Nov 2025
Our work SAM 3D Body was released.

Publications

Selected · author-ordered newest first
2026SAM 3D Body
CVPR 2026

SAM 3D Body: Robust Full-Body Human Mesh Recovery

Xitong Yang, Devansh Kukreja, Don Pinkus, Anushka Sagar, Taosha Fan, Jinhyung Park, Soyong Shin, Jinkun Cao, Jiawei Liu, Nicolas Ugrinovic, Matt Feiszli, Jitendra Malik, Piotr Dollár, Kris Kitani
Paper Project
2026DuoMo
CVPR 2026

DuoMo: Dual Motion Diffusion for World-Space Human Reconstruction

Yufu Wang, Evonne Ng, Soyong Shin, Rawal Khirodkar, Yuan Dong, Zhaoen Su, Jinhyung Park, Kris Kitani, Alexander Richard, Fabian Prada, Michael Zollhöfer
arXiv Project Code
2026Ground Reaction Inertial Poser
CVPR 2026

Ground Reaction Inertial Poser: Physics-based Human Motion Capture from Sparse IMUs and Insole Pressure Sensors

Ryosuke Hori, Jyun-Ting Song, Zhengyi Luo, Jinkun Cao, Soyong Shin, Hideo Saito, Kris Kitani
arXiv Project
2026MAMMA
CVPR 2026

MAMMA: Markerless & Automatic Multi-Person Motion Action Capture

Hanz Cuevas-Velasquez, Anastasios Yiannakidis, Soyong Shin, Giorgio Becherini, Markus Höschle, Joachim Tesch, Taylor Obersat, Tsvetelina Alexiadis, Michael J. Black
arXiv
2026BodyContact4D
3DV 2026

BodyContact4D: A Multi-view Video Dataset for Understanding Human and Environment Interactions

Soyong Shin, Chaeeun Lee, Holly Chen, Jyun-Ting Song, Eni Halilaj, Kris Kitani
2026EgoMDM
3DV 2026

EgoMDM: Diffusion-based Human Motion Synthesis from Sparse Egocentric Sensors

Soyong Shin, Anuj Pahuja, Alexander Richard, Kris Kitani, Jason Saragih, Yuhua Chen, Weipeng Xu, Eni Halilaj, Timur Bagautdinov
2025I-MOVE-23
IEEE Trans. Biomedical Engineering

Impact of Multi-View Fusion and Biomechanical Modeling on Markerless Motion Tracking

Zhixiong Li, Soyong Shin, Vu Phan, Evy Meinders, Eni Halilaj
Paper Project
2024WHAM
CVPR 2024

WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion

Soyong Shin, Juyong Kim, Eni Halilaj, Michael J. Black
arXiv Project Code
2023From Skin to Skeleton
ACM TOG (SIGGRAPH Asia) Honorable Mention

From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans

Marilyn Keller, Keenon Werling, Soyong Shin, Scott Delp, Sergi Pujades, C. Karen Liu, Michael J. Black
Paper Project
2023Markerless Motion Tracking
IEEE Trans. Biomedical Engineering

Markerless Motion Tracking with Noisy Video and IMU Data

Soyong Shin, Zhixiong Li, Eni Halilaj
Paper Code
2021IMU Kinematics Estimation
Journal of Biomechanics

Estimation of Kinematics from Inertial Measurement Units Using a Combined Deep Learning and Optimization Framework

Eric Rapp*, Soyong Shin*, Wolf Thomsen, Reed Ferber, Eni Halilaj (*equal contribution)
Paper Code Project
2021Portable Biomechanics Labs
Journal of Biomechanics

Toward Portable and Modular Biomechanics Labs: How Video and IMU Fusion Will Change Gait Analysis

Eni Halilaj, Soyong Shin, Eric Rapp, Donglai Xiang
Paper Code
2020Multi-view HMR
arXiv

Multi-view Human Pose and Shape Estimation Using Learnable Volumetric Aggregation

Soyong Shin, Eni Halilaj
arXiv
2020Multimodal Dataset
American Society of Biomechanics

A Multimodal Dataset for Modeling Human Pose Priors

Eni Halilaj, Soyong Shin, Eric Rapp, Donglai Xiang, Yaadhav Raj
Abstract Project

Additional Publications

Fusion of Video and Inertial Sensing Data via Dynamic Optimization of a Biomechanical Model — O. Pearl, S. Shin, A. Godura, S. Bergbreiter, E. Halilaj. Journal of Biomechanics, 2023.
Computational Analysis of the Performance of a Vertical Axis Turbine in a Water Pipe — H. Yeo, W. Seok, S. Shin, Y.C. Huh, B.C. Jung, C.S. Myung, S.H. Rhee. Energies, 2019.

Honors & Awards

Best Dissertation Award
Dept. of Mechanical Engineering, Carnegie Mellon University · 2026
Pre-doctoral Achievement Award
American Society of Biomechanics (ASB) · 2025
Meta AI Mentorship Program
Meta AI — 1 of 2 recipients · 2024
CoE Dean's Fellowship
CMU College of Engineering · 2024
CMLH Fellowship in Digital Health
CMU School of Computer Science — 1 of 10 · 2022
Nick-G. Vlahakis Fellowship
CMU College of Engineering — 1 of 2 · 2021
Korean Government Scholarship for Study Overseas
Korean National Institute for International Education · 2020
Minister Prize — 1st, Korean AI Grand Challenge
Korean Ministry of Science and ICT (OCR task) · 2019