Multi-view Single-person SMPL Estimator

  • Overview

  • Arguments

  • Run

    • Step0: estimate_keypoints2d

    • Step1: estimate_keypoints3d

    • Step2: estimate_smpl

  • Example

Overview

This tool takes multi-view videos and multi-view calibrated camera parameters as input. By a simple call of run(), it outputs detected keypoints2d, triangulated keypoints3d and SMPL parameters for the single person in the multi-view scene.

Arguments

To construct an estimator instance of this class, you will need a config file like config/estimation/mview_sperson_smpl_estimator.py. work_dir makes no sense in this estimator, could be any value, while bbox_detector, kps2d_estimator, triangulator and smplify are necessary. triangulator shall be a config for triangulator defined in xrmocap/ops/triangulation, instead of xrprimer triangulator. smplify can be a config for either class SMPLify or class SMPLifyX. cam_pre_selector , cam_selector and every list element of final_selectors are configs for point selector defined in xrmocap/ops/triangulation/point_selection. kps3d_optimizers is a list of kps3d_optimizer, defined in xrmocap/transform/keypoints3d/optim. When inferring images stored on disk, set load_batch_size to a reasonable value will prevent your machine from out of memory.

For more details, see the docstring in code.

Run

Inside run(), there are three major steps of estimation, and details of each step are shown in the diagram below.

Step0: estimate keypoints2d

In this step, we perform a top-down keypoints2d estimation, detect bbox2d by bbox_detector, and detect keypoints2d in every bbox by kps2d_estimator. You can choose the model and weight you like by modifying the config file.

Step1: estimate keypoints3d

In this step, we split the estimation into four sub-steps: camera selection, point selection, triangulation and optimization. Every sub-step can be skipped by passing None in config except triangulation. First, we use cam_pre_selector to select good 2D points from all detected keypoints2d, and select well-calibrated cameras by cam_selector. Second, we use cascaded point selectors in final_selectors to select 2D points from well-calibrated views, for triangulation. After multi-view triangulation, in the forth sub-step, we use cascaded keypoints3d optimizers in kps3d_optimizers to optimize keypoints3d, and the result of optimization will be the return value of step estimate_keypoints3d.

Step2: estimate SMPL

In this step, we estimate SMPL or SMPLX parameters from keypoints3d of last step. For details of smpl fitting, see smplify doc.

Example

import numpy as np

from xrmocap.estimation.builder import build_estimator
from xrprimer.data_structure.camera import FisheyeCameraParameter


# multi-view camera parameter list
cam_param_list = [FisheyeCameraParameter.fromfile(
    f'fisheye_param_{idx:02d}.json') for idx in range(10)]
# multi-view image array
mview_img_arr = np.zeros(shape=(10, 150, 1080, 1920, 3), dtype=np.uint8)
# build an estimator
mview_sperson_smpl_estimator = build_estimator(estimator_config)

# run the estimator on image array
keypoints2d_list, keypoints3d, smpl_data = mview_sperson_smpl_estimator.run(
    cam_param=cam_param_list, img_arr=mview_img_arr)