Multi-view Multi-person Top-down SMPL Estimator¶
Overview
Arguments
Run
Step0: estimate perception2d
Step1: establish cross-frame and cross-person associations
Step2: estimate keypoints3d
Step3: estimate smpl
Example
Overview¶
This tool takes multi-view RGB sequences and multi-view calibrated camera parameters as input. By a simple call of run()
, it outputs triangulated keypoints3d and SMPL parameters for the multi-person in the multi-view scene.
Arguments¶
output_dir:
output_dir
is the path to the directory saving all possible output files, including keypoints3d, SMPLData and visualization videos.estimator_config:
estimator_config
is the path to aMultiViewMultiPersonTopDownEstimator
config file, wherebbox_detector
,kps2d_estimator
,associator
,triangulator
andsmplify
are necessary. Every element ofpoint_selectors
are configs for point selector defined inxrmocap/ops/triangulation/point_selection
.kps3d_optimizers
is a list of kps3d_optimizer, defined inxrmocap/transform/keypoints3d/optim
. When inferring images stored on disk, setload_batch_size
to a reasonable value will prevent your machine from out of memory. For more details, see config and the docstring in code.image_and_camera_param:
image_and_camera_param
is a text file contains the image path and the corresponding camera parameters. Line 0 is the image path of the first view, and line 1 is the corresponding camera parameter path. Line 2 is the image path of the second view, and line 3 is the corresponding camera parameter path, and so on.
xrmocap_data/Shelf_50/Shelf/Camera0/
xrmocap_data/Shelf_50/xrmocap_meta_testset_small/scene_0/camera_parameters/fisheye_param_00.json
xrmocap_data/Shelf_50/Shelf/Camera1/
xrmocap_data/Shelf_50/xrmocap_meta_testset_small/scene_0/camera_parameters/fisheye_param_01.json
start_frame:
start_frame
is the index of the start frame.end_frame:
end_frame
is the index of the end frame.enable_log_file By default, enable_log_file is False and the tool will only print log to console. Add
--enable_log_file
makes it True and a log file named{smc_file_name}_{time_str}.txt
will be written.disable_visualization By default, disable_visualization is False and the tool will visualize keypoints3d and SMPLData with an orbit camera, overlay SMPL meshes on one view.
Run¶
Inside run()
, there are three major steps of estimation, and details of each step are shown in the diagram below.
Step0: estimate perception2d¶
In this step, we perform a top-down keypoints2d estimation, detect bbox2d by bbox_detector
, and detect keypoints2d in every bbox by kps2d_estimator
. You can choose the model and weight you like by modifying the config file.
Step1: establish cross-frame and cross-person associations¶
In this step, we match the keypoints2d across views by associator
and add temporal tracking and filtering. For recommended configs on associator
, you can check out the README.md
Step2: estimate keypoints3d¶
In this step, we split the estimation into three sub-steps: point selection, triangulation and optimization. Every sub-step can be skipped by passing None
in config except triangulation. We use cascaded point selectors in point_selectors
to select 2D points from well-calibrated views, for triangulation. After multi-view triangulation, in the third sub-step, we use cascaded keypoints3d optimizers in kps3d_optimizers
to optimize keypoints3d.
Step3: estimate smpl¶
In this step, we estimate SMPL parameters from keypoints3d. For details of smpl fitting, see smplify doc.
Example¶
python tools/mview_mperson_topdown_estimator.py \
--image_and_camera_param 'data/image_and_camera_param.txt' \
--start_frame 0 \
--end_frame 10 \
--enable_log_file