xrmocap.core

estimation

evaluation

smplify hook

train

visualization

xrmocap.data

data_converter

dataloader

dataset

data_visualization

xrmocap.data_structure

class xrmocap.data_structure.Keypoints(src_dict: Optional[dict] = None, dtype: Literal[torch, numpy, auto] = 'auto', kps: Optional[Union[numpy.ndarray, torch.Tensor]] = None, mask: Optional[Union[numpy.ndarray, torch.Tensor]] = None, convention: Optional[str] = None, logger: Union[None, str, logging.Logger] = None)

deprecation_warned = False

A class for multi-frame, multi-person keypoints data, based on python dict.

keypoints, mask and convention are the three necessary keys, and we advise you to just call Keypoints(). If you’d like to set them manually, it is recommended to obey the following turn: convention -> keypoints -> mask.

class xrmocap.data_structure.Limbs(connections: Union[numpy.ndarray, torch.Tensor], connection_names: Optional[List[str]] = None, parts: Optional[List[List[int]]] = None, part_names: Optional[List[str]] = None, points: Optional[Union[numpy.ndarray, torch.Tensor]] = None, logger: Union[None, str, logging.Logger] = None)

deprecation_warned = False

A class for person limbs data, recording connection vectors between keypoints.

Connections are the only necessary data, while human parts, points are optional.

class xrmocap.data_structure.SMPLData(gender: Optional[Literal[female, male, neutral]] = None, fullpose: Optional[Union[numpy.ndarray, torch.Tensor]] = None, transl: Optional[Union[numpy.ndarray, torch.Tensor]] = None, betas: Optional[Union[numpy.ndarray, torch.Tensor]] = None, mask: Optional[Union[numpy.ndarray, torch.Tensor]] = None, logger: Union[None, str, logging.Logger] = None)

dump(npz_path: str, overwrite: bool = True)

Dump keys and items to an npz file.

Parameters

• npz_path (str) – Path to a dumped npz file.

• overwrite (bool, optional) – Whether to overwrite if there is already a file. Defaults to True.

Raises

• ValueError – npz_path does not end with ‘.npz’.

• FileExistsError – When overwrite is False and file exists.

classmethod from_dict(smpl_data_dict: Union[xrmocap.data_structure.body_model.smpl_data.SMPLData, dict]) → xrmocap.data_structure.body_model.smpl_data.SMPLData

Construct a body model data structure from a SMPLData, or a degraded smpl_data in dict type.

Parameters

smpl_data_dict (dict) – A degraded smpl_data in dict type.

Returns

SMPLData – A SMPLData instance load from dict.

from_param_dict(smpl_dict: dict) → None

Load SMPL parameters from smpl_dict, which is the output of a body model in most cases.

Parameters

smpl_dict (dict) – A dict of ndarray|Tensor parameters. global_orient and body_pose are necessary, transl and betas are optional. Other keys are ignored.

Raises

KeyError – missing necessary keys.

classmethod fromfile(npz_path: str) → xrmocap.data_structure.body_model.smpl_data.SMPLData

Construct a body model data structure from an npz file.

Parameters

npz_path (str) – Path to a dumped npz file.

Returns

SMPLData – A SMPLData instance load from file.

get_batch_size() → int

Get batch size.

Returns

int – batch size of fullpose.

get_betas(repeat_betas: bool = True) → numpy.ndarray

Get betas.

Parameters

repeat_betas (bool, optional) – Whether to repeat betas when its first dim doesn’t match batch_size. Defaults to True.

Returns

ndarray – betas in shape [batch_size, betas_dims] or [1, betas_dims].

get_body_pose() → numpy.ndarray

Get body_pose.

Returns

ndarray – body_pose in shape [batch_size, BODY_POSE_LEN, 3].

get_fullpose() → numpy.ndarray

Get fullpose.

Returns

ndarray – fullpose in shape [batch_size, fullpose_dim, 3].

classmethod get_fullpose_dim() → int

Get dimension of full pose.

Returns

int – Dim value. Full pose shall be in shape (frame_n, dim, 3)

get_gender() → str

Get gender.

Returns

str – gender in [‘neutral’, ‘female’, ‘male’].

get_global_orient() → numpy.ndarray

Get global_orient.

Returns

ndarray – global_orient in shape [batch_size, 3].

get_mask() → numpy.ndarray

Get mask.

Returns

ndarray – mask in shape [batch_size, ].

get_transl() → numpy.ndarray

Get translation.

Returns

ndarray – translation in shape [batch_size, 3].

load(npz_path: str)

Load data from npz_path and update them to self.

Parameters

npz_path (str) – Path to a dumped npz file.

set_betas(betas: Union[numpy.ndarray, torch.Tensor]) → None

Set betas data.

Parameters

betas (Union[np.ndarray, torch.Tensor]) – Body shape parameters in ndarray or tensor, in shape [batch_size, n]. n stands for any positive int, typically it’s 10.

Raises

TypeError – Type of betas is not correct.

set_fullpose(fullpose: Union[numpy.ndarray, torch.Tensor]) → None

Set full pose data.

Parameters

fullpose (Union[np.ndarray, torch.Tensor]) – Full pose in ndarray or tensor, in shape [batch_size, fullpose_dim, 3]. global_orient at [:, 0, :].

Raises

TypeError – Type of fullpose is not correct.

set_gender(gender: Literal[female, male, neutral] = 'neutral') → None

Set gender.

Parameters

gender (Literal["female", "male", "neutral"], optional) – Gender of the body model. Should be one among [“female”, “male”, “neutral”]. Defaults to ‘neutral’.

Raises

ValueError – Value of gender is not correct.

set_mask(mask: Union[numpy.ndarray, torch.Tensor]) → None

Set framewise mask data.

Parameters

mask (Union[np.ndarray, torch.Tensor]) – Visibility mask in ndarray or tensor, in shape [batch_size, ].

Raises

TypeError – Type of mask is not correct.

set_transl(transl: Union[numpy.ndarray, torch.Tensor]) → None

Set translation data.

Parameters

transl (Union[np.ndarray, torch.Tensor]) – Translation in ndarray or tensor, in shape [batch_size, 3].

Raises

TypeError – Type of transl is not correct.

to_param_dict(repeat_betas: bool = True) → dict

Split fullpose into global_orient and body_pose, return all the necessary parameters in one dict.

Parameters

repeat_betas (bool, optional) – Whether to repeat betas when its first dim doesn’t match batch_size. Defaults to True.

Returns

dict – A dict of SMPL data, whose keys are betas, body_pose, global_orient and transl.

to_tensor_dict(repeat_betas: bool = True, device: Union[torch.device, str] = 'cpu') → dict

It is almost same as self.to_param_dict, but all the values are tensors in a specified device. Split fullpose into global_orient and body_pose, return all the necessary parameters in one dict.

Parameters

• repeat_betas (bool, optional) – Whether to repeat betas when its first dim doesn’t match batch_size. Defaults to True.

• device (Union[torch.device, str], optional) – A specified device. Defaults to CPU_DEVICE. Defaults to ‘cpu’.

Returns

dict – A dict of SMPL data, whose keys are betas, body_pose, global_orient and transl.

class xrmocap.data_structure.SMPLXDData(gender: Optional[Literal[female, male, neutral]] = None, fullpose: Optional[Union[numpy.ndarray, torch.Tensor]] = None, transl: Optional[Union[numpy.ndarray, torch.Tensor]] = None, betas: Optional[Union[numpy.ndarray, torch.Tensor]] = None, expression: Optional[Union[numpy.ndarray, torch.Tensor]] = None, displacement: Optional[Union[numpy.ndarray, torch.Tensor]] = None, mask: Optional[Union[numpy.ndarray, torch.Tensor]] = None, logger: Union[None, str, logging.Logger] = None)

classmethod from_dict(smpl_data_dict: Union[xrmocap.data_structure.body_model.smplxd_data.SMPLXDData, dict]) → xrmocap.data_structure.body_model.smplxd_data.SMPLXDData

Construct a body model data structure from a SMPLXDData, or a degraded smplxd_data in dict type.

Parameters

smplxd_data_dict (dict) – A degraded smplxd_data in dict type.

Returns

SMPLXData – A SMPLXDData instance load from dict.

from_param_dict(smplxd_dict: dict) → None

Load SMPLX+D parameters from smplxd_dict, which is the output of a body model in most cases.

Parameters

smplx_dict (dict) – A dict of ndarray|Tensor parameters. global_orient and body_pose are necessary, jaw_pose, leye_pose, reye_pose, left_hand_pose, right_hand_pose, transl and betas are optional. Other keys are ignored.

Raises

KeyError – missing necessary keys.

get_displacement() → numpy.ndarray

Get displacement.

Returns

ndarray – Displacement in shape [batch_size, NUM_VERTS, 3].

set_displacement(displacement: Union[numpy.ndarray, torch.Tensor]) → None

Set displacement data.

Parameters

displacement (Union[np.ndarray, torch.Tensor]) – Displacement parameters in ndarray or tensor, in shape [batch_size, NUM_VERTS, 3].

Raises

TypeError – Type of displacement is not correct.

to_param_dict(repeat_betas: bool = True, repeat_expression: bool = True) → dict

Split fullpose into global_orient, body_pose, jaw_pose, leye_pose, reye_pose, left_hand_pose, right_hand_pose, return all the necessary parameters in one dict.

Parameters

• repeat_betas (bool, optional) – Whether to repeat betas when its first dim doesn’t match batch_size. Defaults to True.

• repeat_expression (bool, optional) – Whether to repeat expression when its first dim doesn’t match batch_size. Defaults to True.

Returns

dict – A dict of SMPLX data, whose keys are betas, global_orient, transl, global_orient, body_pose, jaw_pose, leye_pose, reye_pose, left_hand_pose, right_hand_pose, expression.

class xrmocap.data_structure.SMPLXData(gender: Optional[Literal[female, male, neutral]] = None, fullpose: Optional[Union[numpy.ndarray, torch.Tensor]] = None, transl: Optional[Union[numpy.ndarray, torch.Tensor]] = None, betas: Optional[Union[numpy.ndarray, torch.Tensor]] = None, expression: Optional[Union[numpy.ndarray, torch.Tensor]] = None, mask: Optional[Union[numpy.ndarray, torch.Tensor]] = None, logger: Union[None, str, logging.Logger] = None)

classmethod from_dict(smpl_data_dict: Union[xrmocap.data_structure.body_model.smplx_data.SMPLXData, dict]) → xrmocap.data_structure.body_model.smplx_data.SMPLXData

Construct a body model data structure from a SMPLXData, or a degraded smplx_data in dict type.

Parameters

smplx_data_dict (dict) – A degraded smplx_data in dict type.

Returns

SMPLXData – A SMPLXData instance load from dict.

from_param_dict(smplx_dict: dict) → None

Load SMPLX parameters from smplx_dict, which is the output of a body model in most cases.

Parameters

smplx_dict (dict) – A dict of ndarray|Tensor parameters. global_orient and body_pose are necessary, expression, jaw_pose, leye_pose, reye_pose, left_hand_pose, right_hand_pose, transl and betas are optional. Other keys are ignored.

Raises

KeyError – missing necessary keys.

get_expression(repeat_expression: bool = True) → numpy.ndarray

Get expression.

Parameters

repeat_expression (bool, optional) – Whether to repeat expression when its first dim doesn’t match batch_size. Defaults to True.

Returns

ndarray – expression in shape [batch_size, expression_dims] or [1, expression_dims].

classmethod get_fullpose_dim() → int

Get dimension of full pose.

Returns

int – Dim value. Full pose shall be in shape (frame_n, dim, 3)

set_expression(expression: Union[numpy.ndarray, torch.Tensor]) → None

Set expression data.

Parameters

expression (Union[np.ndarray, torch.Tensor]) – Expression parameters in ndarray or tensor, in shape [batch_size, n]. n stands for any positive int, typically it’s 10.

Raises

TypeError – Type of expression is not correct.

to_param_dict(repeat_betas: bool = True, repeat_expression: bool = True) → dict

Split fullpose into global_orient, body_pose, jaw_pose, leye_pose, reye_pose, left_hand_pose, right_hand_pose, return all the necessary parameters in one dict.

Parameters

• repeat_betas (bool, optional) – Whether to repeat betas when its first dim doesn’t match batch_size. Defaults to True.

• repeat_expression (bool, optional) – Whether to repeat expression when its first dim doesn’t match batch_size. Defaults to True.

Returns

dict – A dict of SMPLX data, whose keys are betas, global_orient, transl, global_orient, body_pose, jaw_pose, leye_pose, reye_pose, left_hand_pose, right_hand_pose, expression.

to_tensor_dict(repeat_betas: bool = True, repeat_expression: bool = True, device: Union[torch.device, str] = 'cpu') → dict

It is almost same as self.to_param_dict, but all the values are tensors in a specified device. Split fullpose into global_orient and body_pose, return all the necessary parameters in one dict.

Parameters

• repeat_betas (bool, optional) – Whether to repeat betas when its first dim doesn’t match batch_size. Defaults to True.

• repeat_expression (bool, optional) – Whether to repeat expression when its first dim doesn’t match batch_size. Defaults to True.

• device (Union[torch.device, str], optional) – A specified device. Defaults to CPU_DEVICE. Defaults to ‘cpu’.

Returns

dict – A dict of SMPLX data, whose keys are betas, body_pose, global_orient and transl, etc.

xrmocap.human_perception

bbox_detection

keypoints_estimation

xrmocap.io

xrmocap.io.get_all_color_kinect_parameter_from_smc(smc_reader: xrmocap.data_structure.smc_reader.SMCReader, align_floor: bool = True, logger: Union[None, str, logging.Logger] = None) → List[xrprimer.data_structure.camera.fisheye_camera.FisheyeCameraParameter]

Get an RGB FisheyeCameraParameter from an smc reader.

Parameters

• smc_reader (SMCReader) – An SmcReader instance containing kinect and iphone camera parameters.

• align_floor (bool) – Whether to rotate camera extrinsics, makes the xOz plane parallel to floor calibrated by Kinect_0’s depth camera. Defaults to True.

• logger (Union[None, str, logging.Logger], optional) – Logger for logging. If None, root logger will be selected. Defaults to None.

Returns

List[FisheyeCameraParameter]

xrmocap.io.get_color_camera_parameter_from_smc(smc_reader: xrmocap.data_structure.smc_reader.SMCReader, camera_type: Literal[kinect, iphone], camera_id: int, align_floor: bool = True, logger: Union[None, str, logging.Logger] = None) → xrprimer.data_structure.camera.fisheye_camera.FisheyeCameraParameter

Get an RGB FisheyeCameraParameter from an smc reader.

Parameters

• smc_reader (SMCReader) – An SmcReader instance containing kinect and iphone camera parameters.

• camera_type (Literal['kinect', 'iphone']) – Which type of camera to get.

• camera_id (int) – ID of the selected camera.

• align_floor (bool) – Whether to rotate camera extrinsics, makes the xOz plane parallel to floor calibrated by Kinect_0’s depth camera. Defaults to True.

• logger (Union[None, str, logging.Logger], optional) – Logger for logging. If None, root logger will be selected. Defaults to None.

Raises

• NotImplementedError – iphone has not been supported yet.

• KeyError – camera_type is neither kinect nor iphone.

Returns

FisheyeCameraParameter

xrmocap.io.get_n_frame_from_mview_src(img_arr: Union[None, numpy.ndarray] = None, img_paths: Union[None, List[List[str]]] = None, video_paths: Optional[List[str]] = None, logger: Union[None, str, logging.Logger] = None) → int

Get number of frames from multi-view image source. It needs one images input among [img_arr, img_paths, video_paths].

Parameters

• img_arr (Union[None, np.ndarray], optional) – A multi-view image array, in shape [n_view, n_frame, h, w, c]. Defaults to None.

• img_paths (Union[None, List[List[str]]], optional) – A nested list of image paths, in shape [n_view, n_frame]. Defaults to None.

• video_paths (Union[None, List[str]], optional) – A list of video paths, each is a view. Defaults to None.

• logger (Union[None, str, logging.Logger], optional) – Logger for logging. If None, root logger will be selected. Defaults to None.

Returns

int – Number of frames.

xrmocap.io.load_camera_parameters_from_zoemotion_dir(camera_parameter_path: str, enable_camera_id: Optional[list] = None) → Tuple[list, list]

Load camera parameter and get an RGB FisheyeCameraParameter.

Parameters

• camera_parameter_path (str) – path to the camera parameter

• enable_camera_id (Union[list, None], optional) – camera ID(str).

• to None. (Defaults) –

Returns

• cam_param_list (list) – FisheyeCameraParameter

• enable_camera_list (list) – enable camera list e.g.[‘0’, ‘1’]

xrmocap.io.load_clip_from_mview_src(start_idx: int, end_idx: int, img_arr: Union[None, numpy.ndarray] = None, img_paths: Union[None, List[List[str]]] = None, video_paths: Optional[List[str]] = None, logger: Union[None, str, logging.Logger] = None) → numpy.ndarray

Get image array of a clip from multi-view image source. It needs one images input among [img_arr, img_paths, video_paths].

Parameters

• img_arr (Union[None, np.ndarray], optional) – A multi-view image array, in shape [n_view, n_frame, h, w, c]. Defaults to None.

• img_paths (Union[None, List[List[str]]], optional) – A nested list of image paths, in shape [n_view, n_frame]. Defaults to None.

• video_paths (Union[None, List[str]], optional) – A list of video paths, each is a view. Defaults to None.

• logger (Union[None, str, logging.Logger], optional) – Logger for logging. If None, root logger will be selected. Defaults to None.

Returns

np.ndarray – Image array of the selected range.

xrmocap.io.load_multiview_images(img_paths: Union[None, List[List[str]]]) → numpy.ndarray

Load multi-view images to an ndarray.

Parameters

img_paths (Union[None, List[List[str]]]) – A nested list of image paths, in shape [view_n, frame_n].

Returns

np.ndarray – Multi-view image array, in shape [view_n, frame_n, h, w, c].

xrmocap.model

architecture

xrmocap.ops

projection

class xrmocap.ops.projection.AniposelibProjector(camera_parameters: List[xrprimer.data_structure.camera.fisheye_camera.FisheyeCameraParameter], logger: Union[None, str, logging.Logger] = None)

project(points: Union[numpy.ndarray, list, tuple], points_mask: Optional[Union[numpy.ndarray, list, tuple]] = None) → numpy.ndarray

Project points with self.camera_parameters.

Parameters

• points (Union[np.ndarray, list, tuple]) – An ndarray or a nested list of points3d, in shape [n_point, 3].

• points_mask (Union[np.ndarray, list, tuple], optional) – An ndarray or a nested list of mask, in shape [n_point, 1]. If points_mask[index] == 1, points[index] is valid for projection, else it is ignored. Defaults to None.

Returns

np.ndarray – An ndarray of points2d, in shape [n_view, n_point, 2].

project_single_point(points: Union[numpy.ndarray, list, tuple]) → numpy.ndarray

Project a single point with self.camera_parameters.

Parameters

points (Union[np.ndarray, list, tuple]) – An ndarray or a list of points3d, in shape [3].

Returns

np.ndarray – An ndarray of points2d, in shape [n_view, 2].

class xrmocap.ops.projection.PytorchProjector(camera_parameters: List[xrprimer.data_structure.camera.fisheye_camera.FisheyeCameraParameter], logger: Union[None, str, logging.Logger] = None)

project(points: torch.Tensor, points_mask: Optional[torch._VariableFunctionsClass.tensor] = None) → torch.Tensor

Project points with self.camera_parameters.

Parameters

• points (torch.Tensor) – points3d, in shape [n_point, 3].

• points_mask (torch.Tensor, optional) – mask, in shape [n_point, 1]. If points_mask[index] == 1, points[index] is valid for projection, else it is ignored. Defaults to None.

Returns

torch.Tensor – points2d, in shape [n_view, n_point, 2].

project_single_point(points: torch.Tensor) → torch.Tensor

Project a single point with self.camera_parameters.

Parameters

points (torch.Tensor) – points3d, in shape [3].

Returns

torch.Tensor – points2d, in shape [n_view, 2].

xrmocap.transform

keypoints3d.optim

xrmocap.utils