Resources#

All resources which are specific to a computing environment like host or device are contained within, and managed by, raft::resources. This design simplifies the APIs and eases user burden by making the APIs opaque by default but allowing customization based on user preference.

Vocabulary#

#include <raft/core/resource/resource_types.hpp>

namespace raft::resource

enum resource_type#

Resource types can apply to any resource and don’t have to be host- or device-specific.

Values:

enumerator CUBLAS_HANDLE#

enumerator CUSOLVER_DN_HANDLE#

enumerator CUSOLVER_SP_HANDLE#

enumerator CUSPARSE_HANDLE#

enumerator CUDA_STREAM_VIEW#

enumerator CUDA_STREAM_POOL#

enumerator CUDA_STREAM_SYNC_EVENT#

enumerator COMMUNICATOR#

enumerator SUB_COMMUNICATOR#

enumerator DEVICE_PROPERTIES#

enumerator DEVICE_ID#

enumerator STREAM_VIEW#

enumerator THRUST_POLICY#

enumerator WORKSPACE_RESOURCE#

enumerator CUBLASLT_HANDLE#

enumerator CUSTOM#

enumerator LAST_KEY#

class resource#

#include <resource_types.hpp>

A resource constructs and contains an instance of some pre-determined object type and facades that object behind a common API.

Subclassed by raft::resource::comms_resource, raft::resource::cublas_resource, raft::resource::cublaslt_resource, raft::resource::cuda_event_resource, raft::resource::cuda_stream_pool_resource, raft::resource::cuda_stream_resource, raft::resource::cusolver_dn_resource, raft::resource::cusolver_sp_resource, raft::resource::cusparse_resource, raft::resource::custom_resource, raft::resource::device_id_resource, raft::resource::device_properties_resource, raft::resource::empty_resource, raft::resource::limiting_memory_resource, raft::resource::stream_view_resource, raft::resource::sub_comms_resource, raft::resource::thrust_policy_resource

class empty_resource : public raft::resource::resource #

#include <resource_types.hpp>

class resource_factory#

#include <resource_types.hpp>

A resource factory knows how to construct an instance of a specific raft::resource::resource.

Subclassed by raft::resource::comms_resource_factory, raft::resource::cublas_resource_factory, raft::resource::cublaslt_resource_factory, raft::resource::cuda_stream_pool_resource_factory, raft::resource::cuda_stream_resource_factory, raft::resource::cusolver_dn_resource_factory, raft::resource::cusolver_sp_resource_factory, raft::resource::cusparse_resource_factory, raft::resource::custom_resource_factory, raft::resource::device_id_resource_factory, raft::resource::device_properties_resource_factory, raft::resource::empty_resource_factory, raft::resource::stream_view_resource_factory, raft::resource::sub_comms_resource_factory, raft::resource::thrust_policy_resource_factory, raft::resource::workspace_resource_factory

Public Functions

virtual resource_type get_resource_type() = 0#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

virtual resource *make_resource() = 0#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

class empty_resource_factory : public raft::resource::resource_factory #

#include <resource_types.hpp>

A resource factory knows how to construct an instance of a specific raft::resource::resource.

Public Functions

inline virtual resource_type get_resource_type() override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource() override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Device Resources#

raft::device_resources is a convenience over using raft::resources directly. It provides accessor methods to retrieve resources such as the CUDA stream, stream pool, and handles to the various CUDA math libraries like cuBLAS and cuSOLVER.

#include <raft/core/device_resources.hpp>

namespace raft::core

class device_resources : public raft::resources#

Main resource container object that stores all necessary resources used for calling necessary device functions, cuda kernels and/or libraries.

Public Functions

inline device_resources(rmm::cuda_stream_view stream_view = rmm::cuda_stream_per_thread, std::shared_ptr<rmm::cuda_stream_pool> stream_pool = {nullptr}, std::shared_ptr<rmm::mr::device_memory_resource> workspace_resource = {nullptr}, std::optional<std::size_t> allocation_limit = std::nullopt)#

Construct a resources instance with a stream view and stream pool.

Parameters:

stream_view – the default stream (which has the default per-thread stream if unspecified)
stream_pool – [in] the stream pool used (which has default of nullptr if unspecified)
workspace_resource – [in] an optional resource used by some functions for allocating temporary workspaces.
allocation_limit – [in] the total amount of memory in bytes available to the temporary workspace resources.

inline virtual ~device_resources()#: Destroys all held-up resources

inline void sync_stream(rmm::cuda_stream_view stream) const#: synchronize a stream on the current container

inline void sync_stream() const#: synchronize main stream on the current container

inline rmm::cuda_stream_view get_stream() const#: returns main stream on the current container

inline bool is_stream_pool_initialized() const#: returns whether stream pool was initialized on the current container

inline const rmm::cuda_stream_pool &get_stream_pool() const#: returns stream pool on the current container

inline rmm::cuda_stream_view get_stream_from_stream_pool() const#: return stream from pool

inline rmm::cuda_stream_view get_stream_from_stream_pool(std::size_t stream_idx) const#: return stream from pool at index

inline rmm::cuda_stream_view get_next_usable_stream() const#: return stream from pool if size > 0, else main stream on current container

inline rmm::cuda_stream_view get_next_usable_stream(std::size_t stream_idx) const#

return stream from pool at index if size > 0, else main stream on current container

Parameters:: stream_idx – [in] the required index of the stream in the stream pool if available

inline void sync_stream_pool() const#: synchronize the stream pool on the current container

inline void sync_stream_pool(const std::vector<std::size_t> stream_indices) const#

synchronize subset of stream pool

Parameters:: stream_indices – [in] the indices of the streams in the stream pool to synchronize

inline void wait_stream_pool_on_stream() const#: ask stream pool to wait on last event in main stream

Device Resources Manager#

While raft::device_resources provides a convenient way to access device-related resources for a sequence of RAFT calls, it is sometimes useful to be able to limit those resources across an entire application. For instance, in highly multi-threaded applications, it can be helpful to limit the total number of streams rather than relying on the default stream per thread. raft::device_resources_manager offers a way to access raft::device_resources instances that draw from a limited pool of underlying device resources.

#include <raft/core/device_resources_manager.hpp>

namespace raft::core

Warning

doxygenclass: Cannot find class “raft::device_resources_manager” in doxygen xml output for project “RAFT” from directory: ../../cpp/doxygen/_xml/

Resource Functions#

Comms#

#include <raft/core/resource/comms.hpp>

namespace raft::resource

inline bool comms_initialized(resources const &res)#

inline comms::comms_t const &get_comms(resources const &res)#

inline void set_comms(resources const &res, std::shared_ptr<comms::comms_t> communicator)#

cuBLAS Handle#

#include <raft/core/resource/cublas_handle.hpp>

namespace raft::resource

inline cublasHandle_t get_cublas_handle(resources const &res)#

Load a cublasHandle_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cublas handle

cuBLASLt Handle#

#include <raft/core/resource/cublaslt_handle.hpp>

namespace raft::resource

inline auto get_cublaslt_handle(resources const &res) -> cublasLtHandle_t#

Load a cublasLtHandle_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cublasLt handle

CUDA Stream#

#include <raft/core/resource/cuda_stream.hpp>

namespace raft::resource

inline rmm::cuda_stream_view get_cuda_stream(resources const &res)#

Load a rmm::cuda_stream_view from a resources instance (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

inline void set_cuda_stream(resources const &res, rmm::cuda_stream_view stream_view)#

Load a rmm::cuda_stream_view from a resources instance (and populate it on the res if needed).

Parameters:

res – [in] raft resources object for managing resources

stream_view – cuda stream view

inline void sync_stream(const resources &res, rmm::cuda_stream_view stream)#

synchronize a specific stream

Parameters:

res – [in] the raft resources object

stream – [in] stream to synchronize

inline void sync_stream(const resources &res)#

synchronize main stream on the resources instance

CUDA Stream Pool#

#include <raft/core/resource/cuda_stream_pool.hpp>

namespace raft::resource

inline const rmm::cuda_stream_pool &get_cuda_stream_pool(const resources &res)#

Load a cuda_stream_pool, and create a new one if it doesn’t already exist

Parameters:: res – raft res object for managing resources
Returns:

inline void set_cuda_stream_pool(const resources &res, std::shared_ptr<rmm::cuda_stream_pool> stream_pool)#

Explicitly set a stream pool on the current res. Note that this will overwrite an existing stream pool on the res.

Parameters:

res –
stream_pool –

inline std::size_t get_stream_pool_size(const resources &res)#

inline rmm::cuda_stream_view get_stream_from_stream_pool(const resources &res)#: return stream from pool

inline rmm::cuda_stream_view get_stream_from_stream_pool(const resources &res, std::size_t stream_idx)#: return stream from pool at index

inline rmm::cuda_stream_view get_next_usable_stream(const resources &res)#: return stream from pool if size > 0, else main stream on res

inline rmm::cuda_stream_view get_next_usable_stream(const resources &res, std::size_t stream_idx)#

return stream from pool at index if size > 0, else main stream on res

Parameters:

res – [in] the raft resources object
stream_idx – [in] the required index of the stream in the stream pool if available

inline void sync_stream_pool(const resources &res)#

synchronize the stream pool on the res

Parameters:: res – [in] the raft resources object

inline void sync_stream_pool(const resources &res, const std::vector<std::size_t> stream_indices)#

synchronize subset of stream pool

Parameters:

res – [in] the raft resources object
stream_indices – [in] the indices of the streams in the stream pool to synchronize

inline void wait_stream_pool_on_stream(const resources &res)#

ask stream pool to wait on last event in main stream

Parameters:: res – [in] the raft resources object

cuSolverDn Handle#

#include <raft/core/resource/cusolver_dn_handle.hpp> namespace raft::resource

inline cusolverDnHandle_t get_cusolver_dn_handle(resources const &res)#

Load a cusolverSpres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusolver dn handle

class cusolver_dn_resource_factory : public raft::resource::resource_factory #

#include <cusolver_dn_handle.hpp>

Factory that knows how to construct a specific raft::resource to populate the res_t.

Public Functions

inline virtual resource_type get_resource_type() override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource() override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

cuSolverSp Handle#

#include <raft/core/resource/cusolver_sp_handle.hpp>

namespace raft::resource

inline cusolverSpHandle_t get_cusolver_sp_handle(resources const &res)#

Load a cusolverSpres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusolver sp handle

cuSparse Handle#

#include <raft/core/resource/cusparse_handle.hpp>

namespace raft::resource

inline cusparseHandle_t get_cusparse_handle(resources const &res)#

Load a cusparseres_t from raft res if it exists, otherwise add it and return it.

Parameters:

res – [in] the raft resources object

Returns:

cusparse handle

Device ID#

#include <raft/core/resource/device_id.hpp>

namespace raft::resource

inline int get_device_id(resources const &res)#

Load a device id from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

device id

Device Memory Resource#

#include <raft/core/resource/device_memory_resource.hpp>

namespace raft::resource

inline auto get_workspace_resource(resources const &res) -> rmm::mr::limiting_resource_adaptor<rmm::mr::device_memory_resource>*#

Load a temp workspace resource from a resources instance (and populate it on the res if needed).

Parameters:

res – raft resources object for managing resources

Returns:

device memory resource object

inline auto get_workspace_total_bytes(resources const &res) -> size_t#

Get the total size of the workspace resource.

inline auto get_workspace_used_bytes(resources const &res) -> size_t#

Get the already allocated size of the workspace resource.

inline auto get_workspace_free_bytes(resources const &res) -> size_t#

Get the available size of the workspace resource.

inline void set_workspace_resource(resources const &res, std::shared_ptr<rmm::mr::device_memory_resource> mr = {nullptr}, std::optional<std::size_t> allocation_limit = std::nullopt, std::optional<std::size_t> alignment = std::nullopt)#

Set a temporary workspace resource on a resources instance.

Parameters:

res – raft resources object for managing resources

mr – an optional RMM device_memory_resource

allocation_limit – the total amount of memory in bytes available to the temporary workspace resources.

alignment – optional alignment requirements passed to RMM allocations

inline void set_workspace_to_pool_resource(resources const &res, std::optional<std::size_t> allocation_limit = std::nullopt)#

Set the temporary workspace resource to a pool on top of the global memory resource (rmm::mr::get_current_device_resource().

Parameters:

res – raft resources object for managing resources

allocation_limit – the total amount of memory in bytes available to the temporary workspace resources; if not provided, a last used or default limit is used.

inline void set_workspace_to_global_resource(resources const &res, std::optional<std::size_t> allocation_limit = std::nullopt)#

Set the temporary workspace resource the same as the global memory resource (rmm::mr::get_current_device_resource().

Note, the workspace resource is always limited; the limit here defines how much of the global memory resource can be consumed by the workspace allocations.

Parameters:

res – raft resources object for managing resources

allocation_limit – the total amount of memory in bytes available to the temporary workspace resources.

class limiting_memory_resource : public raft::resource::resource #

#include <device_memory_resource.hpp>

class workspace_resource_factory : public raft::resource::resource_factory #

#include <device_memory_resource.hpp>

Factory that knows how to construct a specific raft::resource to populate the resources instance.

Public Functions

inline virtual auto get_resource_type() -> resource_type override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual auto make_resource() -> resource* override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Public Static Functions

static inline auto default_pool_resource(std::size_t limit) -> std::shared_ptr<rmm::mr::device_memory_resource>#

Construct a sensible default pool memory resource.

static inline auto default_plain_resource() -> std::shared_ptr<rmm::mr::device_memory_resource>#

Get the global memory resource wrapped into an unmanaged shared_ptr (with no deleter).

Note: the lifetime of the underlying rmm::mr::get_current_device_resource() is managed somewhere else, since it’s passed by a raw pointer. Hence, this shared_ptr wrapper is not allowed to delete the pointer on destruction.

Device Properties#

#include <raft/core/resource/device_properties.hpp>

namespace raft::resource

inline cudaDeviceProp &get_device_properties(resources const &res)#

Load a cudaDeviceProp from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

populated cuda device properties instance

class device_properties_resource_factory : public raft::resource::resource_factory #

#include <device_properties.hpp>

Factory that knows how to construct a specific raft::resource to populate the res_t.

Public Functions

inline virtual resource_type get_resource_type() override#

Return the resource_type associated with the current factory.

Returns:

resource_type corresponding to the current factory

inline virtual resource *make_resource() override#

Construct an instance of the factory’s underlying resource.

Returns:

resource instance

Sub Communicators#

#include <raft/core/resource/sub_comms.hpp>

namespace raft::resource

inline const comms::comms_t &get_subcomm(const resources &res, std::string key)#

inline void set_subcomm(resources const &res, std::string key, std::shared_ptr<comms::comms_t> subcomm)#

Thrust Exec Policy#

#include <raft/core/resource/thrust_policy.hpp>

namespace raft::resource

inline rmm::exec_policy &get_thrust_policy(resources const &res)#

Load a thrust policy from a res (and populate it on the res if needed).

Parameters:

res – raft res object for managing resources

Returns:

thrust execution policy

Custom runtime-shared resources#

A custom resource is an arbitrary default-constructible C++ class. The consumer of the API can keep such a resource in the raft::resources handle. For example, consider a function that is expected to be called repeatedly and involves a costly kernel configuration. One can cache the kernel configuration in a custom resource. The cost of accessing it is one hashmap lookup.

#include <raft/core/resource/custom_resource.hpp>

namespace raft::resource

template<typename ResourceT>
auto get_custom_resource(resources const &res) -> ResourceT*#

Get the custom default-constructible resource if it exists, create it otherwise.

Note: in contrast to the other, hard-coded resources, there’s no information about the custom resources at compile time. Hence, custom resources are kept in a hashmap and looked-up at runtime. This leads to slightly slower access times.

Template Parameters:

ResourceT – the type of the resource; it must be complete and default-constructible.

Parameters:

res – [in] the raft resources object

Returns:

a pointer to the custom resource.