DualDeviceBuffer Class Reference

Simple memory buffer using un-managed pinned host memory when compiled with NVCC. Obviously this class is making explicit used of CUDA so replace it with your own memory allocator if you are not using CUDA. More...

#include <nanovdb/cuda/DeviceBuffer.h>

Public Types
using	PtrT = std::shared_ptr<DualDeviceBuffer>

Public Member Functions
	DualDeviceBuffer ()
	Default constructor of an empty buffer.
	DualDeviceBuffer (uint64_t size, int device=cudaCpuDeviceId, cudaStream_t stream=0)
	Constructor with a specified device and size.
	DualDeviceBuffer (uint64_t size, bool host, void *stream)
	Constructor.
	DualDeviceBuffer (uint64_t size, void *cpuData, void *gpuData)
	Constructor for externally managed host and device buffers.
	DualDeviceBuffer (uint64_t size, void *cpuData, std::initializer_list< std::pair< int, void * > > list)
	Constructor for externally managed host and multiple device buffers.
	DualDeviceBuffer (const DualDeviceBuffer &)=delete
	Disallow copy-construction.
	DualDeviceBuffer (DualDeviceBuffer &&other) noexcept
	Move copy-constructor.
	DualDeviceBuffer (const HostBuffer &buffer, int device=cudaCpuDeviceId, cudaStream_t stream=0)
	Copy-constructor from a HostBuffer.
	~DualDeviceBuffer ()
	Destructor frees memory on both the host and device.
DualDeviceBuffer &	operator= (const DualDeviceBuffer &)=delete
	Disallow copy assignment operation.
DualDeviceBuffer &	operator= (DualDeviceBuffer &&other) noexcept
	Move copy assignment operation.
void *	data () const
	Retuns a raw void pointer to the host/CPU buffer managed by this allocator.
template<typename T>
T *	data (ptrdiff_t count=0, int device=cudaCpuDeviceId) const
	Returns an offset pointer of a specific type from the allocated host memory.
void *	data (ptrdiff_t byteOffset, int device=cudaCpuDeviceId) const
	Returns a byte offset void pointer from the allocated host memory.
void	orderAfterPriorUses (int device, cudaStream_t stream) const
	Order work subsequently issued on stream after every prior use of this device buffer, whichever stream those uses were issued on. The consume-side companion of recordUse: an external consumer (e.g. a zero-copy array-interface export) calls this with its own stream before reading, so it cannot observe a partially-written buffer after asynchronous uploads or recorded kernels.
void	recordUse (int device, cudaStream_t stream)
	Record that this buffer's device data was just used on stream, so that the buffer's device frees (destructor, move-assignment, clear) are ordered after that work. Uses issued through deviceUpload/deviceDownload are recorded automatically; callers that enqueue their own kernels or copies against the raw pointer returned by deviceData() should call this afterwards. Without it, such work is only safe if it is on a blocking stream (which the free, issued on the default stream, waits on implicitly) or if the caller synchronizes before the buffer is cleared/destroyed.
void *	deviceData (int device) const
	Retuns a raw pointer to the specified device/GPU buffer managed by this allocator.
void *	deviceData () const
	Retuns a raw pointer to the current device/GPU buffer managed by this allocator.
void	deviceUpload (int device=0, cudaStream_t stream=0, bool sync=true)
	Uploads buffer on the host to a specific device. If it doesn't exist it's created first.
void	deviceUpload (int device, void *stream, bool sync)
void	deviceUpload (cudaStream_t stream, bool sync)
	Upload buffer from the host to ALL the existing devices, i.e. CPU -> GPU. If no device buffers exist one is created for the current device (typically 0) and subsequently populated with the host data.
void	deviceUpload (void *stream, bool sync)
void	deviceDownload (int device=0, cudaStream_t stream=0, bool sync=true)
	Download data from a specified device to the host. If the host buffer des not exist it will first be allocated.
void	deviceDownload (int device, void *stream, bool sync)
void	deviceDownload (void *stream, bool sync)
	Download the buffer from the current device to the host, i.e. GPU -> CPU. If the host buffer des not exist it will first be allocated.
uint64_t	size () const
	Returns the size in bytes of the raw memory buffer managed by this allocator.
uint64_t	capacity () const
int	bufferCount () const
	Returns the number of buffers that are not NULL.
int	deviceCount () const
void	clear (cudaStream_t stream=0)
	De-allocate all memory managed by this allocator and set all pointers to NULL.
void	clear (void *stream)
bool	empty () const
	Returns true if this allocator is empty, i.e. has no allocated memory.
bool	isEmpty () const
	Returns true if this allocator is empty, i.e. has no allocated memory.

Static Public Member Functions
static DualDeviceBuffer	create (uint64_t size, const DualDeviceBuffer *dummy, bool host, void *stream)
	Static factory method that return an instance of this buffer.
static DualDeviceBuffer	create (uint64_t size, const DualDeviceBuffer *dummy=nullptr, int device=cudaCpuDeviceId, cudaStream_t stream=0)
	Static factory method that returns an instance of this buffer.
static DualDeviceBuffer	create (uint64_t size, void *cpuData, void *gpuData)
	Static factory method that returns an instance of this buffer that wraps externally managed memory.
static DualDeviceBuffer	create (uint64_t size, void *cpuData, std::initializer_list< std::pair< int, void * > > list)
	Static factory method that returns an instance of this buffer that wraps externally managed host and device memory.
static DualDeviceBuffer	create (const HostBuffer &buffer, int device=cudaCpuDeviceId, cudaStream_t stream=0)
	Static factory method that returns an instance of this buffer constructed from a HostBuffer.
static PtrT	createPtr (uint64_t size, const DualDeviceBuffer *=nullptr, int device=cudaCpuDeviceId, cudaStream_t stream=0)
	Factory methods that create a shared pointer to an DualDeviceBuffer instance.
static PtrT	createPtr (uint64_t size, void *cpuData, void *gpuData)
	Factory methods that create a shared pointer to an DualDeviceBuffer instance.
static PtrT	createPtr (uint64_t size, void *cpuData, std::initializer_list< std::pair< int, void * > > list)
	Factory methods that create a shared pointer to an DualDeviceBuffer instance.
static PtrT	createPtr (const HostBuffer &buffer, int device=cudaCpuDeviceId, cudaStream_t stream=0)
	Factory methods that create a shared pointer to an DualDeviceBuffer instance.

Detailed Description

Simple memory buffer using un-managed pinned host memory when compiled with NVCC. Obviously this class is making explicit used of CUDA so replace it with your own memory allocator if you are not using CUDA.

Note

While CUDA's pinned host memory allows for asynchronous memory copy between host and device it is significantly slower then cached (un-pinned) memory on the host.

This is the implementation behind the deprecated DeviceBuffer alias below, renamed so the [[deprecated]] attribute reaches only code that spells the public name: the GPU tools' signature defaults reference this implementation, so default-using callers stay warning-free until the defaults change at removal. Transitional – do not adopt this name; it is deleted together with the alias. The header keeps its long-standing name and include path for the same reason: renaming a header breaks existing includes outright, and the old path is where external code will find the alias and its migration message.

Member Typedef Documentation

PtrT

using PtrT = std::shared_ptr<DualDeviceBuffer>

Constructor & Destructor Documentation

DualDeviceBuffer() [1/8]

DualDeviceBuffer ( )

inline

Default constructor of an empty buffer.

DualDeviceBuffer() [2/8]

DualDeviceBuffer ( uint64_t size, int device = cudaCpuDeviceId, cudaStream_t stream = 0 )

inline

Constructor with a specified device and size.

Parameters

size	byte size of buffer to be initialized
device	id of the device on which to initialize the buffer
stream	cuda stream

DualDeviceBuffer() [3/8]

DualDeviceBuffer ( uint64_t size, bool host, void * stream )

inline

Constructor.

Parameters

size	byte size of buffer to be initialized
host	If true buffer is initialized only on the host/CPU, else on the current device/GPU
stream	optional stream argument (defaults to stream NULL)

DualDeviceBuffer() [4/8]

DualDeviceBuffer ( uint64_t size, void * cpuData, void * gpuData )

inline

Constructor for externally managed host and device buffers.

Parameters

size	byte size of the two external buffers
cpuData	host buffer, assumed to NOT be NULL
gpuData	device buffer, assumed to NOT be NULL;

Note

The device buffer, gpuData, will be associated with the current device ID given by cudaGetDevice

DualDeviceBuffer() [5/8]

DualDeviceBuffer ( uint64_t size, void * cpuData, std::initializer_list< std::pair< int, void * > > list )

inline

Constructor for externally managed host and multiple device buffers.

Parameters

size	byte size of the two external buffers
cpuData	host buffer, assumed to NOT be NULL
list	list of device IDs and external device buffers, all assumed to not be NULL

DualDeviceBuffer() [6/8]

DualDeviceBuffer ( const DualDeviceBuffer & )

delete

Disallow copy-construction.

DualDeviceBuffer() [7/8]

DualDeviceBuffer ( DualDeviceBuffer && other )

inlinenoexcept

Move copy-constructor.

DualDeviceBuffer() [8/8]

DualDeviceBuffer ( const HostBuffer & buffer, int device = cudaCpuDeviceId, cudaStream_t stream = 0 )

inline

Copy-constructor from a HostBuffer.

Parameters

buffer	host buffer from which to copy data
device	id of the device on which to initialize the buffer
stream	cuda stream

~DualDeviceBuffer()

~DualDeviceBuffer ( )

inline

Destructor frees memory on both the host and device.

Note

Each managed device free waits on that device's tracking event first, so it is ordered after every stream the buffer was used on, not just the most recent one.

Member Function Documentation

bufferCount()

int bufferCount ( ) const

inline

Returns the number of buffers that are not NULL.

capacity()

uint64_t capacity ( ) const

inline

clear() [1/2]

void clear ( cudaStream_t stream = 0 )

inline

De-allocate all memory managed by this allocator and set all pointers to NULL.

Parameters

stream

Stream the device frees are issued on. The frees are additionally ordered after every stream the buffer was used on (via the per-device tracking event), so stream selects where the free is enqueued, not what it is ordered against - any stream is safe to pass here regardless of where the buffer was used.

clear() [2/2]

void clear ( void * stream )

inline

create() [1/5]

DualDeviceBuffer create ( const HostBuffer & buffer, int device = cudaCpuDeviceId, cudaStream_t stream = 0 )

inlinestatic

Static factory method that returns an instance of this buffer constructed from a HostBuffer.

Parameters

buffer	host buffer from which to copy data
device	id of the device on which to initialize the buffer
stream	cuda stream

create() [2/5]

DualDeviceBuffer create ( uint64_t size, const DualDeviceBuffer * dummy, bool host, void * stream )

inlinestatic

Static factory method that return an instance of this buffer.

Parameters

size	byte size of buffer to be initialized
dummy	this argument is currently ignored but required to match the API of the HostBuffer
host	If true buffer is initialized only on the host/CPU, else only on the device/GPU
stream	optional stream argument (defaults to stream NULL)

Returns

An instance of this class using move semantics

create() [3/5]

DualDeviceBuffer create ( uint64_t size, const DualDeviceBuffer * dummy = nullptr, int device = cudaCpuDeviceId, cudaStream_t stream = 0 )

inlinestatic

Static factory method that returns an instance of this buffer.

Parameters

size	byte size of buffer to be initialized
dummy	this argument is currently ignored but required to match the API of the HostBuffer
device	id of the device on which to initialize the buffer
stream	cuda stream

create() [4/5]

DualDeviceBuffer create ( uint64_t size, void * cpuData, std::initializer_list< std::pair< int, void * > > list )

inlinestatic

Static factory method that returns an instance of this buffer that wraps externally managed host and device memory.

Parameters

size	byte size of buffer to be initialized
cpuData	pointer to externally managed host memory
list	list of device IDs and device memory pointers

create() [5/5]

DualDeviceBuffer create ( uint64_t size, void * cpuData, void * gpuData )

inlinestatic

Static factory method that returns an instance of this buffer that wraps externally managed memory.

Parameters

size	byte size of buffer specified by external memory
cpuData	pointer to externally managed host memory
gpuData	pointer to externally managed device memory

Returns

An instance of this class using move semantics

createPtr() [1/4]

PtrT createPtr ( const HostBuffer & buffer, int device = cudaCpuDeviceId, cudaStream_t stream = 0 )

inlinestatic

Factory methods that create a shared pointer to an DualDeviceBuffer instance.

createPtr() [2/4]

PtrT createPtr ( uint64_t size, const DualDeviceBuffer * = nullptr, int device = cudaCpuDeviceId, cudaStream_t stream = 0 )

inlinestatic

Factory methods that create a shared pointer to an DualDeviceBuffer instance.

createPtr() [3/4]

PtrT createPtr ( uint64_t size, void * cpuData, std::initializer_list< std::pair< int, void * > > list )

inlinestatic

Factory methods that create a shared pointer to an DualDeviceBuffer instance.

createPtr() [4/4]

PtrT createPtr ( uint64_t size, void * cpuData, void * gpuData )

inlinestatic

Factory methods that create a shared pointer to an DualDeviceBuffer instance.

data() [1/3]

void * data ( ) const

inline

Retuns a raw void pointer to the host/CPU buffer managed by this allocator.

Warning

Note that the pointer can be NULL!

data() [2/3]

void * data ( ptrdiff_t byteOffset, int device = cudaCpuDeviceId ) const

inline

Returns a byte offset void pointer from the allocated host memory.

Parameters

byteOffset	offset of return pointer in units of bytes
device	Device whose buffer is returned, or cudaCpuDeviceId for the host buffer

Warning

assumes that this instance is not empty!

data() [3/3]

template<typename T>

T * data ( ptrdiff_t count = 0, int device = cudaCpuDeviceId ) const

inline

Returns an offset pointer of a specific type from the allocated host memory.

Template Parameters

T	Type of the pointer returned

Parameters

count	Numbers of elements of parameter type T to skip
device	Device whose buffer is returned, or cudaCpuDeviceId for the host buffer

Warning

might return NULL

deviceCount()

int deviceCount ( ) const

inline

deviceData() [1/2]

void * deviceData ( ) const

inline

Retuns a raw pointer to the current device/GPU buffer managed by this allocator.

Warning

Note that the pointer can be NULL!

deviceData() [2/2]

void * deviceData ( int device ) const

inline

Retuns a raw pointer to the specified device/GPU buffer managed by this allocator.

Warning

Note that the pointer can be NULL!

Note

Work enqueued against this raw pointer is invisible to the buffer's lifetime tracking: on a non-blocking stream, call recordUse afterwards (or synchronize before the buffer is cleared/destroyed) so the device free is ordered after it.

deviceDownload() [1/3]

void deviceDownload ( int device, void * stream, bool sync )

inline

deviceDownload() [2/3]

void deviceDownload ( int device = 0, cudaStream_t stream = 0, bool sync = true )

inline

Download data from a specified device to the host. If the host buffer des not exist it will first be allocated.

Parameters

device	device ID to download source data from
stream	cuda stream
sync	if false the memory copy is asynchronous.

Warning

Assumes that the specifed device buffer already exists!

deviceDownload() [3/3]

void deviceDownload ( void * stream, bool sync )

inline

Download the buffer from the current device to the host, i.e. GPU -> CPU. If the host buffer des not exist it will first be allocated.

Parameters

stream	CUDA stream
sync	if false the memory copy is asynchronous

Note

If the host/CPU buffer does not exist it is first allocated

Warning

Assumes that the device/GPU buffer already exists

deviceUpload() [1/4]

void deviceUpload ( cudaStream_t stream, bool sync )

inline

Upload buffer from the host to ALL the existing devices, i.e. CPU -> GPU. If no device buffers exist one is created for the current device (typically 0) and subsequently populated with the host data.

Parameters

stream	CUDA stream.
sync	if false the memory copy is asynchronous.

Warning

Assumes that the host buffer already exists!

deviceUpload() [2/4]

void deviceUpload ( int device, void * stream, bool sync )

inline

deviceUpload() [3/4]

void deviceUpload ( int device = 0, cudaStream_t stream = 0, bool sync = true )

inline

Uploads buffer on the host to a specific device. If it doesn't exist it's created first.

Parameters

device	Device ID that the data is copied to
stream	cuda stream
sync	if false the memory copy is asynchronous.

Warning

Assumes that the host buffer already exists!

Note

determine the current device with cudaGetDevice

deviceUpload() [4/4]

void deviceUpload ( void * stream, bool sync )

inline

empty()

bool empty ( ) const

inline

Returns true if this allocator is empty, i.e. has no allocated memory.

isEmpty()

bool isEmpty ( ) const

inline

Returns true if this allocator is empty, i.e. has no allocated memory.

operator=() [1/2]

DualDeviceBuffer & operator= ( const DualDeviceBuffer & )

delete

Disallow copy assignment operation.

operator=() [2/2]

DualDeviceBuffer & operator= ( DualDeviceBuffer && other )

inlinenoexcept

Move copy assignment operation.

orderAfterPriorUses()

void orderAfterPriorUses ( int device, cudaStream_t stream ) const

inline

Order work subsequently issued on stream after every prior use of this device buffer, whichever stream those uses were issued on. The consume-side companion of recordUse: an external consumer (e.g. a zero-copy array-interface export) calls this with its own stream before reading, so it cannot observe a partially-written buffer after asynchronous uploads or recorded kernels.

Parameters

device	Device whose buffer is about to be read
stream	Stream the consumer's work will be issued on

recordUse()

void recordUse ( int device, cudaStream_t stream )

inline

Record that this buffer's device data was just used on stream, so that the buffer's device frees (destructor, move-assignment, clear) are ordered after that work. Uses issued through deviceUpload/deviceDownload are recorded automatically; callers that enqueue their own kernels or copies against the raw pointer returned by deviceData() should call this afterwards. Without it, such work is only safe if it is on a blocking stream (which the free, issued on the default stream, waits on implicitly) or if the caller synchronizes before the buffer is cleared/destroyed.

Parameters

device	Device whose buffer was used
stream	Stream the work was issued on

Note

Recording chains across streams: stream is first ordered after the previously recorded use (if any) so the single per-device event transitively covers every recorded use, not just the last one. Without this, concurrent uses on streams A then B would leave only B's event, and the device free could run while A's work is still in flight. The side effect is that work subsequently issued on stream also waits on the previously recorded use – acceptable for a shared buffer, where later-recorded consumers observing earlier writes is the expected ordering. Note this also serializes CONCURRENT READERS that record uses (the single event cannot distinguish read-read from write-read); if that ever matters in a profile, the upgrade path is a read/write-separated or per-record event scheme, not a revert.

size()

uint64_t size ( ) const

inline

Returns the size in bytes of the raw memory buffer managed by this allocator.