IOQueue: I/O Event Dispatching with Proactor Pattern

Data Structures
struct	pj_ioqueue_op_key_t
struct	pj_ioqueue_callback
struct	pj_ioqueue_cfg

Macros
#define	PJ_IOQUEUE_MAX_EVENTS_IN_SINGLE_POLL   (16)
#define	PJ_IOQUEUE_MAX_CAND_EVENTS   PJ_IOQUEUE_MAX_EVENTS_IN_SINGLE_POLL
#define	PJ_IOQUEUE_ALWAYS_ASYNC   ((pj_uint32_t)1 << (pj_uint32_t)31)

Enumerations
enum	pj_ioqueue_operation_e {   PJ_IOQUEUE_OP_NONE = 0 , PJ_IOQUEUE_OP_READ = 1 , PJ_IOQUEUE_OP_RECV = 2 , PJ_IOQUEUE_OP_RECV_FROM = 4 ,   PJ_IOQUEUE_OP_WRITE = 8 , PJ_IOQUEUE_OP_SEND = 16 , PJ_IOQUEUE_OP_SEND_TO = 32 , PJ_IOQUEUE_OP_ACCEPT = 64 ,   PJ_IOQUEUE_OP_CONNECT = 128 }
enum	pj_ioqueue_epoll_flag { PJ_IOQUEUE_EPOLL_EXCLUSIVE = 1 , PJ_IOQUEUE_EPOLL_ONESHOT = 2 , PJ_IOQUEUE_EPOLL_AUTO }

Functions
void	pj_ioqueue_cfg_default (pj_ioqueue_cfg *cfg)
const char *	pj_ioqueue_name (void)
pj_status_t	pj_ioqueue_create (pj_pool_t *pool, pj_size_t max_fd, pj_ioqueue_t **ioqueue)
pj_status_t	pj_ioqueue_create2 (pj_pool_t *pool, pj_size_t max_fd, const pj_ioqueue_cfg *cfg, pj_ioqueue_t **ioqueue)
pj_status_t	pj_ioqueue_destroy (pj_ioqueue_t *ioque)
pj_status_t	pj_ioqueue_set_lock (pj_ioqueue_t *ioque, pj_lock_t *lock, pj_bool_t auto_delete)
pj_status_t	pj_ioqueue_set_default_concurrency (pj_ioqueue_t *ioqueue, pj_bool_t allow)
pj_status_t	pj_ioqueue_register_sock (pj_pool_t *pool, pj_ioqueue_t *ioque, pj_sock_t sock, void *user_data, const pj_ioqueue_callback *cb, pj_ioqueue_key_t **key)
pj_status_t	pj_ioqueue_register_sock2 (pj_pool_t *pool, pj_ioqueue_t *ioque, pj_sock_t sock, pj_grp_lock_t *grp_lock, void *user_data, const pj_ioqueue_callback *cb, pj_ioqueue_key_t **key)
pj_status_t	pj_ioqueue_unregister (pj_ioqueue_key_t *key)
void *	pj_ioqueue_get_user_data (pj_ioqueue_key_t *key)
pj_status_t	pj_ioqueue_set_user_data (pj_ioqueue_key_t *key, void *user_data, void **old_data)
pj_status_t	pj_ioqueue_set_concurrency (pj_ioqueue_key_t *key, pj_bool_t allow)
pj_status_t	pj_ioqueue_lock_key (pj_ioqueue_key_t *key)
pj_status_t	pj_ioqueue_trylock_key (pj_ioqueue_key_t *key)
pj_status_t	pj_ioqueue_unlock_key (pj_ioqueue_key_t *key)
void	pj_ioqueue_op_key_init (pj_ioqueue_op_key_t *op_key, pj_size_t size)
pj_bool_t	pj_ioqueue_is_pending (pj_ioqueue_key_t *key, pj_ioqueue_op_key_t *op_key)
pj_status_t	pj_ioqueue_post_completion (pj_ioqueue_key_t *key, pj_ioqueue_op_key_t *op_key, pj_ssize_t bytes_status)
pj_status_t	pj_ioqueue_clear_key (pj_ioqueue_key_t *key)
pj_status_t	pj_ioqueue_accept (pj_ioqueue_key_t *key, pj_ioqueue_op_key_t *op_key, pj_sock_t *new_sock, pj_sockaddr_t *local, pj_sockaddr_t *remote, int *addrlen)
pj_status_t	pj_ioqueue_connect (pj_ioqueue_key_t *key, const pj_sockaddr_t *addr, int addrlen)
int	pj_ioqueue_poll (pj_ioqueue_t *ioque, const pj_time_val *timeout)
pj_status_t	pj_ioqueue_recv (pj_ioqueue_key_t *key, pj_ioqueue_op_key_t *op_key, void *buffer, pj_ssize_t *length, pj_uint32_t flags)
pj_status_t	pj_ioqueue_recvfrom (pj_ioqueue_key_t *key, pj_ioqueue_op_key_t *op_key, void *buffer, pj_ssize_t *length, pj_uint32_t flags, pj_sockaddr_t *addr, int *addrlen)
pj_status_t	pj_ioqueue_send (pj_ioqueue_key_t *key, pj_ioqueue_op_key_t *op_key, const void *data, pj_ssize_t *length, pj_uint32_t flags)
pj_status_t	pj_ioqueue_sendto (pj_ioqueue_key_t *key, pj_ioqueue_op_key_t *op_key, const void *data, pj_ssize_t *length, pj_uint32_t flags, const pj_sockaddr_t *addr, int addrlen)

Detailed Description

I/O Queue provides API for performing asynchronous I/O operations. It conforms to proactor pattern, which allows application to submit an asynchronous operation and to be notified later when the operation has completed.

The I/O Queue can work on both socket and file descriptors. For asynchronous file operations however, one must make sure that the correct file I/O back-end is used, because not all file I/O back-end can be used with the ioqueue. Please see File I/O for more details.

The framework works natively in platforms where asynchronous operation API exists, such as in Windows NT with IoCompletionPort/IOCP. In other platforms, the I/O queue abstracts the operating system's event poll API to provide semantics similar to IoCompletionPort with minimal penalties (i.e. per ioqueue and per handle mutex protection).

The I/O queue provides more than just unified abstraction. It also:

• makes sure that the operation uses the most effective way to utilize the underlying mechanism, to achieve the maximum theoritical throughput possible on a given platform.
• choose the most efficient mechanism for event polling on a given platform.

Currently, the I/O Queue is implemented using:

• select(), as the common denominator, but the least efficient. Also the number of descriptor is limited to PJ_IOQUEUE_MAX_HANDLES (which by default is 64).
• /dev/epoll on Linux (user mode and kernel mode), a much faster replacement for select() on Linux (and more importantly doesn't have limitation on number of descriptors).
• I/O Completion ports on Windows NT/2000/XP, which is the most efficient way to dispatch events in Windows NT based OSes, and most importantly, it doesn't have the limit on how many handles to monitor. And it works with files (not only sockets) as well.

Concurrency Rules

The ioqueue has been fine tuned to allow multiple threads to poll the handles simultaneously, to maximize scalability when the application is running on multiprocessor systems. When more than one threads are polling the ioqueue and there are more than one handles are signaled, more than one threads will execute the callback simultaneously to serve the events. These parallel executions are completely safe when the events happen for two different handles.

However, with multithreading, care must be taken when multiple events happen on the same handle, or when event is happening on a handle (and the callback is being executed) and application is performing unregistration to the handle at the same time.

The treatments of above scenario differ according to the concurrency setting that are applied to the handle.

Concurrency Settings for Handles

Concurrency can be set on per handle (key) basis, by using pj_ioqueue_set_concurrency() function. The default key concurrency value for the handle is inherited from the key concurrency setting of the ioqueue, and the key concurrency setting for the ioqueue can be changed by using pj_ioqueue_set_default_concurrency(). The default key concurrency setting for ioqueue itself is controlled by compile time setting PJ_IOQUEUE_DEFAULT_ALLOW_CONCURRENCY.

Note that this key concurrency setting only controls whether multiple threads are allowed to operate on the same key at the same time. The ioqueue itself always allows multiple threads to enter the ioqeuue at the same time, and also simultaneous callback calls to differrent keys is always allowed regardless to the key concurrency setting.

Parallel Callback Executions for the Same Handle

Note that when key concurrency is enabled (i.e. parallel callback calls on the same key is allowed; this is the default setting), the ioqueue will only perform simultaneous callback executions on the same key when the key has invoked multiple pending operations. This could be done for example by calling pj_ioqueue_recvfrom() more than once on the same key, each with the same key but different operation key (pj_ioqueue_op_key_t). With this scenario, when multiple packets arrive on the key at the same time, more than one threads may execute the callback simultaneously, each with the same key but different operation key.

When there is only one pending operation on the key (e.g. there is only one pj_ioqueue_recvfrom() invoked on the key), then events occuring to the same key will be queued by the ioqueue, thus no simultaneous callback calls will be performed.

Concurrency is Enabled (Default Value)

The default setting for the ioqueue is to allow multiple threads to execute callbacks for the same handle/key. This setting is selected to promote good performance and scalability for application.

However this setting has a major drawback with regard to synchronization, and application MUST carefully follow the following guidelines to ensure that parallel access to the key does not cause problems:

• Always note that callback may be called simultaneously for the same key.
• Care must be taken when unregistering a key from the ioqueue. Application must take care that when one thread is issuing an unregistration, other thread is not simultaneously invoking the callback to the same key.
  This happens because the ioqueue functions are working with a pointer to the key, and there is a possible race condition where the pointer has been rendered invalid by other threads before the ioqueue has a chance to acquire mutex on it.

Concurrency is Disabled

Alternatively, application may disable key concurrency to make synchronization easier. As noted above, there are three ways to control key concurrency setting:

• by controlling on per handle/key basis, with pj_ioqueue_set_concurrency().
• by changing default key concurrency setting on the ioqueue, with pj_ioqueue_set_default_concurrency().
• by changing the default concurrency on compile time, by declaring PJ_IOQUEUE_DEFAULT_ALLOW_CONCURRENCY macro to zero in your config_site.h

Examples

For some examples on how to use the I/O Queue, please see:

• Test: I/O Queue (TCP)
• Test: I/O Queue (UDP)
• Test: I/O Queue Performance

Macro Definition Documentation

PJ_IOQUEUE_ALWAYS_ASYNC

#define PJ_IOQUEUE_ALWAYS_ASYNC   ((pj_uint32_t)1 << (pj_uint32_t)31)

When this flag is specified in ioqueue's recv() or send() operations, the ioqueue will always mark the operation as asynchronous.

PJ_IOQUEUE_MAX_CAND_EVENTS

#define PJ_IOQUEUE_MAX_CAND_EVENTS   PJ_IOQUEUE_MAX_EVENTS_IN_SINGLE_POLL

This macro specifies the maximum event candidates collected by each polling thread to be able to reach maximum number of processed events (i.e: PJ_IOQUEUE_MAX_EVENTS_IN_SINGLE_POLL) in each poll cycle. An event candidate will be dispatched to application as event unless it is already being dispatched by other polling thread. So in order to anticipate such race condition, each poll operation should collects its event candidates more than PJ_IOQUEUE_MAX_EVENTS_IN_SINGLE_POLL, the recommended value is (PJ_IOQUEUE_MAX_EVENTS_IN_SINGLE_POLL * number of polling threads).

The value is only meaningfull when specified during PJLIB build and is only effective on multiple polling threads environment.

PJ_IOQUEUE_MAX_EVENTS_IN_SINGLE_POLL

#define PJ_IOQUEUE_MAX_EVENTS_IN_SINGLE_POLL   (16)

This macro specifies the maximum number of events that can be processed by the ioqueue on a single poll cycle, on implementation that supports it. The value is only meaningfull when specified during PJLIB build.

Enumeration Type Documentation

pj_ioqueue_epoll_flag

enum pj_ioqueue_epoll_flag

Epoll flags.

Enumerator
PJ_IOQUEUE_EPOLL_EXCLUSIVE	Use of EPOLLEXCLUSIVE.
PJ_IOQUEUE_EPOLL_ONESHOT	Use of EPOLLONESHOT.
PJ_IOQUEUE_EPOLL_AUTO	Default flag to specify which epoll type to use, which mean to use EPOLLEXCLUSIVE if available, otherwise EPOLLONESHOT, otherwise "bare" epoll when neither are available.

pj_ioqueue_operation_e

enum pj_ioqueue_operation_e

Types of pending I/O Queue operation. This enumeration is only used internally within the ioqueue.

Enumerator
PJ_IOQUEUE_OP_NONE	No operation.
PJ_IOQUEUE_OP_READ	read() operation.
PJ_IOQUEUE_OP_RECV	recv() operation.
PJ_IOQUEUE_OP_RECV_FROM	recvfrom() operation.
PJ_IOQUEUE_OP_WRITE	write() operation.
PJ_IOQUEUE_OP_SEND	send() operation.
PJ_IOQUEUE_OP_SEND_TO	sendto() operation.
PJ_IOQUEUE_OP_ACCEPT	accept() operation.
PJ_IOQUEUE_OP_CONNECT	connect() operation.

Function Documentation

pj_ioqueue_accept()

pj_status_t pj_ioqueue_accept	(	pj_ioqueue_key_t *	key,
		pj_ioqueue_op_key_t *	op_key,
		pj_sock_t *	new_sock,
		pj_sockaddr_t *	local,
		pj_sockaddr_t *	remote,
		int *	addrlen
	)

Instruct I/O Queue to accept incoming connection on the specified listening socket. This function will return immediately (i.e. non-blocking) regardless whether a connection is immediately available. If the function can't complete immediately, the caller will be notified about the incoming connection when it calls pj_ioqueue_poll(). If a new connection is immediately available, the function returns PJ_SUCCESS with the new connection; in this case, the callback WILL NOT be called.

Parameters

key	The key which registered to the server socket.
op_key	An operation specific key to be associated with the pending operation, so that application can keep track of which operation has been completed when the callback is called.
new_sock	Argument which contain pointer to receive the new socket for the incoming connection.
local	Optional argument which contain pointer to variable to receive local address.
remote	Optional argument which contain pointer to variable to receive the remote address.
addrlen	On input, contains the length of the buffer for the address, and on output, contains the actual length of the address. This argument is optional.

Returns

• PJ_SUCCESS When connection is available immediately, and the parameters will be updated to contain information about the new connection. In this case, a completion callback WILL NOT be called.
• PJ_EPENDING If no connection is available immediately. When a new connection arrives, the callback will be called.
• non-zero which indicates the appropriate error code.

pj_ioqueue_cfg_default()

void pj_ioqueue_cfg_default

(

pj_ioqueue_cfg *

cfg

)

Initialize the ioqueue configuration with the default values.

Parameters

cfg	The configuration to be initialized.

pj_ioqueue_clear_key()

pj_status_t pj_ioqueue_clear_key

(

pj_ioqueue_key_t *

key

)

Clear ioqueue key states. This function will cancel any outstanding operations on that key, without invoking any completion callback. After calling this function, application should reinit its all operation keys, i.e: using pj_ioqueue_op_key_init(), before reusing them.

Parameters

key

The key.

Returns

PJ_SUCCESS on success or the appropriate error code.

pj_ioqueue_connect()

pj_status_t pj_ioqueue_connect	(	pj_ioqueue_key_t *	key,
		const pj_sockaddr_t *	addr,
		int	addrlen
	)

Initiate non-blocking socket connect. If the socket can NOT be connected immediately, asynchronous connect() will be scheduled and caller will be notified via completion callback when it calls pj_ioqueue_poll(). If socket is connected immediately, the function returns PJ_SUCCESS and completion callback WILL NOT be called.

Parameters

key	The key associated with TCP socket
addr	The remote address.
addrlen	The remote address length.

Returns

• PJ_SUCCESS If socket is connected immediately. In this case, the completion callback WILL NOT be called.
• PJ_EPENDING If operation is queued, or
• non-zero Indicates the error code.

pj_ioqueue_create()

pj_status_t pj_ioqueue_create	(	pj_pool_t *	pool,
		pj_size_t	max_fd,
		pj_ioqueue_t **	ioqueue
	)

Create a new I/O Queue framework.

Parameters

pool	The pool to allocate the I/O queue structure.
max_fd	The maximum number of handles to be supported, which should not exceed PJ_IOQUEUE_MAX_HANDLES.
ioqueue	Pointer to hold the newly created I/O Queue.

Returns

PJ_SUCCESS on success.

pj_ioqueue_create2()

pj_status_t pj_ioqueue_create2	(	pj_pool_t *	pool,
		pj_size_t	max_fd,
		const pj_ioqueue_cfg *	cfg,
		pj_ioqueue_t **	ioqueue
	)

Create a new I/O Queue framework.

Parameters

pool	The pool to allocate the I/O queue structure.
max_fd	The maximum number of handles to be supported, which should not exceed PJ_IOQUEUE_MAX_HANDLES.
cfg	Optional ioqueue configuration. Application must initialize this structure with pj_ioqueue_cfg_default() first. If this is not specified, default config values as set pj_ioqueue_cfg_default() by will be used.
ioqueue	Pointer to hold the newly created I/O Queue.

Returns

PJ_SUCCESS on success.

pj_ioqueue_destroy()

pj_status_t pj_ioqueue_destroy

(

pj_ioqueue_t *

ioque

)

Destroy the I/O queue.

Parameters

ioque

The I/O Queue to be destroyed.

Returns

PJ_SUCCESS if success.

pj_ioqueue_get_user_data()

void * pj_ioqueue_get_user_data

(

pj_ioqueue_key_t *

key

)

Get user data associated with an ioqueue key.

Parameters

key	The key that was previously obtained from registration.

Returns

The user data associated with the descriptor, or NULL on error or if no data is associated with the key during registration.

pj_ioqueue_is_pending()

pj_bool_t pj_ioqueue_is_pending	(	pj_ioqueue_key_t *	key,
		pj_ioqueue_op_key_t *	op_key
	)

Check if operation is pending on the specified operation key. The op_key must have been initialized with pj_ioqueue_op_key_init() or submitted as pending operation before, or otherwise the result is undefined.

Parameters

key	The key.
op_key	The operation key, previously submitted to any of the I/O functions and has returned PJ_EPENDING.

Returns

Non-zero if operation is still pending.

pj_ioqueue_lock_key()

pj_status_t pj_ioqueue_lock_key

(

pj_ioqueue_key_t *

key

)

Acquire the key's mutex. When the key's concurrency is disabled, application may call this function to synchronize its operation with the key's callback (i.e. this function will block until the key's callback returns).

Parameters

key	The key that was previously obtained from registration.

Returns

PJ_SUCCESS on success or the appropriate error code.

pj_ioqueue_name()

const char * pj_ioqueue_name

(

void

)

Return the name of the ioqueue implementation.

Returns

Implementation name.

pj_ioqueue_op_key_init()

void pj_ioqueue_op_key_init	(	pj_ioqueue_op_key_t *	op_key,
		pj_size_t	size
	)

Initialize operation key.

Parameters

op_key	The operation key to be initialied.
size	The size of the operation key.

pj_ioqueue_poll()

int pj_ioqueue_poll	(	pj_ioqueue_t *	ioque,
		const pj_time_val *	timeout
	)

Poll the I/O Queue for completed events.

Note: polling the ioqueue is not necessary in Symbian. Please see Symbian OS Specific for more info.

Parameters

ioque	the I/O Queue.
timeout	polling timeout, or NULL if the thread wishes to wait indefinetely for the event.

Returns

• zero if timed out (no event).
• (<0) if error occured during polling. Callback will NOT be called.
• (>1) to indicate numbers of events. Callbacks have been called.

pj_ioqueue_post_completion()

pj_status_t pj_ioqueue_post_completion	(	pj_ioqueue_key_t *	key,
		pj_ioqueue_op_key_t *	op_key,
		pj_ssize_t	bytes_status
	)

Post completion status to the specified operation key and call the appropriate callback. When the callback is called, the number of bytes received in read/write callback or the status in accept/connect callback will be set from the bytes_status parameter.

Parameters

key	The key.
op_key	Pending operation key.
bytes_status	Number of bytes or status to be set. A good value to put here is -PJ_ECANCELLED.

Returns

PJ_SUCCESS if completion status has been successfully sent.

pj_ioqueue_recv()

pj_status_t pj_ioqueue_recv	(	pj_ioqueue_key_t *	key,
		pj_ioqueue_op_key_t *	op_key,
		void *	buffer,
		pj_ssize_t *	length,
		pj_uint32_t	flags
	)

Instruct the I/O Queue to read from the specified handle. This function returns immediately (i.e. non-blocking) regardless whether some data has been transferred. If the operation can't complete immediately, caller will be notified about the completion when it calls pj_ioqueue_poll(). If data is immediately available, the function will return PJ_SUCCESS and the callback WILL NOT be called.

Parameters

key	The key that uniquely identifies the handle.
op_key	An operation specific key to be associated with the pending operation, so that application can keep track of which operation has been completed when the callback is called. Caller must make sure that this key remains valid until the function completes.
buffer	The buffer to hold the read data. The caller MUST make sure that this buffer remain valid until the framework completes reading the handle.
length	On input, it specifies the size of the buffer. If data is available to be read immediately, the function returns PJ_SUCCESS and this argument will be filled with the amount of data read. If the function is pending, caller will be notified about the amount of data read in the callback. This parameter can point to local variable in caller's stack and doesn't have to remain valid for the duration of pending operation.
flags	Recv flag. If flags has PJ_IOQUEUE_ALWAYS_ASYNC then the function will never return PJ_SUCCESS.

Returns

• PJ_SUCCESS If immediate data has been received in the buffer. In this case, the callback WILL NOT be called.
• PJ_EPENDING If the operation has been queued, and the callback will be called when data has been received.
• non-zero The return value indicates the error code.

pj_ioqueue_recvfrom()

pj_status_t pj_ioqueue_recvfrom	(	pj_ioqueue_key_t *	key,
		pj_ioqueue_op_key_t *	op_key,
		void *	buffer,
		pj_ssize_t *	length,
		pj_uint32_t	flags,
		pj_sockaddr_t *	addr,
		int *	addrlen
	)

This function behaves similarly as pj_ioqueue_recv(), except that it is normally called for socket, and the remote address will also be returned along with the data. Caller MUST make sure that both buffer and addr remain valid until the framework completes reading the data.

Parameters

key	The key that uniquely identifies the handle.
op_key	An operation specific key to be associated with the pending operation, so that application can keep track of which operation has been completed when the callback is called.
buffer	The buffer to hold the read data. The caller MUST make sure that this buffer remain valid until the framework completes reading the handle.
length	On input, it specifies the size of the buffer. If data is available to be read immediately, the function returns PJ_SUCCESS and this argument will be filled with the amount of data read. If the function is pending, caller will be notified about the amount of data read in the callback. This parameter can point to local variable in caller's stack and doesn't have to remain valid for the duration of pending operation.
flags	Recv flag. If flags has PJ_IOQUEUE_ALWAYS_ASYNC then the function will never return PJ_SUCCESS.
addr	Optional Pointer to buffer to receive the address.
addrlen	On input, specifies the length of the address buffer. On output, it will be filled with the actual length of the address. This argument can be NULL if addr is not specified.

Returns

• PJ_SUCCESS If immediate data has been received. In this case, the callback must have been called before this function returns, and no pending operation is scheduled.
• PJ_EPENDING If the operation has been queued.
• non-zero The return value indicates the error code.

pj_ioqueue_register_sock()

pj_status_t pj_ioqueue_register_sock	(	pj_pool_t *	pool,
		pj_ioqueue_t *	ioque,
		pj_sock_t	sock,
		void *	user_data,
		const pj_ioqueue_callback *	cb,
		pj_ioqueue_key_t **	key
	)

Register a socket to the I/O queue framework. When a socket is registered to the IOQueue, it may be modified to use non-blocking IO. If it is modified, there is no guarantee that this modification will be restored after the socket is unregistered.

Parameters

pool	To allocate the resource for the specified handle, which must be valid until the handle/key is unregistered from I/O Queue.
ioque	The I/O Queue.
sock	The socket.
user_data	User data to be associated with the key, which can be retrieved later.
cb	Callback to be called when I/O operation completes.
key	Pointer to receive the key to be associated with this socket. Subsequent I/O queue operation will need this key.

Returns

PJ_SUCCESS on success, or the error code.

pj_ioqueue_register_sock2()

pj_status_t pj_ioqueue_register_sock2	(	pj_pool_t *	pool,
		pj_ioqueue_t *	ioque,
		pj_sock_t	sock,
		pj_grp_lock_t *	grp_lock,
		void *	user_data,
		const pj_ioqueue_callback *	cb,
		pj_ioqueue_key_t **	key
	)

Variant of pj_ioqueue_register_sock() with additional group lock parameter. If group lock is set for the key, the key will add the reference counter when the socket is registered and decrease it when it is destroyed.

pj_ioqueue_send()

pj_status_t pj_ioqueue_send	(	pj_ioqueue_key_t *	key,
		pj_ioqueue_op_key_t *	op_key,
		const void *	data,
		pj_ssize_t *	length,
		pj_uint32_t	flags
	)

Instruct the I/O Queue to write to the handle. This function will return immediately (i.e. non-blocking) regardless whether some data has been transferred. If the function can't complete immediately, the caller will be notified about the completion when it calls pj_ioqueue_poll(). If operation completes immediately and data has been transferred, the function returns PJ_SUCCESS and the callback will NOT be called.

Parameters

key	The key that identifies the handle.
op_key	An operation specific key to be associated with the pending operation, so that application can keep track of which operation has been completed when the callback is called.
data	The data to send. Caller MUST make sure that this buffer remains valid until the write operation completes.
length	On input, it specifies the length of data to send. When data was sent immediately, this function returns PJ_SUCCESS and this parameter contains the length of data sent. If data can not be sent immediately, an asynchronous operation is scheduled and caller will be notified via callback the number of bytes sent. This parameter can point to local variable on caller's stack and doesn't have to remain valid until the operation has completed.
flags	Send flags. If flags has PJ_IOQUEUE_ALWAYS_ASYNC then the function will never return PJ_SUCCESS.

Returns

• PJ_SUCCESS If data was immediately transferred. In this case, no pending operation has been scheduled and the callback WILL NOT be called.
• PJ_EPENDING If the operation has been queued. Once data base been transferred, the callback will be called.
• non-zero The return value indicates the error code.

pj_ioqueue_sendto()

pj_status_t pj_ioqueue_sendto	(	pj_ioqueue_key_t *	key,
		pj_ioqueue_op_key_t *	op_key,
		const void *	data,
		pj_ssize_t *	length,
		pj_uint32_t	flags,
		const pj_sockaddr_t *	addr,
		int	addrlen
	)

Instruct the I/O Queue to write to the handle. This function will return immediately (i.e. non-blocking) regardless whether some data has been transferred. If the function can't complete immediately, the caller will be notified about the completion when it calls pj_ioqueue_poll(). If operation completes immediately and data has been transferred, the function returns PJ_SUCCESS and the callback will NOT be called.

Parameters

key	the key that identifies the handle.
op_key	An operation specific key to be associated with the pending operation, so that application can keep track of which operation has been completed when the callback is called.
data	the data to send. Caller MUST make sure that this buffer remains valid until the write operation completes.
length	On input, it specifies the length of data to send. When data was sent immediately, this function returns PJ_SUCCESS and this parameter contains the length of data sent. If data can not be sent immediately, an asynchronous operation is scheduled and caller will be notified via callback the number of bytes sent. This parameter can point to local variable on caller's stack and doesn't have to remain valid until the operation has completed.
flags	send flags. If flags has PJ_IOQUEUE_ALWAYS_ASYNC then the function will never return PJ_SUCCESS.
addr	Optional remote address.
addrlen	Remote address length, addr is specified.

Returns

• PJ_SUCCESS If data was immediately written.
• PJ_EPENDING If the operation has been queued.
• non-zero The return value indicates the error code.

pj_ioqueue_set_concurrency()

pj_status_t pj_ioqueue_set_concurrency	(	pj_ioqueue_key_t *	key,
		pj_bool_t	allow
	)

Configure whether the ioqueue is allowed to call the key's callback concurrently/in parallel. The default concurrency setting for the key is controlled by ioqueue's default concurrency value, which can be changed by calling pj_ioqueue_set_default_concurrency().

If concurrency is allowed for the key, it means that if there are more than one pending operations complete simultaneously, more than one threads may call the key's callback at the same time. This generally would promote good scalability for application, at the expense of more complexity to manage the concurrent accesses in application's code.

Alternatively application may disable the concurrent access by setting the allow flag to false. With concurrency disabled, only one thread can call the key's callback at one time.

Parameters

key	The key that was previously obtained from registration.
allow	Set this to non-zero to allow concurrent callback calls and zero (PJ_FALSE) to disallow it.

Returns

PJ_SUCCESS on success or the appropriate error code.

pj_ioqueue_set_default_concurrency()

pj_status_t pj_ioqueue_set_default_concurrency	(	pj_ioqueue_t *	ioqueue,
		pj_bool_t	allow
	)

Set default concurrency policy for this ioqueue. If this function is not called, the default concurrency policy for the ioqueue is controlled by compile time setting PJ_IOQUEUE_DEFAULT_ALLOW_CONCURRENCY.

Note that changing the concurrency setting to the ioqueue will only affect subsequent key registrations. To modify the concurrency setting for individual key, use pj_ioqueue_set_concurrency().

Parameters

ioqueue	The ioqueue instance.
allow	Non-zero to allow concurrent callback calls, or PJ_FALSE to disallow it.

Returns

PJ_SUCCESS on success or the appropriate error code.

pj_ioqueue_set_lock()

pj_status_t pj_ioqueue_set_lock	(	pj_ioqueue_t *	ioque,
		pj_lock_t *	lock,
		pj_bool_t	auto_delete
	)

Set the lock object to be used by the I/O Queue. This function can only be called right after the I/O queue is created, before any handle is registered to the I/O queue.

Initially the I/O queue is created with non-recursive mutex protection. Applications can supply alternative lock to be used by calling this function.

Parameters

ioque	The ioqueue instance.
lock	The lock to be used by the ioqueue.
auto_delete	In non-zero, the lock will be deleted by the ioqueue.

Returns

PJ_SUCCESS or the appropriate error code.

pj_ioqueue_set_user_data()

pj_status_t pj_ioqueue_set_user_data	(	pj_ioqueue_key_t *	key,
		void *	user_data,
		void **	old_data
	)

Set or change the user data to be associated with the file descriptor or handle or socket descriptor.

Parameters

key	The key that was previously obtained from registration.
user_data	User data to be associated with the descriptor.
old_data	Optional parameter to retrieve the old user data.

Returns

PJ_SUCCESS on success or the error code.

pj_ioqueue_trylock_key()

pj_status_t pj_ioqueue_trylock_key

(

pj_ioqueue_key_t *

key

)

Try to acquire the key's mutex. When the key's concurrency is disabled, application may call this function to synchronize its operation with the key's callback.

Parameters

key	The key that was previously obtained from registration.

Returns

PJ_SUCCESS on success or the appropriate error code.

pj_ioqueue_unlock_key()

pj_status_t pj_ioqueue_unlock_key

(

pj_ioqueue_key_t *

key

)

Release the lock previously acquired with pj_ioqueue_lock_key().

Parameters

key	The key that was previously obtained from registration.

Returns

PJ_SUCCESS on success or the appropriate error code.

pj_ioqueue_unregister()

pj_status_t pj_ioqueue_unregister

(

pj_ioqueue_key_t *

key

)

Unregister from the I/O Queue framework. Caller must make sure that the key doesn't have any pending operations before calling this function, by calling pj_ioqueue_is_pending() for all previously submitted operations except asynchronous connect, and if necessary call pj_ioqueue_post_completion() to cancel the pending operations.

Note that asynchronous connect operation will automatically be cancelled during the unregistration.

Also note that when I/O Completion Port backend is used, application MUST close the handle immediately after unregistering the key. This is because there is no unregistering API for IOCP. The only way to unregister the handle from IOCP is to close the handle.

Parameters

key	The key that was previously obtained from registration.

Returns

PJ_SUCCESS on success or the error code.

See also

pj_ioqueue_is_pending