Distributed System-Communication and Invocation-RPC and Socket

This post, we concentrate on communication and invocation as part of the implementation. Invocation at here could be RPC,  RMI,  Event Notification etc… locally or remotely. In this post, we mostly study about RPC and RMI…for others you can easily research on google. In another hand, we also try to give the answer for the question “what wrong with socket? or why do we not use socket?”

Also, the following questions should be clarified at the end of this post by yourself.

1. What communication primitives does it supply?
2. Which protocol does it support and how open is the communication implementation?
3. What steps are taken to make communication as efficient as possible ?
4. What support is provided for high latency and disconnected operation?

First of all, we will go through “Communication Primitives” concept

Communication Primitives: Some kernels designed for distributed systems have provided communication primitives tailored to the types of invocation for example, provides doOperation, getRequest and sendReply as primitives.

Placing relatively high-level communication functionality in the kernel has the advantage of efficiency. If, for example, middleware provides RMI over UNIX’s connected (TCP) sockets, then a client must make two communication system calls (socket write and read) for each remote invocation. Over Amoeba (the V system and Chorus provide group communication primitives), it would require only a single call to doOperation. The savings in system call overhead are liable to be even greater with group communication

In practice, middleware, and not the kernel, provides most high-level communication facilities found in systems today, including RPC/RMI, event notification and group communication

Midleware typically is developed over sockets giving access to Internet standard protocol -Often connected socket using TCP (sometimes UDP socket). The principle reason for using socket are portability and inter-operability : Midleware is required to operate over as many widely used operating system as possible, all common operating system are providing similar socket api giving access to TCP and UDP protocols.

what’s wrong with sockets?

Sockets are a fundamental part of client-server networking. They provide a relatively easy mechanism for a program to establish a connection to another program, either on a remote or local machine and send messages back and forth (we can even use read and write system calls). This interface, however, forces us to design our distributed applications using a read/write (input/output) interface which is not how we generally design non-distributed applications. In designing centralized applications, the procedure call is usually the standard interface model. If we want to make distributed computing look like centralized computing, input/output-based communications is not the way to accomplish this.

Invocation performance:

This is critical factor in distributed system design.

RPC and RMI implementations have been the subject of study because of the widespread acceptance of these mechanisms for general-purpose client-server processing. Much of research has been carried out into invocation over network, and particularly into how invocation mechanism can take advantage of high-performance networks.

schematically, client delay against requested data size. The delay is roughly proportional to the size until the size reaches a threshold at about network packet size. Beyond that threshold, at least one extra packet has to be sent, to carry the extra data. Depending on the protocol, a further packet might be used to acknowledge this extra packet. Jumps in the graph occur each time the number of packets increases.

Delay is not the only figure of interest for an RPC implementation: RPC throughput (or bandwidth) is also of concern when data has to be transferred in bulk.

Marshalling: Marshalling and unmarshalling, which involve copying and converting
data, create a significant overhead as the amount of data grows.

Data copying: Potentially, even after marshalling, message data is copied several
times in the course of an RPC:
1. across the user–kernel boundary, between the client or server address space and
kernel buffers;
2. across each protocol layer (for example, RPC/UDP/IP/Ethernet);
3. between the network interface and kernel buffers.

Transfers between the network interface and main memory are usually handled by
direct memory access (DMA). The processor handles the other copies.

Invocation within a computer(LRPC)

A client stub examines a bit set at bind time that records whether the server is local or remote, and proceeds to use LRPC or RPC, respectively. The application is unaware of which is used. However, migration transparency might be hard to achieve when a resource is transferred from a local server to a remote server or vice versa, because of the need to change invocation mechanisms.

Figure above suggests that a cross-address-space invocation is implemented within a computer exactly as it is between computers, except that the underlying message passing happens to be local. Indeed, this has often been the model implemented. Bershad et al. developed a more efficient invocation mechanism for the case of two processes on the same machine called lightweight RPC (LRPC). The LRPC design is based on optimizations concerning data copying and thread scheduling.

First, they noted that it would be more efficient to use shared memory regions for client-server communication, with a different (private) region between the server and each of its local clients. Such a region contains one or more A (for argument) stacks Instead of RPC parameters being copied between the kernel and user address spaces involved, the client and server are able to pass arguments and return values directly via an A stack. The same stack is used by the client and server stubs. In LRPC, arguments are copied once: when they are marshalled onto the A stack. In an equivalent RPC, they are copied four times: from the client stub’s stack onto a message; from the message to a kernel buffer, from the kernel buffer to a server message, and from the message to the server stub’s stack. There may be several A stacks in a shared region, because several threads in the same client may call the server at the same time.

Bershad et al. also considered the cost of thread scheduling. Compare the model of system call and remote procedure calls in Figure 7.11. When a system call occurs, most kernels do not schedule a new thread to handle the call but instead perform a context switch on the calling thread so that it handles the system call. In an RPC, a remote procedure may exist in a different computer from the client thread, so a different thread must be scheduled to execute it. In the local case, however, it may be more efficient for the client thread – which would otherwise be BLOCKED – to call the invoked procedure in the server’s address space.
A server must be programmed differently in this case to the way we have described servers before. Instead of setting up one or more threads, which then listen on ports for invocation requests, the server exports a set of procedures that it is prepared to have called. Threads in local processes may enter the server’s execution environment as long as they start by calling one of the server’s exported procedures. A client needing to invoke a server’s operations must first bind to the server interface (not shown in the figure). It does this via the kernel, which notifies the server; when the server has responded to the kernel with a list of allowed procedure addresses, the kernel replies to the client with a capability for invoking the server’s operations.
An invocation is shown in Figure 7.13. A client thread enters the server’s execution environment by first trapping to the kernel and presenting it with a capability.

The kernel checks this and only allows a context switch to a valid server procedure; if it is valid, the kernel switches the thread’s context to call the procedure in the server’s execution environment. When the procedure in the server returns, the thread returns to the kernel, which switches the thread back to the client execution environment. Note that clients and servers employ stub procedures to hide the details just described from application writers.

Invocation remotely in different computers

Note that in below figure: 

step 2: Network messages are sent by the client stub to the remote system (via a system call to the local kernel using sockets interfaces).

Step 3: Network messages are transferred by the kernel to the remote system via some protocol (either connectionless or connection-oriented).

The major benefits of RPC are twofold.

1. First, the programmer can now use procedure call semantics to invoke remote functions and get responses.
2. Secondly, writing distributed applications is simplified because RPC hides all of the network code into stub functions. Application programs don’t have to worry about details such as sockets, port numbers, and data conversion and parsing. On the OSI reference model, RPC spans both the session and presentation layers (layers five and six).