2015-06-06

Inter process communication by a shared memory

I made an experience for an inter process communication using a Unix shared memory on a host. I look up on the Internet, but I could not find a good example code. In the end, I found some explanation articles, some Stack overflow threads. (I might do something wrong when I search...)

Some pages suggest to use shmem_open() and mmap(), that is the POSIX direction. I think I can believe the discussion, so I implemented it the following way.

The basic idea is the following:
  1. A process creates an shared memory area using shm_open().  To identify the shared memory area between processes, we use an identifier string (e.g., ``identifier''). (On Linux environment, You can see via /dev/shmem, e.g., /dev/shmem/identifier .)
  2. All processes map that shared memory are to main memory using mmap(). When we succeeded mmap(), we obtained a shared memory pointer (shared_ptr).
  3. The processes can communicate via shared_ptr.
  4. When we finished to use the shared_ptr, remove the map using munmap().
  5. One of the process removes the shared memory object using shm_unlink().
The following implementation shows how two processes (server and client) communicate each other via a shared memory area. Here, the number of server processes and the number of client processes are both one. In this example, each process only writes a value to own area. Thus, we don't need a lock. Each process reads the other process's value and write one grater value of the read value at the own area. This is simple, but enough to demonstrate how to communicate between processes via a shared memory on a host. 

The following is the source code (shmem_test.cpp). You can find how to compile and run in the comment of the code.

/*
  Shared memory inter process communication minimal example
  Copyright (C) 2015 Hitoshi

  Compile:
    g++ shmem_test.cpp -o shmem_test -lrt

  Run (as a server):
    shmem_test server

  Run (as a client)
    shmem_test client

  Note: no locking. (but writes are not the same location.)

  Basic idea: server, shared memory creator

  1. Create a shared memory object by shm_open().
  2. Change the shared memory object by ftruncate().
  3. mmap the shared memory object to access via a pointer.
  4. Use the pointer to share the memory (may need lock, etc.)
  5. munmap the shared memory object
  6. Remove the shared memory object by shm_unlink()

  Basic idea: client

  1. Open the created shared memory object by shm_open().
  2. mmap the shared memory object to access via a pointer.
  3. Use the pointer to share the memory (may need lock, etc.)
  4. munmap the shared memory object

 */

#include <iostream>

#include <fcntl.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>

//----------------------------------------------------------------------
/// shard memory identifier name
const char* SHMEM_NAME = "/test.shmem";
/// shared memory size (align to the page size)
const size_t SHMEM_SIZE = 4096;

//----------------------------------------------------------------------
/// print out the usage and exit of this example
void usage_exit()
{
    std::cerr
        << "Shard memory inter process communication minimal test\n"
        << "shmem_test [server|client]\n"
        << "  server ... create/read/write shared memory\n"
        << "  client ... read/write shared memory"
        << std::endl;
    exit(1);
}

//----------------------------------------------------------------------
/// server
void run_server()
{
    // create/open the shared memory object
    int fd = shm_open(SHMEM_NAME,
                      // oflags
                      O_CREAT|  // create if not exist
                      // O_TRUNK|  // destroy once if exists
                      O_EXCL|   // error if exists
                      O_RDWR    // read/write
                      ,
                      // mode
                      S_IRUSR|  // user read
                      S_IWUSR   // user write
        );

    if (fd >= 0)
    {
        std::cout << "server: shmem_open succeed fd: " << fd << std::endl;
    }
    else
    {
        std::cerr << "server: shmem_open failed: " << fd
                  << ", if /dev/shm"    << SHMEM_NAME
                  << " exists, remove it." << std::endl;
        exit(1);
    }

    // resize the shmem object
    const int ret = ftruncate(fd, SHMEM_SIZE);
    if (ret != 0)
    {
        std::cerr << "Server: failed to ftrancate to " << SHMEM_SIZE
                  << std::endl;
        exit(1);
    }

    // get shmem address
    const size_t access_offset = 0;
    void* shmem_adder = mmap(0, SHMEM_SIZE,
                             // memory protection mode
                             PROT_READ| // Pages may be read.
                             PROT_WRITE // Pages may be written.
                             ,
                             // mapping flag
                             MAP_SHARED, // Share this mapping with other process
                             fd,
                             access_offset);
    if (shmem_adder == 0)
    {
        std::cerr << "Serevr failed to mmap." << std::endl;
        exit(1);
    }
    close(fd);                 // fd is no longer needed

    // server process work
    int* int_ptr = reinterpret_cast<int*>(shmem_adder);

    // initialize
    int_ptr[0] = 0;
    int_ptr[1] = 0;

    for (int i = 0; i < 10; ++i)
    {
        int my_int    = int_ptr[0];
        int other_int = int_ptr[1];

        if (my_int <= other_int)
        {
            my_int = other_int + 1;
            int_ptr[0] = my_int;
            std::cout << "I am taller. " << my_int << ", "
                      << other_int << std::endl;
        }
        usleep(800000);
        std::cout << "Checking client " << i << std::endl;
    }
    std::cout << "Quit server: " << int_ptr[0] << ", " << int_ptr[1] << std::endl;

    // unmap
    int ummap_ret = munmap(shmem_adder, SHMEM_SIZE);
    if (ummap_ret != 0)
    {
        std::cerr << "Failed to munmap. " << ummap_ret << std::endl;
        exit(1);
    }

    // remove shmem object
    int unlink_ret = shm_unlink(SHMEM_NAME);
    if (unlink_ret != 0)
    {
        std::cerr << "Failed to shm_unlink. " << unlink_ret << std::endl;
        exit(1);
    }
}

//----------------------------------------------------------------------
void run_client()
{
    // open the shared memory object
    int fd = shm_open(SHMEM_NAME,
                      // oflags
                      O_RDWR    // read/write
                      ,
                      // mode
                      S_IRUSR|  // user read
                      S_IWUSR   // user write
        );

    if (fd >= 0)
    {
        std::cout << "shmem_open succeed fd: " << fd << std::endl;
    }
    else
    {
        std::cerr << "shmem_open failed: " << fd
                  << ", Are you running the server?" << std::endl;
        exit(1);
    }

    // get shmem address
    const size_t access_offset = 0;
    void* shmem_adder = mmap(0, SHMEM_SIZE,
                             // memory protection mode
                             PROT_READ| // Pages may be read.
                             PROT_WRITE // Pages may be written.
                             ,
                             // mapping flag
                             MAP_SHARED, // Share this mapping with other process
                             fd,
                             access_offset);
    if (shmem_adder == 0)
    {
        std::cerr << "Failed to mmap." << std::endl;
        exit(1);
    }
    close(fd);                 // fd is no longer needed

    // server process work
    int* int_ptr = reinterpret_cast<int*>(shmem_adder);

    // initialize
    int_ptr[0] = 0;
    int_ptr[1] = 0;

    for (int i = 0; i < 10; ++i)
    {
        int my_int    = int_ptr[1];
        int other_int = int_ptr[0];

        if (my_int <= other_int)
        {
            my_int = other_int + 1;
            int_ptr[1] = my_int;
            std::cout << "I am taller. " << my_int << ", "
                      << other_int << std::endl;
        }
        usleep(500000);
        std::cout << "Checking server " << i << std::endl;
    }
    std::cout << "Quit client: " << int_ptr[0] << ", " << int_ptr[1] << std::endl;

    // unmap
    int ummap_ret = munmap(shmem_adder, SHMEM_SIZE);
    if (ummap_ret != 0)
    {
        std::cerr << "Failed to munmap. " << ummap_ret << std::endl;
        exit(1);
    }
}

//----------------------------------------------------------------------
/// main
int main(int argc, char* argv[])
{
    if (argc == 1)
    {
        usage_exit();
    }
    else if ((argc == 2) && std::string(argv[1]) == "server")
    {
        run_server();
    }
    else if ((argc == 2) && std::string(argv[1]) == "client")
    {
        run_client();
    }
    else
    {
        usage_exit();
    }

    return 0;
}

I hope someone can find this example useful.

No comments: