GNet Solutions » GNU/Linux Administration

I have decided to add a memcache layer to the web services provided by the ALT VPS Infrastructure. To accomplish this, I have created an instance of memcached (more correctly repcached) on each node, possibly in a circular replication pattern if supported, then have the memcached instances load-balanced. This will have the benefit of increased performance and availability, with minimal restructuring of current web applications and the potential to utilize memcached for increased MySQL performance.

Installing repcache¶

First you need to visit http://repcached.lab.klab.org/ and download the lastest version (latest at time of writing: 2.2-1.2.8). After downloading the tar file, we will need to install some dependencies:

apt-get install memcached libevent-dev g++ make

From here we can continue the installation:

tar xvf memcached-1.2.8-repcached-2.2.tar
cd memcached-1.2.8-repcached-2.2/
./configure --enable-replication
make
make install

Configuring repcache¶

At this point you have two installations of memcached. Default memcached that came from apt packages, which is installed in /usr/bin/memcached and repcached, that installed itself in /usr/local/bin/memcached, leaving the original memcached intact.

Now that we have both versions installed, we can copy memcached's default settings and init script and modify them to use repcached. This way you can quickly switch between versions. I would even recommend using default ports (just remember to firewall them!) Arguments are saved in /etc/memcached.conf, so we will create /etc/repcached.conf

cp /etc/memcached.conf /etc/repcached.conf
vi /etc/repcached.conf

DAEMON_ARGS="-m 128 -p 11211 -u root -P /var/run/repcached.pid -d -x 172.16.100.2"

Note that the only differences with memcached.conf is the name (repcached) and two extra arguments, make sure to at least define the -x arugment:

1. -x for the server IP of the master memcached service to replicate
2. -X for replication port

Memcached has an enable/disable config in /etc/default so you can quickly switch between daemons or disable them.

Copy the system defaults

cp /etc/default/memcached /etc/default/repcached
vi /etc/default/repcached

Change the line to:

*ENABLE_REPCACHED=yes*

Then edit /etc/default/memcached"

vi /etc/default/memcached

and disable it, by changing the line to

ENABLE_MEMCACHED=no.

Create an init script

vi /etc/init.d/repcached

Then add:

#!/usr/bin/perl -w

# start-repcached
# 2011 - Jean Caffou <jean@briskula.si>
# This script handles the parsing of the /etc/repcached.conf file
# and was originally created for the Debian distribution.
# Anyone may use this little script under the same terms as
# memcached itself.

use strict;

if($> != 0 and $< != 0)
{
    print STDERR "Only root wants to run start-repcached.\n";
    exit;
}

my $params; my $etchandle; my $etcfile = "/etc/repcached.conf";

# This script assumes that repcached is located at /usr/local/bin/memcached, and
# that the pidfile is writable at /var/run/repcached.pid

my $memcached = "/usr/local/bin/memcached";
my $pidfile = "/var/run/repcached.pid";

# If we don't get a valid logfile parameter in the /etc/repcached.conf file,
# we'll just throw away all of our in-daemon output.
my $fd_reopened = "/dev/null";

sub handle_logfile
{
    my ($logfile) = @_;
    $fd_reopened = $logfile;
}

sub reopen_logfile
{
    my ($logfile) = @_;

    open *STDERR, ">>$logfile";
    open *STDOUT, ">>$logfile";
    open *STDIN, ">>/dev/null";
    $fd_reopened = $logfile;
}

# This is set up in place here to support other non -[a-z] directives

my $conf_directives = {
    "logfile" => \&handle_logfile,
};

if(open $etchandle, $etcfile)
{
    foreach my $line (<$etchandle>)
    {
        $line ||= "";
        $line =~ s/\#.*//g;
        $line =~ s/\s+$//g;
        $line =~ s/^\s+//g;
        next unless $line;
        next if $line =~ /^\-[dh]/;

        if($line =~ /^[^\-]/)
        {
            my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/;
            $conf_directives->{$directive}->($arg);
            next;
        }

        push @$params, $line;
    }

}else{
    $params = [];
}

push @$params, "-u root" unless(grep "-u", @$params);
$params = join " ", @$params;

if(-e $pidfile)
{
    open PIDHANDLE, "$pidfile";
    my $localpid = <PIDHANDLE>;
    close PIDHANDLE;

    chomp $localpid;
    if(-d "/proc/$localpid")
    {
        print STDERR "repcached is already running.\n";
        exit;
    }else{
        `rm -f $localpid`;
    }

}

my $pid = fork();

if($pid == 0)
{
    reopen_logfile($fd_reopened);
    exec "$memcached $params";
    exit(0);

}else{
    if(open PIDHANDLE,">$pidfile")
    {
        print PIDHANDLE $pid;
        close PIDHANDLE;
    }else{

        print STDERR "Can't write pidfile to $pidfile.\n";
    }
}

Setting up repcached to start at boot¶

We need to be sure that /etc/init.d/repcached is executable. If you copied it from memcached, everything should be OK, but if init's not recognising the repcached service, you need to

chmod +x /etc/init.d/repcached

After you've run update-rc.d command in the terminal it will create shortcuts in rc?.d files which are read at boot:

update-rc.d repcached defaults

You have successfully configured repcached as a service and to start on boot.

To start/stop repcached use:

service repcached start
service repcached stop

Try to run repcached by hand at first with the configuration you provided in /etc/repcached.conf.
In my example it's this:

/usr/local/bin/memcached -m 64 -p 11211 -u memcache -X 11212 -x 10.11.22.33

After installing repcached on another machine I've found out that the default user for memcached is nobody, not memcache, so please always check the differences from the default memcache config with the repcached config you've modified or copied from here.

Testing¶

Before we move on we will test that the 2 nodes are replicating.

• server1:
  

  telnet 127.0.0.1 11211
  

Then type in:

set foo 0 0 3
bar

You should see the word STORED appear under it. Good:

quit

and it will return to the console. Go to your other server and type in the following.

• server2:
  

  telnet 127.0.0.1 11211
  

Then type in

get foo

and it should return the value you entered on the first server. If it has then the replication is working.

Troubleshooting¶

I encountered an error while compiling repcache:

memcached.c: In function ‘add_iov’:
memcached.c:697:30: error: ‘IOV_MAX’ undeclared (first use in this function)
memcached.c:697:30: note: each undeclared identifier is reported only once for each function it appears in

Luckily enough, someone else had this problem and posted a fix on the developers forum, here

In memcached.c code, we can see that IOV_MAX is defined if the OS is FreeBSD or iOS, so I decided to just define it anyway.

Here is my diff of the code that made it compile, in case you need it.

57,59c57
< #if defined(__FreeBSD__) || defined(__APPLE__)
< # define IOV_MAX 1024
< #endif
---
> #define IOV_MAX 1024

So I just needed to remove the IF condition and define the IOV_MAX.

I also had a problem after installing the modified memcached server where the init script would display an error, preventing the server from starting:

/usr/local/bin/memcached: invalid option -- '-'
Illegal argument "?" 

I had forgotten to add the --enable-replication argument while configuring the software.

Resources¶

http://dev.kafol.net/2011/03/configuring-repcached-service-on.html
http://dev.kafol.net/2012/03/repcached-does-not-compile.html
http://www.howtoforge.com/how-to-install-repcached-memcached-replication-for-high-availability-over-2-nodes-on-ubuntu-11.04