Feature #265
Replicated Memcached Instances For High Availability
Description
I have decided to add a memcache layer to the web services provided by the ALT VPS Infrastructure. To accomplish this, I have created an instance of memcached (more correctly repcached) on each node, possibly in a circular replication pattern if supported, then have the memcached instances load-balanced. This will have the benefit of increased performance and availability, with minimal restructuring of current web applications and the potential to utilize memcached for increased MySQL performance.
Installing repcache¶
First you need to visit http://repcached.lab.klab.org/ and download the lastest version (latest at time of writing: 2.2-1.2.8). After downloading the tar file, we will need to install some dependencies:
apt-get install memcached libevent-dev g++ make
From here we can continue the installation:
tar xvf memcached-1.2.8-repcached-2.2.tar cd memcached-1.2.8-repcached-2.2/ ./configure --enable-replication make make install
Configuring repcache¶
At this point you have two installations of memcached. Default memcached that came from apt packages, which is installed in /usr/bin/memcached
and repcached, that installed itself in /usr/local/bin/memcached
, leaving the original memcached intact.
Now that we have both versions installed, we can copy memcached's default settings and init script and modify them to use repcached. This way you can quickly switch between versions. I would even recommend using default ports (just remember to firewall them!) Arguments are saved in /etc/memcached.conf
, so we will create /etc/repcached.conf
cp /etc/memcached.conf /etc/repcached.conf vi /etc/repcached.conf
Note that the only differences with memcached.conf is the name (repcached) and two extra arguments, make sure to at least define theDAEMON_ARGS="-m 128 -p 11211 -u root -P /var/run/repcached.pid -d -x 172.16.100.2"
-x
arugment:
- -x for the server IP of the master memcached service to replicate
- -X for replication port
Memcached has an enable/disable config in /etc/default
so you can quickly switch between daemons or disable them.
Copy the system defaults
cp /etc/default/memcached /etc/default/repcached vi /etc/default/repcached
Change the line to:
*ENABLE_REPCACHED=yes*
Then edit /etc/default/memcached
"
vi /etc/default/memcached
and disable it, by changing the line to
ENABLE_MEMCACHED=no.
Create an init script
vi /etc/init.d/repcached
Then add:
#!/usr/bin/perl -w # start-repcached # 2011 - Jean Caffou <jean@briskula.si> # This script handles the parsing of the /etc/repcached.conf file # and was originally created for the Debian distribution. # Anyone may use this little script under the same terms as # memcached itself. use strict; if($> != 0 and $< != 0) { print STDERR "Only root wants to run start-repcached.\n"; exit; } my $params; my $etchandle; my $etcfile = "/etc/repcached.conf"; # This script assumes that repcached is located at /usr/local/bin/memcached, and # that the pidfile is writable at /var/run/repcached.pid my $memcached = "/usr/local/bin/memcached"; my $pidfile = "/var/run/repcached.pid"; # If we don't get a valid logfile parameter in the /etc/repcached.conf file, # we'll just throw away all of our in-daemon output. my $fd_reopened = "/dev/null"; sub handle_logfile { my ($logfile) = @_; $fd_reopened = $logfile; } sub reopen_logfile { my ($logfile) = @_; open *STDERR, ">>$logfile"; open *STDOUT, ">>$logfile"; open *STDIN, ">>/dev/null"; $fd_reopened = $logfile; } # This is set up in place here to support other non -[a-z] directives my $conf_directives = { "logfile" => \&handle_logfile, }; if(open $etchandle, $etcfile) { foreach my $line (<$etchandle>) { $line ||= ""; $line =~ s/\#.*//g; $line =~ s/\s+$//g; $line =~ s/^\s+//g; next unless $line; next if $line =~ /^\-[dh]/; if($line =~ /^[^\-]/) { my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/; $conf_directives->{$directive}->($arg); next; } push @$params, $line; } }else{ $params = []; } push @$params, "-u root" unless(grep "-u", @$params); $params = join " ", @$params; if(-e $pidfile) { open PIDHANDLE, "$pidfile"; my $localpid = <PIDHANDLE>; close PIDHANDLE; chomp $localpid; if(-d "/proc/$localpid") { print STDERR "repcached is already running.\n"; exit; }else{ `rm -f $localpid`; } } my $pid = fork(); if($pid == 0) { reopen_logfile($fd_reopened); exec "$memcached $params"; exit(0); }else{ if(open PIDHANDLE,">$pidfile") { print PIDHANDLE $pid; close PIDHANDLE; }else{ print STDERR "Can't write pidfile to $pidfile.\n"; } }
Setting up repcached to start at boot¶
We need to be sure that /etc/init.d/repcached
is executable. If you copied it from memcached, everything should be OK, but if init's not recognising the repcached service, you need to
chmod +x /etc/init.d/repcached
After you've run update-rc.d command in the terminal it will create shortcuts in rc?.d files which are read at boot:
update-rc.d repcached defaults
You have successfully configured repcached as a service and to start on boot.
To start/stop repcached use:
service repcached start service repcached stop
Try to run repcached by hand at first with the configuration you provided in /etc/repcached.conf
.
In my example it's this:
/usr/local/bin/memcached -m 64 -p 11211 -u memcache -X 11212 -x 10.11.22.33
After installing repcached on another machine I've found out that the default user for memcached is nobody, not memcache, so please always check the differences from the default memcache config with the repcached config you've modified or copied from here.
Testing¶
Before we move on we will test that the 2 nodes are replicating.
- server1:
telnet 127.0.0.1 11211
Then type in:
set foo 0 0 3 bar
You should see the word STORED appear under it. Good:
quit
and it will return to the console. Go to your other server and type in the following.
- server2:
telnet 127.0.0.1 11211
Then type in
get foo
and it should return the value you entered on the first server. If it has then the replication is working.
Troubleshooting¶
I encountered an error while compiling repcache:
memcached.c: In function ‘add_iov’:
memcached.c:697:30: error: ‘IOV_MAX’ undeclared (first use in this function)
memcached.c:697:30: note: each undeclared identifier is reported only once for each function it appears in
Luckily enough, someone else had this problem and posted a fix on the developers forum, here
In memcached.c code, we can see that IOV_MAX is defined if the OS is FreeBSD or iOS, so I decided to just define it anyway.
Here is my diff of the code that made it compile, in case you need it.
57,59c57 < #if defined(__FreeBSD__) || defined(__APPLE__) < # define IOV_MAX 1024 < #endif --- > #define IOV_MAX 1024
So I just needed to remove the IF condition and define the IOV_MAX.
I also had a problem after installing the modified memcached server where the init script would display an error, preventing the server from starting:
/usr/local/bin/memcached: invalid option -- '-' Illegal argument "?"
I had forgotten to add the --enable-replication
argument while configuring the software.
Resources¶
http://dev.kafol.net/2011/03/configuring-repcached-service-on.html
http://dev.kafol.net/2012/03/repcached-does-not-compile.html
http://www.howtoforge.com/how-to-install-repcached-memcached-replication-for-high-availability-over-2-nodes-on-ubuntu-11.04
Files