Feature #265
Updated by Daniel Curtis over 10 years ago
I have decided to add a memcache layer to the web services provided by the ALT VPS Infrastructure. To accomplish this, I have created an instance of memcached (more correctly repcached) on each node, possibly in a circular replication pattern if supported, then have the memcached instances load-balanced. This will have the benefit of increased performance and availability, with minimal restructuring of current web applications and the potential to utilize memcached for increased MySQL performance.
h2. Installing repcache
First you need to visit http://repcached.lab.klab.org/ and download the lastest version (latest at time of writing: 2.2-1.2.8). After downloading the tar file, we will need to install some dependencies:
<pre>
apt-get install memcached libevent-dev g++ make
</pre>
From here we can continue the installation:
<pre>
tar xvf memcached-1.2.8-repcached-2.2.tar
cd memcached-1.2.8-repcached-2.2/
./configure --enable-replication
make
make install
</pre>
h2. Configuring repcache
At this point you have two installations of memcached. Default memcached that came from apt packages, which is installed in @/usr/bin/memcached@ and repcached, that installed itself in @/usr/local/bin/memcached@, leaving the original memcached intact.
Now that we have both versions installed, we can copy memcached's default settings and init script and modify them to use repcached. This way you can quickly switch between versions. I would even recommend using default ports (just remember to firewall them!) Arguments are saved in @/etc/memcached.conf@, so we will create *@/etc/repcached.conf@*
<pre>
cp /etc/memcached.conf /etc/repcached.conf
vi /etc/repcached.conf
</pre>
> DAEMON_ARGS="-m 128 -p 11211 -u root -P /var/run/repcached.pid -d -x 172.16.100.2"
Note that the only differences with memcached.conf is the name (repcached) and two extra arguments, make sure to at least define the @-x@ arugment:
# *-x* for the server IP of the master memcached service to replicate
# *-X* for replication port
Memcached has an enable/disable config in @/etc/default@ so you can quickly switch between daemons or disable them.
Copy the system defaults
<pre>
cp /etc/default/memcached /etc/default/repcached
vi /etc/default/repcached
</pre>
Change the line to:
> @*ENABLE_REPCACHED=yes*@
Then edit @/etc/default/memcached@"
<pre>
vi /etc/default/memcached
</pre>
and disable it, by changing the line to
> ENABLE_MEMCACHED=no.
Create an init script
<pre>
vi /etc/init.d/repcached
</pre>
Then add:
<pre>
#!/usr/bin/perl -w
# start-repcached
# 2011 - Jean Caffou <jean@briskula.si>
# This script handles the parsing of the /etc/repcached.conf file
# and was originally created for the Debian distribution.
# Anyone may use this little script under the same terms as
# memcached itself.
use strict;
if($> != 0 and $< != 0)
{
print STDERR "Only root wants to run start-repcached.\n";
exit;
}
my $params; my $etchandle; my $etcfile = "/etc/repcached.conf";
# This script assumes that repcached is located at /usr/local/bin/memcached, and
# that the pidfile is writable at /var/run/repcached.pid
my $memcached = "/usr/local/bin/memcached";
my $pidfile = "/var/run/repcached.pid";
# If we don't get a valid logfile parameter in the /etc/repcached.conf file,
# we'll just throw away all of our in-daemon output.
my $fd_reopened = "/dev/null";
sub handle_logfile
{
my ($logfile) = @_;
$fd_reopened = $logfile;
}
sub reopen_logfile
{
my ($logfile) = @_;
open *STDERR, ">>$logfile";
open *STDOUT, ">>$logfile";
open *STDIN, ">>/dev/null";
$fd_reopened = $logfile;
}
# This is set up in place here to support other non -[a-z] directives
my $conf_directives = {
"logfile" => \&handle_logfile,
};
if(open $etchandle, $etcfile)
{
foreach my $line (<$etchandle>)
{
$line ||= "";
$line =~ s/\#.*//g;
$line =~ s/\s+$//g;
$line =~ s/^\s+//g;
next unless $line;
next if $line =~ /^\-[dh]/;
if($line =~ /^[^\-]/)
{
my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/;
$conf_directives->{$directive}->($arg);
next;
}
push @$params, $line;
}
}else{
$params = [];
}
push @$params, "-u root" unless(grep "-u", @$params);
$params = join " ", @$params;
if(-e $pidfile)
{
open PIDHANDLE, "$pidfile";
my $localpid = <PIDHANDLE>;
close PIDHANDLE;
chomp $localpid;
if(-d "/proc/$localpid")
{
print STDERR "repcached is already running.\n";
exit;
}else{
`rm -f $localpid`;
}
}
my $pid = fork();
if($pid == 0)
{
reopen_logfile($fd_reopened);
exec "$memcached $params";
exit(0);
}else{
if(open PIDHANDLE,">$pidfile")
{
print PIDHANDLE $pid;
close PIDHANDLE;
}else{
print STDERR "Can't write pidfile to $pidfile.\n";
}
}
</pre>
h3. Setting up repcached to start at boot
We need to be sure that @/etc/init.d/repcached@ is executable. If you copied it from memcached, everything should be OK, but if init's not recognising the repcached service, you need to
<pre>
chmod +x /etc/init.d/repcached
</pre>
After you've run update-rc.d command in the terminal it will create shortcuts in rc?.d files which are read at boot:
<pre>
update-rc.d repcached defaults
</pre>
You have successfully configured repcached as a service and to start on boot.
To start/stop repcached use:
<pre>
service repcached start
service repcached stop
</pre>
Try to run repcached by hand at first with the configuration you provided in @/etc/repcached.conf@.
In my example it's this:
<pre>
/usr/local/bin/memcached -m 64 -p 11211 -u memcache -X 11212 -x 10.11.22.33
</pre>
After installing repcached on another machine I've found out that the default user for memcached is *nobody*, not *memcache*, so please always check the differences from the default memcache config with the repcached config you've modified or copied from here.
h2. Testing
Before we move on we will test that the 2 nodes are replicating.
* *server1*:
<pre>
telnet 127.0.0.1 11211
</pre>
Then type in:
<pre>
set foo 0 0 3
bar
</pre>
You should see the word STORED appear under it. Good:
<pre>
quit
</pre>
and it will return to the console. Go to your other server and type in the following.
* *server2*:
<pre>
telnet 127.0.0.1 11211
</pre>
Then type in
<pre>
get foo
</pre>
and it should return the value you entered on the first server. If it has then the replication is working.
h2. Troubleshooting
I encountered an error while compiling repcache:
> memcached.c: In function ‘add_iov’:
> memcached.c:697:30: error: ‘IOV_MAX’ undeclared (first use in this function)
> memcached.c:697:30: note: each undeclared identifier is reported only once for each function it appears in
Luckily enough, someone else had this problem and posted a fix on the developers forum, "here":http://dev.kafol.net/2012/03/repcached-does-not-compile.html
In memcached.c code, we can see that IOV_MAX is defined if the OS is FreeBSD or iOS, so I decided to just define it anyway.
Here is my diff of the code that made it compile, in case you need it.
<pre>
57,59c57
< #if defined(__FreeBSD__) || defined(__APPLE__)
< # define IOV_MAX 1024
< #endif
---
> #define IOV_MAX 1024
</pre>
So I just needed to remove the IF condition and define the IOV_MAX.
-I also had a problem after installing the modified memcached server where the init script would display an error, preventing the server from starting:-
<pre>
/usr/local/bin/memcached: invalid option -- '-'
Illegal argument "?"
</pre>
I had forgotten to add the @--enable-replication@ argument while configuring the software.
h2. Resources
http://dev.kafol.net/2011/03/configuring-repcached-service-on.html
http://dev.kafol.net/2012/03/repcached-does-not-compile.html
http://www.howtoforge.com/how-to-install-repcached-memcached-replication-for-high-availability-over-2-nodes-on-ubuntu-11.04