Feature #265
Updated by Daniel Curtis almost 11 years ago
I have decided to add a memcache layer to the web services provided by the ALT VPS Infrastructure. To accomplish this, I have created an instance of memcached (more correctly repcached) on each node, possibly in a circular replication pattern if supported, then have the memcached instances load-balanced. This will have the benefit of increased performance and availability, with minimal restructuring of current web applications and the potential to utilize memcached for increased MySQL performance. h2. Installing repcache First you need to visit http://repcached.lab.klab.org/ and download the lastest version (latest at time of writing: 2.2-1.2.8). After downloading the tar file, we will need to install some dependencies: <pre> apt-get install memcached libevent-dev g++ make </pre> From here we can continue the installation: <pre> tar xvf memcached-1.2.8-repcached-2.2.tar cd memcached-1.2.8-repcached-2.2/ ./configure --enable-replication make make install </pre> h2. Configuring repcache At this point you have two installations of memcached. Default memcached that came from apt packages, which is installed in @/usr/bin/memcached@ and repcached, that installed itself in @/usr/local/bin/memcached@, leaving the original memcached intact. Now that we have both versions installed, we can copy memcached's default settings and init script and modify them to use repcached. This way you can quickly switch between versions. I would even recommend using default ports (just remember to firewall them!) Arguments are saved in @/etc/memcached.conf@, so we will create *@/etc/repcached.conf@* <pre> cp /etc/memcached.conf /etc/repcached.conf </pre> Note that the only differences with memcached.conf is the name (repcached) and two extra arguments, make sure to at least define the @-x@ arugment: arguments: # *-x* for the server IP # *-X* for replication port. Memcached has an enable/disable config in @/etc/default@ so you can quickly switch between daemons or disable them. Copy the system defaults <pre> cp /etc/default/memcached /etc/default/repcached vi /etc/default/repcached </pre> Change the line to: > @*ENABLE_REPCACHED=yes*@ Then edit @/etc/default/memcached@" <pre> vi /etc/default/memcached </pre> and disable it, by changing the line to > ENABLE_MEMCACHED=no. Create an Move on to init script scripts. <pre> vi /etc/init.d/repcached cd /etc/init.d cp memcached repcached </pre> Edit the file @/etc/init.d/repcached@ Then add: <pre> #!/usr/bin/perl -w # start-repcached # 2011 - Jean Caffou <jean@briskula.si> # This script handles the parsing of the /etc/repcached.conf file # and was originally created for the Debian distribution. # Anyone may use this little script under the same terms as # memcached itself. use strict; if($> != 0 and $< != 0) { print STDERR "Only root wants to run start-repcached.\n"; exit; } my $params; my $etchandle; my $etcfile = "/etc/repcached.conf"; # This script assumes that repcached is located at /usr/local/bin/memcached, and # that the pidfile is writable at /var/run/repcached.pid my $memcached = "/usr/local/bin/memcached"; my $pidfile = "/var/run/repcached.pid"; # If we don't get a valid logfile parameter in the /etc/repcached.conf file, # we'll just throw away all of our in-daemon output. my $fd_reopened = "/dev/null"; sub handle_logfile { my ($logfile) = @_; $fd_reopened = $logfile; } sub reopen_logfile { my ($logfile) = @_; open *STDERR, ">>$logfile"; open *STDOUT, ">>$logfile"; open *STDIN, ">>/dev/null"; $fd_reopened = $logfile; } # This is set up in place here to support other non -[a-z] directives my $conf_directives = { "logfile" => \&handle_logfile, }; if(open $etchandle, $etcfile) { foreach my $line (<$etchandle>) { $line ||= ""; $line =~ s/\#.*//g; $line =~ s/\s+$//g; $line =~ s/^\s+//g; next unless $line; next if $line =~ /^\-[dh]/; if($line =~ /^[^\-]/) { my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/; $conf_directives->{$directive}->($arg); next; } push @$params, $line; } }else{ $params = []; } push @$params, "-u root" unless(grep "-u", @$params); $params = join " ", @$params; if(-e $pidfile) { open PIDHANDLE, "$pidfile"; my $localpid = <PIDHANDLE>; close PIDHANDLE; chomp $localpid; if(-d "/proc/$localpid") { print STDERR "repcached is already running.\n"; exit; }else{ `rm -f $localpid`; } } my $pid = fork(); if($pid == 0) { reopen_logfile($fd_reopened); exec "$memcached $params"; exit(0); }else{ if(open PIDHANDLE,">$pidfile") { print PIDHANDLE $pid; close PIDHANDLE; }else{ print STDERR "Can't write pidfile to $pidfile.\n"; } } </pre> h3. Setting up repcached to start at boot We need to be sure that @/etc/init.d/repcached@ is executable. If you copied it from memcached, everything should be OK, but if init's not recognising the repcached service, you need to <pre> chmod +x /etc/init.d/repcached </pre> After you've run update-rc.d command in the terminal it will create shortcuts in rc?.d files which are read at boot: <pre> update-rc.d repcached defaults </pre> You have successfully configured repcached as a service and to start on boot. To start/stop repcached use: <pre> service repcached start service repcached stop </pre> Try to run repcached by hand at first with the configuration you provided in @/etc/repcached.conf@. In my example it's this: <pre> /usr/local/bin/memcached -m 64 -p 11211 -u memcache -X 11212 -x 10.11.22.33 </pre> After installing repcached on another machine I've found out that the default user for memcached is *nobody*, not *memcache*, so please always check the differences from the default memcache config with the repcached config you've modified or copied from here. h2. Troubleshooting I encountered an error while compiling repcache: > memcached.c: In function ‘add_iov’: > memcached.c:697:30: error: ‘IOV_MAX’ undeclared (first use in this function) > memcached.c:697:30: note: each undeclared identifier is reported only once for each function it appears in Luckily enough, someone else had this problem and posted a fix on the developers forum, "here":http://dev.kafol.net/2012/03/repcached-does-not-compile.html In memcached.c code, we can see that IOV_MAX is defined if the OS is FreeBSD or iOS, so I decided to just define it anyway. Here is my diff of the code that made it compile, in case you need it. <pre> 57,59c57 < #if defined(__FreeBSD__) || defined(__APPLE__) < # define IOV_MAX 1024 < #endif --- > #define IOV_MAX 1024 </pre> So I just needed to remove the IF condition and define the IOV_MAX. I also had a problem after installing the modified memcached server where the init script would display an error, preventing the server from starting: <pre> /usr/local/bin/memcached: invalid option -- '-' Illegal argument "?" </pre> I had forgotten to add the @--enable-replication@ argument while configuring the software. h2. Resources http://dev.kafol.net/2011/03/configuring-repcached-service-on.html http://dev.kafol.net/2012/03/repcached-does-not-compile.html http://www.howtoforge.com/how-to-install-repcached-memcached-replication-for-high-availability-over-2-nodes-on-ubuntu-11.04