Bug #497
Updated by Daniel Curtis over 9 years ago
I came across an error with one of my MariaDB Galera database nodes failing to join after a peroid of time being offline.
* I checked the log file:
<pre>
tail -n 50 /var/log/syslog
</pre>
#* And found the following error:
<pre>
2014-05-31 04:01:40 2391 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (02dc9b38-e825-11e3-a67b-4bfddfde0c98): 1 (Operation not permitted)
at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable.
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20140531 04:01:40.934)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20140531 04:01:40.936)
2014-05-31 04:01:40 2391 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.0.2' --auth 'test_sstuser:123456' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '2391' '' : 32 (Broken pipe)
2014-05-31 04:01:40 2391 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2014-05-31 04:01:40 2391 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2014-05-31 04:01:40 2391 [ERROR] Aborting
2014-05-31 04:01:41 2391 [Warning] WSREP: 1.0 (test.host.com): State transfer to 0.0 (master-2.test.net) failed: -32 (Broken pipe)
2014-05-31 04:01:41 2391 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
</pre>
After looking around I found the solution, which was relatively simple.
* Edit the my.cnf file and add the following to the [mysqld] section
<pre>
vi /etc/mysql/my.cnf
</pre>
#* And add/modify the following
<pre>
[mysqld]
innodb_log_file_size=48M
</pre>
* Now clear the Galera log files (or else the node will fail joining the cluster due to the InnoDB log file size changing):
<pre>
rm /var/lib/mysql/ib_logfile*
</pre>
h2. Resources
* https://stackoverflow.com/questions/23966539/percona-xtradbcluster-error-while-getting-data-from-donor-node
* http://serverfault.com/questions/104014/innodb-error-log-file-ib-logfile0-is-of-different-size