GNet Solutions » GNU/Linux Administration

I came across an error with one of my MariaDB Galera database nodes failing to join after a peroid of time being offline.

• I checked the log file:
  

  tail -n 50 /var/log/syslog
  

  • And found the following error:
    

    2014-05-31 04:01:40 2391 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (02dc9b38-e825-11e3-a67b-4bfddfde0c98): 1 (Operation not permitted)
     at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable.
    WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20140531 04:01:40.934)
    WSREP_SST: [ERROR] Cleanup after exit with status:32 (20140531 04:01:40.936)
    2014-05-31 04:01:40 2391 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.0.2' --auth 'test_sstuser:123456' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --parent '2391'  '' : 32 (Broken pipe)
    2014-05-31 04:01:40 2391 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
    2014-05-31 04:01:40 2391 [ERROR] WSREP: SST failed: 32 (Broken pipe)
    2014-05-31 04:01:40 2391 [ERROR] Aborting
    2014-05-31 04:01:41 2391 [Warning] WSREP: 1.0 (test.host.com): State transfer to 0.0 (master-2.test.net) failed: -32 (Broken pipe)
    2014-05-31 04:01:41 2391 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
    

After looking around I found the solution, which was relatively simple.

• Edit the my.cnf file and add the following to the [mysqld] section
  

  vi /etc/mysql/my.cnf
  

  • And add/modify the following
    

    [mysqld]
    innodb_log_file_size=48M
    

• Now clear the Galera log files (or else the node will fail joining the cluster due to the InnoDB log file size changing):
  

  rm /var/lib/mysql/ib_logfile*
  

Resources¶

• https://stackoverflow.com/questions/23966539/percona-xtradbcluster-error-while-getting-data-from-donor-node
• http://serverfault.com/questions/104014/innodb-error-log-file-ib-logfile0-is-of-different-size