GNet Solutions » FreeBSD Administration

Support #891

Updated by Daniel Curtis over 7 years ago

This is a guide for setting up Apache Kafka on FreeBSD 10. 

 h2. Prepare the Environment 

 * Make sure the system is up to date: 
 <pre> 
 pkg update && pkg upgrade 
 </pre> 

 * Install a few dependencies: 
 <pre> 
 pkg install bash git gradle openjdk7 sudo 
 </pre> 

 * Create a symlink of bash: 
 <pre> 
 ln -s /usr/local/bin/bash /bin/bash 
 </pre> 

 * Install Zookeeper and ensure its running, refer to issue #890 
 <pre> 
 service zookeeper start 
 </pre> 

 * Create the kafka user: 
 <pre> 
 pw useradd -n kafka -m -c "Kafka" -s /bin/sh 
 </pre> 

 h2. Install Kafka 

 * Download Kafka from GitHub: 
 <pre> 
 git clone https://github.com/apache/kafka.git /usr/local/kafka 
 </pre> 

 * Bootstrap Kafka: 
 <pre> 
 cd /usr/local/kafka 
 gradle 
 </pre> 

 * Build the Kafka jar: 
 <pre> 
 ./gradlew jar 
 </pre> 

 * Create log file directory for Kafka: 
 <pre> 
 mkdir /var/log/kafka 
 </pre> 

 * Set the kafka user permissions: 
 <pre> 
 chown -R kafka /usr/local/kafka 
 chown -R kafka /var/log/kafka 
 </pre> 

 * Edit the Kafka config: 
 <pre> 
 vi /usr/local/kafka/config/server.properties 
 </pre> 
 #* And modify the following parameters: 
 <pre> 
 # The id of the broker. This must be set to a unique integer for each broker. 
 broker.id=1 

 # Increase the message size limit 
 message.max.bytes=20000000 
 replica.fetch.max.bytes=30000000 

 log.dirs=/var/log/kafka 
 listeners=PLAINTEXT://192.168.1.104:9092 

 zookeeper.connect=localhost:2181 
 #zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181 
 </pre> 

 * Test start the Kafka server: 
 <pre> 
 /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties 
 </pre> 
 #* Once finished testing press CTRL+C to stop the kafka server. 

 h3. Init Script 

 * Create a kafka init script: 
 <pre> 
 vi /usr/local/etc/rc.d/kafka 
 </pre> 
 #* And add the following: 
 <pre> 
 #!/bin/sh 
 # 
 # PROVIDE: kafka 
 # REQUIRE: zookeeper LOGIN 
 # 
 # Add the following lines to /etc/rc.conf to enable odoo server 
 # 
 # 
 # kafka_enable (bool): Set to "NO" by default, 
 #                           Set it to "YES" to enable kafka odoo server 
 # 
 # kafka_config (str):    The path to the kafka server configuration file 
 #                           (defaults to /usr/local/kafka/config/server.properties) 
 # 
 # kafka_user (str):      The user to run kafka server as. Set to "root" by default 

 . /etc/rc.subr 

 name=kafka 
 command=/usr/local/kafka/bin/kafka-server-start.sh 
 rcvar=kafka_enable 

 load_rc_config $name 

 kafka_enable="${kafka_enable-"NO"}"  
 kafka_config="${kafka_config-"/usr/local/kafka/config/server.properties"}"  
 kafka_user="${kafka_user-"root"}"   
 kafka_flags="${kafka-"${kafka_config}"}"  

 # /etc/rc.subr use $pidfile (not ${name}_pidfile) 
 pidfile="${kafka_pidfile}"  

 required_files="${kafka_config}"  

 start_cmd="su - ${kafka_user} -c '${command} ${kafka_flags}' &"  
 stop_cmd="${name}_stop"  
 status_cmd="${name}_status"  
 getval_cmd="${name}_getval"  

 kafka_stop() 
 { 
     kafka_pid=$(pgrep -f "kafka") 

     echo "Stopping ${name}."  
     pkill -TERM -f ${name} kill -s TERM "$(cat "${kafka_pid}")"  
 } 

 kafka_status() 
 { 
     kafka_pid=$(pgrep -f ${name}) "kafka") 

     if [ -n "${kafka_pid}" ] 
     then 
         echo "${name} running with pid: $kafka_pid"  
     else 
         echo "${name} not running? (pid not found)"  
     fi 
 } 

 command_args=" >/dev/null 2>&1 &"  

 load_rc_config $name 
 run_rc_command "$1" 
 </pre>  

 * Make the script executable: 
 <pre> 
 chmod +x /usr/local/etc/rc.d/kafka 
 </pre> 

 * Start and enable kafka at boot: 
 <pre> 
 echo 'kafka_enable="YES"' >> /etc/rc.conf 
 echo 'kafka_user="kafka"' >> /etc/rc.conf 
 service kafka start 
 </pre> 

 h2. Resources 

 * https://github.com/apache/kafka 
 * http://kafka.apache.org/documentation.html#quickstart 
 * https://github.com/btccom/btcpool/blob/master/docs/INSTALL-Kafka.md

Back