GNet Solutions » FreeBSD Administration

Support #891

Updated by Daniel Curtis over 7 years ago

This is a guide for setting up Apache Kafka on FreeBSD 10. 

 h2. Prepare the Environment 

 * Make sure the system is up to date: 
 <pre> 
 pkg update && pkg upgrade 
 </pre> 

 * Install a few dependencies: 
 <pre> 
 pkg install bash git gradle openjdk7 
 </pre> 

 * Create a symlink of bash: 
 <pre> 
 ln -s /usr/local/bin/bash /bin/bash 
 </pre> 

 * Install Zookeeper and ensure its running, refer to issue #890 
 <pre> 
 service zookeeper start 
 </pre> 

 * Download Kafka from GitHub: 
 <pre> 
 git clone https://github.com/apache/kafka.git /usr/local/kafka 
 </pre> 

 * Bootstrap Kafka: 
 <pre> 
 cd /usr/local/kafka 
 gradle 
 </pre> 

 * Build the Kafka jar: 
 <pre> 
 ./gradlew jar 
 </pre> 

 * Create log file directory for Kafka: 
 <pre> 
 mkdir /var/log/kafka 
 </pre> 

 * Edit the Kafka config: 
 <pre> 
 vi /usr/local/kafka/config/server.properties 
 </pre> 
 #* And modify the following parameters: 
 <pre> 
 # The id of the broker. This must be set to a unique integer for each broker. 
 broker.id=1 

 # Increase the message size limit 
 message.max.bytes=20000000 
 replica.fetch.max.bytes=30000000 

 log.dirs=/var/log/kafka 
 listeners=PLAINTEXT://192.168.1.104:9092 

 zookeeper.connect=localhost:2181 
 #zookeeper.connect=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181 
 </pre> 

 * Test start the Kafka server: 
 <pre> 
 /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties 
 </pre> 
 #* Once finished testing press CTRL+C to stop the kafka server. 

 h3. Init Script System service 

 * Install supervisor: 
 <pre> 
 pkg install py27-supervisor 
 </pre> 

 * Create a kafka init script: supervisor config directory: 
 <pre> 
 vi /usr/local/etc/rc.d/kafka mkdir /usr/local/etc/supervisord.conf.d 
 </pre> 

 * Add a folder inclusion statement: 
 #* And add the following: 
 <pre> 
 #!/bin/sh echo '[include]' >> /usr/local/etc/supervisord.conf 
 # echo 'files = supervisord.conf.d/*.ini' >> /usr/local/etc/supervisord.conf 
 # PROVIDE: </pre> 

 * Create a kafka supervisord config: 
 # REQUIRE: zookeeper LOGIN <pre> 
 # vi /usr/local/etc/supervisord.conf.d/kafka.ini 
 # Add </pre> 
 #* And add the following lines to /etc/rc.conf to enable odoo server following: 
 # <pre> 
 # [program:kafka] 
 # kafka_enable (bool): Set to "NO" by default, directory=/usr/local/kafka 
 #                           Set it to "YES" to enable odoo server 
 # 
 # kafka_config (str):    The path to the kafka server configuration file 
 #                           (defaults to /usr/local/kafka/config/server.properties) 

 . /etc/rc.subr 

 name=kafka 
 command=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties 
 rcvar=kafka_enable 

 load_rc_config $name 

 kafka_enable="${kafka_enable-"NO"}"  
 kafka_config="${kafka_config-"/usr/local/kafka/config/server.properties"}"  
 kafka_user="${kafka_user-"root"}"   
 kafka_flags="${kafka-"${kafka_config}"}"  

 # /etc/rc.subr use $pidfile (not ${name}_pidfile) autostart=true 
 pidfile="${kafka_pidfile}"  

 required_files="${kafka_config}"  

 start_cmd="su - ${kafka_user} -c '${command} ${kafka_flags}' &"  
 stop_cmd="${name}_stop"  
 status_cmd="${name}_status"  
 getval_cmd="${name}_getval"  

 kafka_stop() autorestart=true 
 { 
     kafka_pid=$(pgrep -f "kafka") 

     echo "Stopping ${name}."  
     kill -s TERM "$(cat "${kafka_pid}")"  
 } 

 kafka_status() startsecs=6 
 { 
     kafka_pid=$(pgrep -f "kafka") 

     if [ -n "${kafka_pid}" ] 
     then 
         echo "${name} running with pid: $kafka_pid"  
     else 
         echo "${name} not running? (pid not found)"  
     fi startretries=20 
 } 

 command_args=" >/dev/null 2>&1 &"  

 load_rc_config $name 
 run_rc_command "$1" 
 </pre>  

 * Make the script executable: 
 <pre> 
 chmod +x /usr/local/etc/rc.d/kafka 
 </pre> 

 * Start and enable kafka supervisord at boot: 
 <pre> 
 echo 'kafka_enable="YES"' 'supervisord_enable="YES"' >> /etc/rc.conf 
 service kafka supervisord start 
 </pre> 

 * Run the supervisord control prompt: 
 <pre> 
 supervisorctl 
 </pre> 
 #*  

 h2. Resources 

 * https://github.com/apache/kafka 
 * http://kafka.apache.org/documentation.html#quickstart 
 * https://github.com/btccom/btcpool/blob/master/docs/INSTALL-Kafka.md *https://github.com/btccom/btcpool/blob/master/docs/INSTALL-Kafka.md

Back