Support #737
Updated by Daniel Curtis almost 9 years ago
This is a guide on using nginx to create a reverse proxy package cache for pkgng and ports FreeBSD packages.
h2. Prepare the Environment
* Make sure the system is up to date:
<pre>
pkg update && pkg upgrade
</pre>
h2. Install Nginx
* Install nginx:
<pre>
pkg install nginx
</pre>
* Start and enable nginx at boot:
<pre>
echo 'nginx_enable="YES"' >> /etc/rc.conf
service nginx start
</pre>
* Create a configuration directory to make managing individual server blocks easier
<pre>
mkdir /usr/local/etc/nginx/conf.d
</pre>
* Edit the main nginx config file:
<pre>
vi /usr/local/etc/nginx/nginx.conf
</pre>
#* And strip down the config file and add the include statement at the end to make it easier to handle various server blocks:
<pre>
worker_processes 1;
error_log /var/log/nginx-error.log;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
# nginx may need to resolve domain names at run time
resolver 208.67.222.222 208.67.220.220;
# Load config files from the /etc/nginx/conf.d directory
include /usr/local/etc/nginx/conf.d/*.conf;
}
</pre>
h2. Caching Pkgng
h3. Pkgng Nginx Cache Config
* Create the directory for the cache and adjust the permissions so nginx can write files to it:
<pre>
mkdir /var/cache/pkgmirror
chown www:www /var/cache/pkgmirror
</pre>
* Configure nginx as our dynamic cache:
<pre>
vi /etc/nginx/conf.d/pkgmirror.example.com.conf
</pre>
#* And add the following:
<pre>
server {
listen 80;
server_name pkgmirror.example.com;
root /var/cache/pkgmirror;
autoindex on;
# Requests for actual packages should be served directly from cache if available.
# If not available, retrieve and save the package from an upstream mirror.
location ~ \.txz$ {
try_files $uri @pkg_mirror;
}
# Retrieve package from upstream mirrors and cache for future requests
location @pkg_mirror {
proxy_store on;
proxy_redirect off;
proxy_cache_revalidate on;
proxy_cache_min_uses 3;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
proxy_store_access user:rw group:rw all:r;
proxy_next_upstream error timeout http_404;
proxy_pass http://pkg-mirrors$request_uri;
}
}
# Upstream FreeBSD Mirrors
upstream pkg-mirrors {
server localhost:8001;
server localhost:8002 backup;
server localhost:8003 backup;
}
# FreeBSD Global Mirror Proxy Configuration
server {
listen 8001;
server_name localhost;
location / {
proxy_pass http://pkg.FreeBSD.org$request_uri;
proxy_set_header Host pkg.FreeBSD.org;
}
}
# FreeBSD US West Mirror Proxy Configuration
server {
listen 8002;
server_name localhost;
location / {
proxy_pass http://pkg.us-west.FreeBSD.org$request_uri;
proxy_set_header Host pkg.us-west.FreeBSD.org;
}
}
# FreeBSD US East Mirror Proxy Configuration
server {
listen 8003;
server_name localhost;
location / {
proxy_pass http://pkg.us-east.FreeBSD.org$request_uri;
proxy_set_header Host pkg.us-east.FreeBSD.org;
}
}
</pre>
* Restart nginx:
<pre>
service restart nginx
</pre>
h3. Update System Repo
* Edit the pkg repo config file:
<pre>
vi /usr/local/etc/pkg/repos/FreeBSD.conf
</pre>
* Add the following line to use this new cache
<pre>
FreeBSD: {
url: "pkg+http://pkgmirror.example.com/freebsd:9:x86:64/latest",
mirror_type: "srv",
enabled: yes
}
</pre>
h3. Cache Cleaning
* The following command will clean the package cache:
<pre>
setenv PKG_CACHEDIR /var/cache/pkgmirror
pkg clean
</pre>
h2. Caching the Ports Tree
h3. Pkgng Nginx Cache Config
* Create the directory for the cache and adjust the permissions so nginx can write files to it:
<pre>
mkdir /var/cache/portsmirror
chown www:www /var/cache/portsmirror
</pre>
* Configure nginx as our dynamic cache:
<pre>
vi /etc/nginx/conf.d/portsmirror.example.com.conf
</pre>
#* And add the following:
<pre>
# nginx may need to resolve domain names at run time
server {
listen 80;
server_name portsmirror.example.com;
root /var/cache/portsmirror;
autoindex on;
# Requests for actual packages should be served directly from cache if available.
# If not available, retrieve and save the package from an upstream mirror.
location / {
try_files $uri @ports_mirror;
}
# Retrieve package from upstream mirrors and cache for future requests
location @ports_mirror {
proxy_store on;
proxy_redirect off;
proxy_cache_revalidate on;
proxy_cache_min_uses 3;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
proxy_store_access user:rw group:rw all:r;
proxy_next_upstream error timeout http_404;
proxy_pass http://ports-mirrors$request_uri;
}
}
# Upstream FreeBSD Mirrors
upstream ports-mirrors {
server localhost:8004;
server localhost:8005 backup;
server localhost:8006 backup;
}
# FreeBSD Global Mirror Proxy Configuration
server {
listen 8004;
server_name localhost;
location / {
proxy_pass http://your-org.portsnap.freebsd.org$request_uri;
proxy_set_header Host your-org.portsnap.freebsd.org;
}
}
# FreeBSD US West Mirror Proxy Configuration
server {
listen 8005;
server_name localhost;
location / {
proxy_pass http://sourcefire.portsnap.freebsd.org$request_uri;
proxy_set_header Host sourcefire.portsnap.freebsd.org;
}
}
# FreeBSD US East Mirror Proxy Configuration
server {
listen 8006;
server_name localhost;
location / {
proxy_pass http://isc.portsnap.freebsd.org$request_uri;
proxy_set_header Host isc.portsnap.freebsd.org;
}
}
</pre>
* Restart nginx:
<pre>
service restart nginx
</pre>
h3. Download Ports Tree
* Edit the portsnap config file:
<pre>
vi /etc/portsnap
</pre>
#* And change the SERVERNAME variable to the local proxy:
<pre>
SERVERNAME=portsmirror.example.com
</pre>
* Set the http_proxy environment variable:
<pre>
setenv http_proxy http://portsmirror.example.com
</pre>
* Then fetch the latest ports tree:
<pre>
portsnap fetch extract
</pre>
h3. FreeBSD Install Cache Config
* Create the directory for the cache and adjust the permissions so nginx can write files to it:
<pre>
mkdir /var/cache/freebsdmirror
chown www:www /var/cache/freebsdmirror
</pre>
* Configure nginx as our dynamic cache:
<pre>
vi /etc/nginx/conf.d/freebsdmirror.example.com.conf
</pre>
#* And add the following:
<pre>
server {
listen 80;
server_name freebsdmirror.example.com;
root /var/cache/freebsdmirror;
autoindex on;
# Requests for actual packages should be served directly from cache if available.
# If not available, retrieve and save the package from an upstream mirror.
location ~ \.txz$ {
try_files $uri @freebsd_mirror;
}
# Retrieve package from upstream mirrors and cache for future requests
location @freebsd_mirror {
proxy_store on;
proxy_redirect off;
proxy_cache_revalidate on;
proxy_cache_min_uses 3;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
proxy_store_access user:rw group:rw all:r;
proxy_next_upstream error timeout http_404;
proxy_pass http://freebsd-mirrors$request_uri;
}
}
# Upstream FreeBSD Mirrors
upstream freebsd-mirrors {
server localhost:8013;
server localhost:8014 backup;
server localhost:8015 backup;
}
# FreeBSD Global Mirror Proxy Configuration
server {
listen 8013;
server_name localhost;
location / {
proxy_pass http://ftp.FreeBSD.org$request_uri;
proxy_set_header Host ftp.FreeBSD.org;
}
}
# FreeBSD Mirror Proxy 2 Configuration
server {
listen 8014;
server_name localhost;
location / {
proxy_pass http://ftp2.FreeBSD.org$request_uri;
proxy_set_header Host ftp2.FreeBSD.org;
}
}
# FreeBSD Mirror Proxy 3 Configuration
server {
listen 8015;
server_name localhost;
location / {
proxy_pass http://ftp3.FreeBSD.org$request_uri;
proxy_set_header Host ftp3.FreeBSD.org;
}
}
</pre>
* Restart nginx:
<pre>
service restart nginx
</pre>
* Now during an installation of FreeBSD, during the mirror selection phase select *Other* and enter http://freebsdmirror.example.com