Support #737
Updated by Daniel Curtis almost 9 years ago
This is a guide on using nginx to create a reverse proxy package cache for FreebSD pkgng, ports, installer pkgng and ports FreeBSD packages. h2. Prepare the Environment * Make sure the system is up to date: <pre> pkg update && pkg upgrade </pre> h2. Install Nginx * Install nginx: <pre> pkg install nginx </pre> * Start and enable nginx at boot: <pre> echo 'nginx_enable="YES"' >> /etc/rc.conf service nginx start </pre> * Create a configuration directory to make managing individual server blocks easier <pre> mkdir /usr/local/etc/nginx/conf.d </pre> * Edit the main nginx config file: <pre> vi /usr/local/etc/nginx/nginx.conf </pre> #* And strip down the config file and add the include statement at the end to make it easier to handle various server blocks: <pre> worker_processes 1; error_log /var/log/nginx-error.log; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; # nginx may need to resolve domain names at run time resolver 208.67.222.222 208.67.220.220; # Load config files from the /etc/nginx/conf.d directory include /usr/local/etc/nginx/conf.d/*.conf; } </pre> h2. Caching Pkgng h3. Pkgng Nginx Cache Config * Create the directory for the cache and adjust the permissions so nginx can write files to it: <pre> mkdir /var/cache/pkgmirror chown www:www /var/cache/pkgmirror </pre> * Configure nginx as our dynamic cache: <pre> vi /etc/nginx/conf.d/pkgmirror.example.com.conf </pre> #* And add the following: <pre> server { listen 80; server_name pkgmirror.example.com; root /var/cache/pkgmirror; autoindex on; # Requests for actual packages should be served directly from cache if available. # If not available, retrieve and save the package from an upstream mirror. location ~ \.txz$ { try_files $uri @pkg_mirror; } # Retrieve package from upstream mirrors and cache for future requests location @pkg_mirror { proxy_store on; proxy_redirect off; proxy_cache_revalidate on; proxy_cache_min_uses 3; proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504; proxy_store_access user:rw group:rw all:r; proxy_next_upstream error timeout http_404; proxy_pass http://pkg-mirrors$request_uri; } } # Upstream FreeBSD Mirrors upstream pkg-mirrors { server localhost:8007; localhost:8001; server localhost:8008 localhost:8002 backup; server localhost:8009 localhost:8003 backup; } # FreeBSD Global Mirror Proxy Configuration server { listen 8007; 8001; server_name localhost; location / { proxy_pass http://pkg.FreeBSD.org$request_uri; proxy_set_header Host pkg.FreeBSD.org; } } # FreeBSD US West Mirror Proxy Configuration server { listen 8008; 8002; server_name localhost; location / { proxy_pass http://pkg.us-west.FreeBSD.org$request_uri; proxy_set_header Host pkg.us-west.FreeBSD.org; } } # FreeBSD US East Mirror Proxy Configuration server { listen 8009; 8003; server_name localhost; location / { proxy_pass http://pkg.us-east.FreeBSD.org$request_uri; proxy_set_header Host pkg.us-east.FreeBSD.org; } } </pre> * Restart nginx: <pre> service restart nginx </pre> h3. Update System Repo * Edit the pkg repo config file: <pre> vi /usr/local/etc/pkg/repos/FreeBSD.conf </pre> * Add the following line to use this new cache <pre> FreeBSD: { url: "pkg+http://pkgmirror.example.com/freebsd:9:x86:64/latest", mirror_type: "srv", enabled: yes } </pre> h3. Cache Cleaning * The following command will clean the package cache: <pre> setenv PKG_CACHEDIR /var/cache/pkgmirror pkg clean </pre> h2. Caching the Ports Tree h3. Pkgng Nginx Cache Config * Create the directory for the cache and adjust the permissions so nginx can write files to it: <pre> mkdir /var/cache/portsmirror chown www:www /var/cache/portsmirror </pre> * Configure nginx as our dynamic cache: <pre> vi /etc/nginx/conf.d/portsmirror.example.com.conf </pre> #* And add the following: <pre> # nginx may need to resolve domain names at run time server { listen 80; server_name portsmirror.example.com; root /var/cache/portsmirror; autoindex on; # Requests for actual packages should be served directly from cache if available. # If not available, retrieve and save the package from an upstream mirror. location / { try_files $uri @ports_mirror; } # Retrieve package from upstream mirrors and cache for future requests location @ports_mirror { proxy_store on; proxy_redirect off; proxy_cache_revalidate on; proxy_cache_min_uses 3; proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504; proxy_store_access user:rw group:rw all:r; proxy_next_upstream error timeout http_404; proxy_pass http://ports-mirrors$request_uri; } } # Upstream FreeBSD Mirrors upstream ports-mirrors { server localhost:8010; localhost:8004; server localhost:8011 localhost:8005 backup; server localhost:8012 localhost:8006 backup; } # FreeBSD Global Mirror Proxy Configuration server { listen 8010; 8004; server_name localhost; location / { proxy_pass http://your-org.portsnap.freebsd.org$request_uri; proxy_set_header Host your-org.portsnap.freebsd.org; } } # FreeBSD US West Mirror Proxy Configuration server { listen 8011; 8005; server_name localhost; location / { proxy_pass http://sourcefire.portsnap.freebsd.org$request_uri; proxy_set_header Host sourcefire.portsnap.freebsd.org; } } # FreeBSD US East Mirror Proxy Configuration server { listen 8012; 8006; server_name localhost; location / { proxy_pass http://isc.portsnap.freebsd.org$request_uri; proxy_set_header Host isc.portsnap.freebsd.org; } } </pre> * Restart nginx: <pre> service restart nginx </pre> h3. Download Ports Tree * Edit the portsnap config file: <pre> vi /etc/portsnap </pre> #* And change the SERVERNAME variable to the local proxy: <pre> SERVERNAME=portsmirror.example.com </pre> * Set the http_proxy environment variable: <pre> setenv http_proxy http://portsmirror.example.com </pre> * Then fetch the latest ports tree: <pre> portsnap fetch extract </pre> h3. FreeBSD Install Cache Config * Create the directory for the cache and adjust the permissions so nginx can write files to it: <pre> mkdir /var/cache/freebsdmirror chown www:www /var/cache/freebsdmirror </pre> * Configure nginx as our dynamic cache: <pre> vi /etc/nginx/conf.d/freebsdmirror.example.com.conf </pre> #* And add the following: <pre> server { listen 80; server_name freebsdmirror.example.com; root /var/cache/freebsdmirror; autoindex on; # Requests for actual packages should be served directly from cache if available. # If not available, retrieve and save the package from an upstream mirror. location ~ \.txz$ { try_files $uri @freebsd_mirror; } # Retrieve package from upstream mirrors and cache for future requests location @freebsd_mirror { proxy_store on; proxy_redirect off; proxy_cache_revalidate on; proxy_cache_min_uses 3; proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504; proxy_store_access user:rw group:rw all:r; proxy_next_upstream error timeout http_404; proxy_pass http://freebsd-mirrors$request_uri; } } # Upstream FreeBSD Mirrors upstream freebsd-mirrors { server localhost:8013; server localhost:8014 backup; server localhost:8015 backup; } # FreeBSD Global Mirror Proxy Configuration server { listen 8013; server_name localhost; location / { proxy_pass http://ftp.FreeBSD.org$request_uri; proxy_set_header Host ftp.FreeBSD.org; } } # FreeBSD Mirror Proxy 2 Configuration server { listen 8014; server_name localhost; location / { proxy_pass http://ftp2.FreeBSD.org$request_uri; proxy_set_header Host ftp2.FreeBSD.org; } } # FreeBSD Mirror Proxy 3 Configuration server { listen 8015; server_name localhost; location / { proxy_pass http://ftp3.FreeBSD.org$request_uri; proxy_set_header Host ftp3.FreeBSD.org; } } </pre> * Restart nginx: <pre> service restart nginx </pre> * Now during an installation of FreeBSD, during the mirror selection phase select *Other* and use the new installer URL: http://freebsdmirror.example.com/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE enter http://freebsdmirror.example.com