Project

General

Profile

Support #737

Install a Reverse Proxy Package Cache With Nginx on FreeBSD

Added by Daniel Curtis about 8 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Package Management
Target version:
Start date:
01/31/2016
Due date:
% Done:

100%

Estimated time:
1.00 h
Spent time:

Description

This is a guide on using nginx to create a reverse proxy package cache for FreeBSD pkgng, ports, installer packages. The intent is to allow development of commonly used applications during times that I am not connected to the internet.

Prepare the Environment

  • Make sure the system is up to date:
    pkg update && pkg upgrade
    

Install Nginx

  • Install nginx:
    pkg install nginx
    
  • Enable nginx at boot:
    sysrc nginx_enable=YES
    
  • Create a configuration directory to make managing individual server blocks easier
    mkdir /usr/local/etc/nginx/conf.d
    
  • Edit the main nginx config file:
    vi /usr/local/etc/nginx/nginx.conf
    
    • And strip down the config file and add the include statement at the end to make it easier to handle various server blocks:
      worker_processes  1;
      error_log  /var/log/nginx-error.log;
      
      events {
          worker_connections  1024;
      }
      
      http {
          include       mime.types;
          default_type  application/octet-stream;
          sendfile        on;
          keepalive_timeout  65;
      
          # nginx may need to resolve domain names at run time
          resolver 208.67.222.222 208.67.220.220;
      
          # Load config files from the /etc/nginx/conf.d directory
          include /usr/local/etc/nginx/conf.d/*.conf;
      }
      

Caching pkgng

Pkgng Nginx Cache Config

  • Create the directory for the cache and adjust the permissions so nginx can write files to it:
    mkdir /var/cache/pkgmirror
    chown www:www /var/cache/pkgmirror
    
  • Configure nginx as our dynamic cache:
    vi /etc/nginx/conf.d/pkgmirror.example.com.conf
    
    • And add the following:
      server {
          listen      80;
          server_name pkgmirror.example.com;
          root        /var/cache/pkgmirror;
          autoindex   on;
      
          # Requests for actual packages should be served directly from cache if available.
          #   If not available, retrieve and save the package from an upstream mirror.
          location / {
              try_files $uri @pkg_mirror;
          }
      
          # Retrieve package from upstream mirrors and cache for future requests
          location @pkg_mirror {
              proxy_store on;
              proxy_redirect off;
              proxy_cache_revalidate on;
              proxy_cache_min_uses 3;
              proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
              proxy_store_access user:rw group:rw all:r;
              proxy_next_upstream error timeout http_404;
              proxy_pass http://pkg-mirrors$request_uri;
          }
      }
      
      # Upstream FreeBSD Mirrors
      upstream pkg-mirrors {
          server localhost:8007;
          server localhost:8008 backup;
          server localhost:8009 backup;
      }
      
      # FreeBSD Global Mirror Proxy Configuration
      server {
          listen      8007;
          server_name localhost;
      
          location / {
              proxy_pass       http://pkg.FreeBSD.org$request_uri;
              proxy_set_header Host pkg.FreeBSD.org;
          }
      }
      
      # FreeBSD US West Mirror Proxy Configuration
      server {
          listen      8008;
          server_name localhost;
      
          location / {
              proxy_pass       http://pkg.us-west.FreeBSD.org$request_uri;
              proxy_set_header Host pkg.us-west.FreeBSD.org;
          }
      }
      
      # FreeBSD US East Mirror Proxy Configuration
      server {
          listen      8009;
          server_name localhost;
      
          location / {
              proxy_pass       http://pkg.us-east.FreeBSD.org$request_uri;
              proxy_set_header Host pkg.us-east.FreeBSD.org;
          }
      }
      
  • Restart nginx:
    service restart nginx
    

Using a Private Repo

  • Edit the pkg repo config file:
    mkdir -p /usr/local/etc/pkg/repos
    vi /usr/local/etc/pkg/repos/FreeBSD.conf
    
  • Add the following line to use the new proxy cache with quarterly updates
    FreeBSD: {
      url: "pkg+http://pkgmirror.example.com/${ABI}/quarterly",
      mirror_type: "srv",
      enabled: yes
    }
    

Cache Cleaning

  • The following command will clean the package cache:
    setenv PKG_CACHEDIR /var/cache/pkgmirror
    pkg clean
    

Caching the Ports Tree

Pkgng Nginx Cache Config

  • Create the directory for the cache and adjust the permissions so nginx can write files to it:
    mkdir /var/cache/portsmirror
    chown www:www /var/cache/portsmirror
    
  • Configure nginx as our dynamic cache:
    vi /etc/nginx/conf.d/portsmirror.example.com.conf
    
    • And add the following:
      # nginx may need to resolve domain names at run time
      server {
          listen      80;
          server_name portsmirror.example.com;
          root        /var/cache/portsmirror;
          autoindex   on;
      
          # Requests for actual packages should be served directly from cache if available.
          #   If not available, retrieve and save the package from an upstream mirror.
          location / {
              try_files $uri @ports_mirror;
          }
      
          # Retrieve package from upstream mirrors and cache for future requests
          location @ports_mirror {
              proxy_store    on;
              proxy_redirect off;
              proxy_cache_revalidate on;
              proxy_cache_min_uses 3;
              proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
              proxy_store_access  user:rw group:rw all:r;
              proxy_next_upstream error timeout http_404;
              proxy_pass          http://ports-mirrors$request_uri;
          }
      }
      
      # Upstream FreeBSD Mirrors
      upstream ports-mirrors {
          server localhost:8010;
          server localhost:8011 backup;
          server localhost:8012 backup;
      }
      
      # FreeBSD Global Mirror Proxy Configuration
      server {
          listen      8010;
          server_name localhost;
      
          location / {
              proxy_pass       http://your-org.portsnap.freebsd.org$request_uri;
              proxy_set_header Host your-org.portsnap.freebsd.org;
          }
      }
      
      # FreeBSD US West Mirror Proxy Configuration
      server {
          listen      8011;
          server_name localhost;
      
          location / {
              proxy_pass       http://sourcefire.portsnap.freebsd.org$request_uri;
              proxy_set_header Host sourcefire.portsnap.freebsd.org;
          }
      }
      
      # FreeBSD US East Mirror Proxy Configuration
      server {
          listen      8012;
          server_name localhost;
      
          location / {
              proxy_pass       http://isc.portsnap.freebsd.org$request_uri;
              proxy_set_header Host isc.portsnap.freebsd.org;
          }
      }
      
  • Restart nginx:
    service restart nginx
    

Download Ports Tree

  • Edit the portsnap config file:
    vi /etc/portsnap
    
    • And change the SERVERNAME variable to the local proxy:
      SERVERNAME=portsmirror.example.com
      
  • Set the http_proxy environment variable:
    setenv http_proxy http://portsmirror.example.com
    
  • Then fetch the latest ports tree:
    portsnap fetch extract
    

FreeBSD Install Cache Config

  • Create the directory for the cache and adjust the permissions so nginx can write files to it:
    mkdir /var/cache/freebsdmirror
    chown www:www /var/cache/freebsdmirror
    
  • Configure nginx as our dynamic cache:
    vi /etc/nginx/conf.d/freebsdmirror.example.com.conf
    
    • And add the following:
      server {
          listen      80;
          server_name freebsdmirror.example.com;
          root        /var/cache/freebsdmirror;
          autoindex   on;
      
          # Requests for actual packages should be served directly from cache if available.
          #   If not available, retrieve and save the package from an upstream mirror.
          location ~ \.txz$ {
              try_files $uri @freebsd_mirror;
          }
      
          # Retrieve package from upstream mirrors and cache for future requests
          location @freebsd_mirror {
              proxy_store    on;
              proxy_redirect off;
              proxy_cache_revalidate on;
              proxy_cache_min_uses 3;
              proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
              proxy_store_access  user:rw group:rw all:r;
              proxy_next_upstream error timeout http_404;
              proxy_pass          http://freebsd-mirrors$request_uri;
          }
      }
      
      # Upstream FreeBSD Mirrors
      upstream freebsd-mirrors {
          server localhost:8013;
          server localhost:8014 backup;
          server localhost:8015 backup;
      }
      
      # FreeBSD Global Mirror Proxy Configuration
      server {
          listen      8013;
          server_name localhost;
      
          location / {
              proxy_pass       http://ftp.FreeBSD.org$request_uri;
              proxy_set_header Host ftp.FreeBSD.org;
          }
      }
      
      # FreeBSD Mirror Proxy 2 Configuration
      server {
          listen      8014;
          server_name localhost;
      
          location / {
              proxy_pass       http://ftp2.FreeBSD.org$request_uri;
              proxy_set_header Host ftp2.FreeBSD.org;
          }
      }
      
      # FreeBSD Mirror Proxy 3 Configuration
      server {
          listen      8015;
          server_name localhost;
      
          location / {
              proxy_pass       http://ftp3.FreeBSD.org$request_uri;
              proxy_set_header Host ftp3.FreeBSD.org;
          }
      }
      
  • Restart nginx:
    service restart nginx
    
  • Now during an installation of FreeBSD, during the mirror selection phase select Other and use the new installer URL:
    http://freebsdmirror.example.com/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE
    

Pkg Metadata Backup

NOTE: The pkg metadata rotation method is a much more robust solution

  • Install rename:
    pkg install rename
    
  • Create the pkg metadata backup script:
    vi /usr/local/bin/backup-pkg-meta.sh
    
    • And add the following:
      #!/bin/sh
      cd /var/cache/pkgmirror
      
      find . -name "meta.txz" -exec renamex -s/\.txz/\.txz\.bak/g {} \;
      find . -name "packagesite.txz" -exec renamex -s/\.txz/\.txz\.bak/g {} \;
      
    • Make the script executable:
      chmod +x /usr/local/bin/backup-pkg-meta.sh
      
  • Since the nginx proxy cache config is set to cache the pkg metadata files, they need to be moved so the newer metadata files can be downloaded and cached. Run the script frequently or cron job to run the script:
    backup-pkg-meta.sh
    

Pkg Metadata Rotation

  • Create a pkg metadata rotation script:
    vi /usr/local/bin/rotate-pkgmirror-meta.sh
    
    • And add the following:
      #!/bin/sh
      # Rotate the metadata archives for the pkg mirror
      
      IP_HOST=8.8.8.8
      
      META_PACKAGE=meta.txz
      META_BACKUP="${META_PACKAGE}.bak" 
      META_TMP="${META_PACKAGE}.tmp" 
      PACKAGESITE_PACKAGE=packagesite.txz
      PACKAGESITE_BACKUP="${PACKAGESITE_PACKAGE}.bak" 
      
      LOCAL_PATH=/var/cache/pkgmirror/FreeBSD:11:amd64/quarterly/
      LOCAL_META_PACKAGE_PATH="${LOCAL_PATH}${META_PACKAGE}" 
      LOCAL_META_BACKUP_PATH="${LOCAL_PATH}${META_BACKUP}" 
      LOCAL_META_TMP_PATH="${LOCAL_PATH}${META_TMP}" 
      LOCAL_PACKAGESITE_PACKAGE_PATH="${LOCAL_PATH}${PACKAGESITE_PACKAGE}" 
      LOCAL_PACKAGESITE_BACKUP_PATH="${LOCAL_PATH}${PACKAGESITE_BACKUP}" 
      
      REMOTE_PATH=http://pkg.freebsd.org/FreeBSD:11:amd64/quarterly/
      REMOTE_META_PATH="${REMOTE_PATH}${META_PACKAGE}" 
      REMOTE_PACKAGESITE_PATH="${REMOTE_PATH}${PACKAGESITE_PACKAGE}" 
      
      if ping -q -c1 -W1 ${IP_HOST} >/dev/null ; then
        echo "IPv4 is up. Fetching current meta.txz." 
      
        fetch -qo ${LOCAL_META_TMP_PATH} ${REMOTE_META_PATH}
      
        if `diff ${LOCAL_META_PACKAGE_PATH} ${LOCAL_META_TMP_PATH} >/dev/null` ; then
          echo "meta.txz is current. Cleaning up and not rotating." 
          rm ${LOCAL_META_TMP_PATH}
        else 
          echo "meta.txz is different. Updating and rotating meta.txz/packagesite.txz." 
      
          mv ${LOCAL_META_PACKAGE_PATH} ${LOCAL_META_BACKUP_PATH} 
          mv ${LOCAL_PACKAGESITE_PACKAGE_PATH} ${LOCAL_PACKAGESITE_BACKUP_PATH} 
          mv ${LOCAL_META_TMP_PATH} ${LOCAL_META_PACKAGE_PATH}
      
          fetch -o ${LOCAL_PACKAGESITE_PACKAGE_PATH} ${REMOTE_PACKAGESITE_PATH} 
        fi
      else
        echo "IPv4 is down. Not rotating." 
      fi
      
  • Make the script executable:
    chmod +x /usr/local/bin/rotate-pkgmirror-meta.sh
    
  • Edit the crontab:
    crontab -e
    
    • And add the following to run the script every half hour:
      */30 * * * * /usr/local/bin/rotate-pkgmirror-meta.sh >/tmp/rotate-pkgmirror.log
      
#1

Updated by Daniel Curtis about 8 years ago

  • Description updated (diff)
  • Status changed from New to In Progress
  • Start date changed from 01/30/2016 to 01/31/2016
  • % Done changed from 0 to 50
#2

Updated by Daniel Curtis about 8 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 50 to 100
#3

Updated by Daniel Curtis about 8 years ago

  • Subject changed from Install a Reverse Proxy Package Cache With Nginx on FreeBSD to Install a Reverse Proxy Package and Ports Cache With Nginx on FreeBSD
  • Description updated (diff)
#4

Updated by Daniel Curtis about 8 years ago

  • Description updated (diff)
#5

Updated by Daniel Curtis about 8 years ago

  • Description updated (diff)
#6

Updated by Daniel Curtis about 8 years ago

  • Subject changed from Install a Reverse Proxy Package and Ports Cache With Nginx on FreeBSD to Install a Reverse Proxy Package Cache With Nginx on FreeBSD
  • Description updated (diff)
#7

Updated by Daniel Curtis about 8 years ago

  • Description updated (diff)
#8

Updated by Daniel Curtis about 8 years ago

  • Description updated (diff)
#9

Updated by Daniel Curtis about 8 years ago

  • Description updated (diff)
  • Status changed from Resolved to Closed
#10

Updated by Daniel Curtis about 6 years ago

  • Description updated (diff)
#11

Updated by Daniel Curtis about 6 years ago

  • Target version changed from FreeBSD 9 to FreeBSD 11
#12

Updated by Daniel Curtis about 6 years ago

  • Description updated (diff)
#13

Updated by Daniel Curtis almost 6 years ago

  • Description updated (diff)
#14

Updated by Daniel Curtis almost 6 years ago

  • Description updated (diff)

Also available in: Atom PDF