Project

General

Profile

Support #737

Install a Reverse Proxy Package Cache With Nginx on FreeBSD

Added by Daniel Curtis almost 9 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Daniel Curtis
Category:
Package Management
Target version:
FreeBSD 11
Start date:
01/31/2016
Due date:
% Done:

100%

Estimated time:
1.00 h
Spent time:
4.50 h

Description

This is a guide on using nginx to create a reverse proxy package cache for FreeBSD pkgng, ports, installer packages. The intent is to allow development of commonly used applications during times that I am not connected to the internet.

Prepare the Environment

  • Make sure the system is up to date:
    pkg update && pkg upgrade

Install Nginx

  • Install nginx:
    pkg install nginx
  • Enable nginx at boot:
    sysrc nginx_enable=YES
  • Create a configuration directory to make managing individual server blocks easier
    mkdir /usr/local/etc/nginx/conf.d
  • Edit the main nginx config file:
    vi /usr/local/etc/nginx/nginx.conf
    • And strip down the config file and add the include statement at the end to make it easier to handle various server blocks:
      worker_processes  1;
error_log  /var/log/nginx-error.log;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    # nginx may need to resolve domain names at run time
    resolver 208.67.222.222 208.67.220.220;

    # Load config files from the /etc/nginx/conf.d directory
    include /usr/local/etc/nginx/conf.d/*.conf;
}

Caching pkgng

Pkgng Nginx Cache Config

  • Create the directory for the cache and adjust the permissions so nginx can write files to it:
    mkdir /var/cache/pkgmirror
chown www:www /var/cache/pkgmirror
  • Configure nginx as our dynamic cache:
    vi /etc/nginx/conf.d/pkgmirror.example.com.conf
    • And add the following:
      server {
    listen      80;
    server_name pkgmirror.example.com;
    root        /var/cache/pkgmirror;
    autoindex   on;

    # Requests for actual packages should be served directly from cache if available.
    #   If not available, retrieve and save the package from an upstream mirror.
    location / {
        try_files $uri @pkg_mirror;
    }

    # Retrieve package from upstream mirrors and cache for future requests
    location @pkg_mirror {
        proxy_store on;
        proxy_redirect off;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 3;
        proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
        proxy_store_access user:rw group:rw all:r;
        proxy_next_upstream error timeout http_404;
        proxy_pass http://pkg-mirrors$request_uri;
    }
}

# Upstream FreeBSD Mirrors
upstream pkg-mirrors {
    server localhost:8007;
    server localhost:8008 backup;
    server localhost:8009 backup;
}

# FreeBSD Global Mirror Proxy Configuration
server {
    listen      8007;
    server_name localhost;

    location / {
        proxy_pass       http://pkg.FreeBSD.org$request_uri;
        proxy_set_header Host pkg.FreeBSD.org;
    }
}

# FreeBSD US West Mirror Proxy Configuration
server {
    listen      8008;
    server_name localhost;

    location / {
        proxy_pass       http://pkg.us-west.FreeBSD.org$request_uri;
        proxy_set_header Host pkg.us-west.FreeBSD.org;
    }
}

# FreeBSD US East Mirror Proxy Configuration
server {
    listen      8009;
    server_name localhost;

    location / {
        proxy_pass       http://pkg.us-east.FreeBSD.org$request_uri;
        proxy_set_header Host pkg.us-east.FreeBSD.org;
    }
}
  • Restart nginx:
    service restart nginx

Using a Private Repo

  • Edit the pkg repo config file:
    mkdir -p /usr/local/etc/pkg/repos
vi /usr/local/etc/pkg/repos/FreeBSD.conf
  • Add the following line to use the new proxy cache with quarterly updates
    FreeBSD: {
  url: "pkg+http://pkgmirror.example.com/${ABI}/quarterly",
  mirror_type: "srv",
  enabled: yes
}

Cache Cleaning

  • The following command will clean the package cache:
    setenv PKG_CACHEDIR /var/cache/pkgmirror
pkg clean

Caching the Ports Tree

Pkgng Nginx Cache Config

  • Create the directory for the cache and adjust the permissions so nginx can write files to it:
    mkdir /var/cache/portsmirror
chown www:www /var/cache/portsmirror
  • Configure nginx as our dynamic cache:
    vi /etc/nginx/conf.d/portsmirror.example.com.conf
    • And add the following:
      # nginx may need to resolve domain names at run time
server {
    listen      80;
    server_name portsmirror.example.com;
    root        /var/cache/portsmirror;
    autoindex   on;

    # Requests for actual packages should be served directly from cache if available.
    #   If not available, retrieve and save the package from an upstream mirror.
    location / {
        try_files $uri @ports_mirror;
    }

    # Retrieve package from upstream mirrors and cache for future requests
    location @ports_mirror {
        proxy_store    on;
        proxy_redirect off;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 3;
        proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
        proxy_store_access  user:rw group:rw all:r;
        proxy_next_upstream error timeout http_404;
        proxy_pass          http://ports-mirrors$request_uri;
    }
}

# Upstream FreeBSD Mirrors
upstream ports-mirrors {
    server localhost:8010;
    server localhost:8011 backup;
    server localhost:8012 backup;
}

# FreeBSD Global Mirror Proxy Configuration
server {
    listen      8010;
    server_name localhost;

    location / {
        proxy_pass       http://your-org.portsnap.freebsd.org$request_uri;
        proxy_set_header Host your-org.portsnap.freebsd.org;
    }
}

# FreeBSD US West Mirror Proxy Configuration
server {
    listen      8011;
    server_name localhost;

    location / {
        proxy_pass       http://sourcefire.portsnap.freebsd.org$request_uri;
        proxy_set_header Host sourcefire.portsnap.freebsd.org;
    }
}

# FreeBSD US East Mirror Proxy Configuration
server {
    listen      8012;
    server_name localhost;

    location / {
        proxy_pass       http://isc.portsnap.freebsd.org$request_uri;
        proxy_set_header Host isc.portsnap.freebsd.org;
    }
}
  • Restart nginx:
    service restart nginx

Download Ports Tree

  • Edit the portsnap config file:
    vi /etc/portsnap
    • And change the SERVERNAME variable to the local proxy:
      SERVERNAME=portsmirror.example.com
  • Set the http_proxy environment variable:
    setenv http_proxy http://portsmirror.example.com
  • Then fetch the latest ports tree:
    portsnap fetch extract

FreeBSD Install Cache Config

  • Create the directory for the cache and adjust the permissions so nginx can write files to it:
    mkdir /var/cache/freebsdmirror
chown www:www /var/cache/freebsdmirror
  • Configure nginx as our dynamic cache:
    vi /etc/nginx/conf.d/freebsdmirror.example.com.conf
    • And add the following:
      server {
    listen      80;
    server_name freebsdmirror.example.com;
    root        /var/cache/freebsdmirror;
    autoindex   on;

    # Requests for actual packages should be served directly from cache if available.
    #   If not available, retrieve and save the package from an upstream mirror.
    location ~ \.txz$ {
        try_files $uri @freebsd_mirror;
    }

    # Retrieve package from upstream mirrors and cache for future requests
    location @freebsd_mirror {
        proxy_store    on;
        proxy_redirect off;
        proxy_cache_revalidate on;
        proxy_cache_min_uses 3;
        proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
        proxy_store_access  user:rw group:rw all:r;
        proxy_next_upstream error timeout http_404;
        proxy_pass          http://freebsd-mirrors$request_uri;
    }
}

# Upstream FreeBSD Mirrors
upstream freebsd-mirrors {
    server localhost:8013;
    server localhost:8014 backup;
    server localhost:8015 backup;
}

# FreeBSD Global Mirror Proxy Configuration
server {
    listen      8013;
    server_name localhost;

    location / {
        proxy_pass       http://ftp.FreeBSD.org$request_uri;
        proxy_set_header Host ftp.FreeBSD.org;
    }
}

# FreeBSD Mirror Proxy 2 Configuration
server {
    listen      8014;
    server_name localhost;

    location / {
        proxy_pass       http://ftp2.FreeBSD.org$request_uri;
        proxy_set_header Host ftp2.FreeBSD.org;
    }
}

# FreeBSD Mirror Proxy 3 Configuration
server {
    listen      8015;
    server_name localhost;

    location / {
        proxy_pass       http://ftp3.FreeBSD.org$request_uri;
        proxy_set_header Host ftp3.FreeBSD.org;
    }
}
  • Restart nginx:
    service restart nginx
  • Now during an installation of FreeBSD, during the mirror selection phase select Other and use the new installer URL:
    http://freebsdmirror.example.com/pub/FreeBSD/releases/amd64/amd64/10.2-RELEASE

Pkg Metadata Backup

NOTE: The pkg metadata rotation method is a much more robust solution

  • Install rename:
    pkg install rename
  • Create the pkg metadata backup script:
    vi /usr/local/bin/backup-pkg-meta.sh
    • And add the following:
      #!/bin/sh
cd /var/cache/pkgmirror

find . -name "meta.txz" -exec renamex -s/\.txz/\.txz\.bak/g {} \;
find . -name "packagesite.txz" -exec renamex -s/\.txz/\.txz\.bak/g {} \;
    • Make the script executable:
      chmod +x /usr/local/bin/backup-pkg-meta.sh
  • Since the nginx proxy cache config is set to cache the pkg metadata files, they need to be moved so the newer metadata files can be downloaded and cached. Run the script frequently or cron job to run the script:
    backup-pkg-meta.sh

Pkg Metadata Rotation

  • Create a pkg metadata rotation script:
    vi /usr/local/bin/rotate-pkgmirror-meta.sh
    • And add the following:
      #!/bin/sh
# Rotate the metadata archives for the pkg mirror

IP_HOST=8.8.8.8

META_PACKAGE=meta.txz
META_BACKUP="${META_PACKAGE}.bak" 
META_TMP="${META_PACKAGE}.tmp" 
PACKAGESITE_PACKAGE=packagesite.txz
PACKAGESITE_BACKUP="${PACKAGESITE_PACKAGE}.bak" 

LOCAL_PATH=/var/cache/pkgmirror/FreeBSD:11:amd64/quarterly/
LOCAL_META_PACKAGE_PATH="${LOCAL_PATH}${META_PACKAGE}" 
LOCAL_META_BACKUP_PATH="${LOCAL_PATH}${META_BACKUP}" 
LOCAL_META_TMP_PATH="${LOCAL_PATH}${META_TMP}" 
LOCAL_PACKAGESITE_PACKAGE_PATH="${LOCAL_PATH}${PACKAGESITE_PACKAGE}" 
LOCAL_PACKAGESITE_BACKUP_PATH="${LOCAL_PATH}${PACKAGESITE_BACKUP}" 

REMOTE_PATH=http://pkg.freebsd.org/FreeBSD:11:amd64/quarterly/
REMOTE_META_PATH="${REMOTE_PATH}${META_PACKAGE}" 
REMOTE_PACKAGESITE_PATH="${REMOTE_PATH}${PACKAGESITE_PACKAGE}" 

if ping -q -c1 -W1 ${IP_HOST} >/dev/null ; then
  echo "IPv4 is up. Fetching current meta.txz." 

  fetch -qo ${LOCAL_META_TMP_PATH} ${REMOTE_META_PATH}

  if `diff ${LOCAL_META_PACKAGE_PATH} ${LOCAL_META_TMP_PATH} >/dev/null` ; then
    echo "meta.txz is current. Cleaning up and not rotating." 
    rm ${LOCAL_META_TMP_PATH}
  else 
    echo "meta.txz is different. Updating and rotating meta.txz/packagesite.txz." 

    mv ${LOCAL_META_PACKAGE_PATH} ${LOCAL_META_BACKUP_PATH} 
    mv ${LOCAL_PACKAGESITE_PACKAGE_PATH} ${LOCAL_PACKAGESITE_BACKUP_PATH} 
    mv ${LOCAL_META_TMP_PATH} ${LOCAL_META_PACKAGE_PATH}

    fetch -o ${LOCAL_PACKAGESITE_PACKAGE_PATH} ${REMOTE_PACKAGESITE_PATH} 
  fi
else
  echo "IPv4 is down. Not rotating." 
fi
  • Make the script executable:
    chmod +x /usr/local/bin/rotate-pkgmirror-meta.sh
  • Edit the crontab:
    crontab -e
    • And add the following to run the script every half hour:
      */30 * * * * /usr/local/bin/rotate-pkgmirror-meta.sh >/tmp/rotate-pkgmirror.log

Also available in: Atom PDF