Support #736
Updated by Daniel Curtis about 6 years ago
{{>toc}}
This is a guide on how I setup a dynamic reverse proxy package cache using Nginx on Arch Linux.
*+WARNING+*: This method has a limitation. You must use mirrors that use the same relative path to package files and you must configure your cache to use that same path. In this example, we are using mirrors that use the relative path @/archlinux/$repo/os/$arch@ and our cache's Server setting in mirrorlist is configured similarly.
h2. Prepare the Environment
* Make sure everything is up to date using the following command:
<pre>
pacman -Syu
</pre>
h2. Install Nginx
* Install Nginx
<pre>
pacman -S nginx
</pre>
* Start and enable nginx at boot:
<pre>
systemctl enable nginx
systemctl start nginx
</pre>
* Create a configuration directory to make managing individual server blocks easier
<pre>
mkdir /etc/nginx/conf.d
</pre>
* Edit the main nginx config file:
<pre>
vi /etc/nginx/nginx.conf
</pre>
#* And strip down the config file and add the include statement at the end to make it easier to handle various server blocks:
<pre>
worker_processes 1;
error_log /var/log/nginx-error.log;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
# nginx may need to resolve domain names at run time. ipv6 may also break if not enabled on host. time
resolver 208.67.222.222 208.67.220.220 ipv6=off; 208.67.220.220;
include /etc/nginx/conf.d/*.conf;
}
</pre>
h2. Vanilla Arch h3. Package Cache Config
* Create the directory for the cache and adjust the permissions so nginx can write files to it:
<pre>
mkdir /srv/http/pacmirror
chown http:http /srv/http/pacmirror
</pre>
* Configure nginx as our dynamic cache:
<pre>
vi /etc/nginx/conf.d/pacmirror.example.com.conf
</pre>
#* And add the following:
<pre>
server {
listen 80;
server_name pacmirror.example.com;
root /srv/http/pacmirror;
autoindex on;
# Requests for actual packages should be served directly from cache if available.
# If not available, retrieve and save the package from an upstream mirror.
location / {
try_files $uri @pkg_mirror;
}
# Retrieve package from upstream mirrors and cache for future requests
location @pkg_mirror {
proxy_store on;
proxy_redirect off;
proxy_store_access user:rw group:rw all:r;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
proxy_next_upstream error timeout http_404;
proxy_pass http://mirrors$request_uri;
}
}
# Upstream Arch Linux Mirrors
upstream mirrors {
server localhost:8001;
server localhost:8002 backup;
server localhost:8003 backup;
}
# Arch Mirror 1 Proxy Configuration
server {
listen 8001;
server_name localhost;
location / {
proxy_pass http://mirror.us.leaseweb.net$request_uri;
proxy_set_header Host mirror.us.leaseweb.net;
}
}
# Arch Mirror 2 Proxy Configuration
server {
listen 8002;
server_name localhost;
location / {
proxy_pass http://mirror.rit.edu$request_uri;
proxy_set_header Host mirror.rit.edu;
}
}
# Arch Mirror 3 Proxy Configuration
server {
listen 8003;
server_name localhost;
location / {
proxy_pass http://lug.mtu.edu$request_uri;
proxy_set_header Host lug.mtu.edu;
}
}
</pre>
* Restart nginx:
<pre>
systemctl restart nginx
</pre>
h3. h2. Update System Mirrorlist
* Edit the mirrorlist file:
<pre>
vi /etc/pacman.d/mirrorlist
</pre>
#* Add the following line to use this new cache
<pre>
Server = http://pacmirror.example.com/archlinux/$repo/os/$arch
</pre>
h2. Arch Linux ARM Cache Config
* Create the directory for the cache and adjust the permissions so nginx can write files to it:
<pre>
mkdir /srv/http/pacmirror
chown http:http /srv/http/pacmirror
</pre>
* Configure nginx as our dynamic cache:
<pre>
vi /etc/nginx/conf.d/alarm.example.com.conf
</pre>
#* And add the following:
<pre>
server {
listen 80;
server_name alarm.example.com;
root /srv/http/pacmirror;
autoindex on;
# Requests for actual packages should be served directly from cache if available.
# If not available, retrieve and save the package from an upstream mirror.
location / {
try_files $uri @pkg_mirror;
}
# Retrieve package from upstream mirrors and cache for future requests
location @pkg_mirror {
proxy_store on;
proxy_redirect off;
proxy_store_access user:rw group:rw all:r;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
proxy_next_upstream error timeout http_404;
proxy_pass http://mirrors$request_uri;
}
}
# Upstream Arch Linux Mirrors
upstream mirrors {
server localhost:8001;
server localhost:8002 backup;
server localhost:8003 backup;
}
# Arch Mirror 1 Proxy Configuration
server {
listen 8001;
server_name localhost;
location / {
proxy_pass http://ca.us.mirror.archlinuxarm.org$request_uri;
proxy_set_header Host ca.us.mirror.archlinuxarm.org;
}
}
# Arch Mirror 2 Proxy Configuration
server {
listen 8002;
server_name localhost;
location / {
proxy_pass http://co.us.mirror.archlinuxarm.org$request_uri;
proxy_set_header Host co.us.mirror.archlinuxarm.org;
}
}
# Arch Mirror 3 Proxy Configuration
server {
listen 8003;
server_name localhost;
location / {
proxy_pass http://il.us.mirror.archlinuxarm.org$request_uri;
proxy_set_header Host il.us.mirror.archlinuxarm.org;
}
}
</pre>
* Restart nginx:
<pre>
systemctl restart nginx
</pre>
h3. Update System Mirrorlist
* Edit the mirrorlist file:
<pre>
vi /etc/pacman.d/mirrorlist
</pre>
#* Add the following line to use this new cache
<pre>
Server = http://pacmirror.example.com/$arch/$repo
</pre>
h2. Manual Cache Cleaning
*NOTE*: You will need to create a method to clear old packages, as this directory will continue to grow over time. paccache (which is included with pacman) can be used to automate this using retention criteria of your choosing. For example the following command will keep the last 3 versions of packages in your cache directory.:
<pre>
find /var/cache/pacmirror/ -type d -exec paccache -v -r -k 3 -c {} \;
</pre>
h2. Pacman Metadata Backup
* Install rename:
<pre>
pacman -S rename
</pre>
* Create the pacman metadata backup script:
<pre>
vi /usr/local/bin/backup-pacman-meta.sh
</pre>
#* And add the following:
<pre>
#!/bin/sh
cd /srv/http/pacmirror
find . -name "*.db" -exec rename --yes -s 's/.db/.db.bak/g' {} \;
</pre>
#* Make the script executable:
<pre>
chmod +x /usr/local/bin/backup-pacman-meta.sh
</pre>
* Since the nginx proxy cache config is set to cache the pacman metadata files, they need to be moved so the newer metadata files can be downloaded and cached. Run the script frequently or setup a systemd timer to run the script:
<pre>
backup-pacman-meta.sh
</pre>
h2. Resources
* https://wiki.archlinux.org/index.php/Pacman/Tips_and_tricks#Dynamic_reverse_proxy_cache_using_nginx