Saturday, 28 July 2018

  A Local Devuan Package Mirror

 (( with Xinetd and approx ))


  Verbose Version


A shorter simpler version is also available (one post earlier).

About the packages:

===================

------

approx

------

Description::

  A caching proxy server for Debian archive files

    - Package: approx
    - Version: 5.10-1
    - Installed-Size: 4709
    - Maintainer: Eric Cooper
    - Architecture: i386
    - Depends: libc6 (>= 2.17), libpcre3, adduser,
    bzip2, curl, rsyslog | system-log-daemon, xz-utils

Homepage: http://git.debian.org/?p=pkg-ocaml-maint/packages/approx.git

Tags::
     admin::package-management, implemented-in::ocaml,
     interface::daemon,  network::server,
     protocol::http, suite::debian,
     use::downloading, use::proxying

Section::  admin
Size::     1024368
Filename:: pool/DEBIAN/main/a/approx/approx_5.10-1_i386.deb

------

xinetd

------

A daemon for services.

*Xinetd* is a more extensible, flexible, backwardly compatible
replacement for the classic inetd super-server.

Essentially it stands in front of daemons/services and brokers access and
system resource allocations.

Inetd is (still) a viable alternative and please note that
in use Xinetd will accommodate inetd config files .
(we are not doing that here though)

Package: xinetd
Description ::
 
  replacement for inetd with many enhancements

Version::   1:2.3.15.3-1
Maintainer: Salvo 'LtWorf' Tomaselli
Homepage: https://github.com/openSUSE/xinetd
Architecture:: i386

Recommends ::

  rsyslog | system-log-daemon, logrotate, update-inetd

Tags ::

  interface::daemon, network::server, network::service, role::program

Section:: net

Filename:: pool/DEBIAN/main/x/xinetd/xinetd_2.3.15.3-1_i386.deb

Size: 140624

Installation


Is as easy as ::

  $ sudo apt-get install approx xinetd


Configuration


For practical uses there are three parts to this process.

1. configure approx_
   Approx provides the engine fo the caching service
   - /etc/approx/approx.conf
    
2. configure xinetd_
   xinetd makes the service available.
  
   - create /etc/xinetd.d/approx
   - tweak /etc/services_ (optional)
   - tweak /etc/networks_ (optional)

3. configure clients apt_


  
---------

1 Approx_

---------


::
   ~$ apropos approx
   approx (8)           - proxy server for Debian archive files
   approx-import (8)    - copy local .deb files into the approx cache
   approx.conf (5)      - configuration file for approx proxy server


file /etc/approx/approx.conf


approx.conf is the primary configuration file for the
location and structure of the package cache.


The first line is by far the most important.
The others are examples or defaults.

::

   ## /etc/approx/approx.conf
  
   devuan     http://deb.devuan.org/merged

   #vivaldi   http://repo.vivaldi.com/stable/deb
   #spotify   http://repository.spotify.com
   #naibed    http://mirror.internode.on.net/pub/debian
   #ubuntu    http://mirror.internode.on.net/pub/ubuntu
  
  
   # The following are the default parameter values, so there is
   # no need to uncomment them unless you want a different value.
   # See approx.conf(5) for details.

   ## set the port number in  /etc/xinetd.d/approx ##

   #$cache              /var/cache/approx
   #$interval   60
   #$max_rate   unlimited
   #$max_redirects 5
   #$user               approx
   #$group              approx
   #$syslog     daemon
   #$pdiffs     true
   #$offline    false
   #$max_wait   10
   #$curl_path  /usr/bin/curl
   #$verbose    false
   #$debug              false

   # end of working example

That's it really.
see: man approx.conf


    The single active line in the above example has implications.
    The others are examples of using approx to
    *also* mirror other distros or repositories.

So to *break this down* a little more.

::

   devuan     http://deb.devuan.org/merged


When approx receives a request with _devuan_ as the final path of the URI it

* acts as a proxy
* and caches the requested .deb packages
  under a file tree starting at::
     
    /var/cache/approx/devuan

To produce that request, the following pattern is followed in your sources.list(s)


**deb http://REMOTE.URL/devuan stable main contrib non-free**

(or deb-src)
 
This also facilitates hosting other *deb* based distributions
or smaller repositories.

So using the above #disabled examples,
requests for ubuntu or Debian would be cached under their own trees.

e.g. ::
 
  /var/cache/approx/ubuntu
  /var/cache/approx/vivaldi
  /var/cache/approx/naibed

 
Cache Note:
...........

These identifiers may be any simple unique string.
e.g. ::

  naibed
debian drawkcab
I haven't fully tested / studied / hacked or used the approx-import tool. It will, with some understandable bastardly efficiency write over any /var/cache/approx/debian (or even DEBIAN) pool ... your Thesis etal So we are avoiding that by calling debian naibed ... this should be transparent to your users of course.
---------

2 Xinetd_

---------


man pages ::

  ~$ apropos xinetd
  xinetd (8)      - the extended Internet services daemon
  xinetd.conf (5) - Extended Internet Services Daemon configuration file
  xinetd.log (5)  - xinetd service log format
  itox (8)        - converts inetd.conf style configuration files to xinet...
  xconv.pl (8)    - inetd.conf to xinetd.conf converter
 


note
....

*You should not need to manually evoke itox or xconv.pl
unless you are moving away from inetd (classic).
Xinetd seems to cope quite well without human conversion
but you may find otherwise .*



configuration
-------------


The default /etc/xinitd.conf does little other than point to an include directory. It is in that directory that we will describe the network service for approx.

xinitd.d/approx
---------------

Create the configuration at /etc/xinetd.d/approx .

The example_ file will need adjustment to these at least:

-  bind =  10.1.1.10  an address *available on the server*
   and *visible from the rest your Local Area Network*.
   
-  only_from = 10.0.0.0/8  to allow access from *your networks*.
   hint ::

     ip r
  
     ip r | awk '{print $1}'
            (or)
     ip r | cut -d " " -f 1
        
- set "port = 9999"  to another if you need or want to. 
- see /etc/services_ and /etc/networks_ below for some convienient alternatives.

approx starts and stops with its own user and group; approx .
These  were set up when installed.
     

example_
--------

::
  
   ## /etc/xinitd.d/approx  -*- conf -*-
   ## description: approx is a caching Debian package proxy


   service approx
   {
   id           = approx
   disable      = no
   only_from    = 127.0.0.1/8 10.1.1.1/8
   protocol     = tcp
   socket_type  = stream
   wait         = no
   bind         = 10.1.1.10   
   port         = 9999
   user         = approx
   server       = /usr/sbin/approx
   #    server_args  = ""
   #    cps                 = 30
   #    instances   = 20
   #    per_source  = 10
   #    max_load    = 3.0
   #    nice        = 12
   #    log_type     = FILE /var/log/approx-xinetd.log
   #    log_on_success = HOST DURATION
   #    log_on_failure = HOST USERID

   }
  
see more at::
 
  /usr/share/doc/xinetd/examples/empty.conf and sample.conf.gz (zless)
  /usr/share/doc/approx/examples/approx.xinetd
 

services
........

Optionally you may add an entry to /etc/services
e.g.

::
   approx          9999/tcp       # approx deb package proxy

xinitd and other applications will use this entry if there is no other configuration option set.

  
networks
........


Optional and convenience for netstat and xinetd reports.

::

   default         0.0.0.0
   loopback        127.0.0.0
   link-local      169.254.0.0
   MYLAN           10.0.0.0
   OTHERLAN        172.16.0.0

This file is accessed (read) by many tools, mostly for reporting network spans in text format. Xinitd will search this file for network names if given. In particular ::

  only_from    = 127.0.0.1/8 10.1.1.0/8 172.16.0.0

may be written as ::
 
  only_from  = loopback MYLAN OTHERLAN

see: man xinetd.conf
------------------------

3. Clients Configuration

------------------------

apt

...

All that needs attention on the clients is to set up

/etc/sources.list or /etc/sources.list.d/files.list
::
  
   deb http://10.1.1.10:9999/devuan stable main contrib non-free
   deb-src http://10.1.1.10:9999/devuan stable main contrib non-free

It is also possible to set a proxy under /etc/apt/apt.conf.d/
and use standard sources.lists. This is particulaly useful for mobile devices.
However it does mean you have to be carefull to keep the final URI path unique.

e.g.
Create a file under /etc/apt/apt.conf.d/
say 00-homelan-proxy or 00-package-proxy

The contents should be adjusted to match your network but approximately ::

  // /etc/apt/apt.conf.d/00-package-proxy
  // use this proxy when at home
  Aquire::http::proxy "http://10.1.1.10:9999";
  // use this at Bilbo Baggins's
  // Acquire::http::proxy "http://cacher.bilbonet:3128";
  // Only one enabled please.
 


address resolution
..................


/etc/hosts may or may not be convenient.
::
  
   127.0.0.1    localhost
   10.1.1.10    cache.mynet cache


A Local Deb Package Mirror

     (( with Devuan xinetd and approx ))


There are a number of ways to build a shareable "mirroring cache" of packages on a non public Network. This saves bandwidth, time, and possibly enables disconnected use with some small tweaks.
This post is a shorter description of one possible process. There is a longer even more verbose version one post later.

e.g. These proxies all scale to small enterprise level at least.
I have used them all successfully at some time or other.

  * squid-deb-proxy - Squid proxy configuration to optimise package downloads
  * apt-cacher-ng - caching proxy server for software repositories
  * approx - caching proxy server for Debian archive files


The squid approach (? probably ?) supports multiple connections better than the last two. Though thats just a guess, for more industrial uses its worth at least a look. It works OK on a home LAN but not on any machine you reboot a lot.
Waiting for squid to shutdown can get "old" quite quickly.


This Walk through documents setting up a HTTP based proxy using
*approx_ and xinetd_*.

Features of this approach include:

  - Reliable and _low_ maintenance
  - Small and light enough for home use
  - efficient enough for enterprise level deployment.
    Should scale up to 100's of LAN clients including virtual systems.
    (depends on hardware and bandwidth)

  - Tunable access by network space or whitelist
    and optionally time of day. (xinetd)
    (No need to worry your iptables or other filtering.)
  - Tunable system loads and concurrency.  (xinetd)
   
  - A simply structured *pool based mirror*  under /var/cache/approx  (default).
  - Readily and cleanly supports other Debian type systems
    and 3rd party repositories
 
  - Low standby resource demand (xinetd)
  - Fast start up and shut down. (xinetd)

  - Logs flexibly. To file or syslog. ( approx and / or xinetd )

The downsides are:
  - editing the clients sources.list . (though there are options around that even)
  - setting up xinetd the first time may be an issue. 
  - this is not a simple apt-get install  and walk away.
_You_ will have to create at least one new file and edit all your clients sources.lists.

However its really not difficult take your time and test as you go.

The Shortest Version

root@yours~# apt-get install xinetd approx

The Short Version

Adjust values to suit network(s) and remote repositories.

root@yours~# apt-get install xinetd approx

 1. edit the servers
/etc/approx/approx.conf
::
    devuan http://deb.devuan.org/merged



 2. edit a clients sources.list. e.g.
/etc/apt/sources.list.d/01-stable.list
    (? disable the rest while you test ?)
    *The client can be the same machine as the server.*
   
 ::
    deb http://10.1.1.10:9999/devuan stable  main contrib non-free

 3. create xinetd approx service file at
/etc/xinetd.d/approx
   
 ::
    # /etc/xinetd.d/approx
    service approx
    {
    id           = approx
    disable      = no
    bind         = 10.1.1.10   
    port         = 9999
    only_from    = 127.0.0.1/8 10.1.1.1/8
    protocol     = tcp
    socket_type  = stream
    wait         = no
    user         = approx
    server       = /usr/sbin/approx
    }

4. Restart xinetd

::
   root@yours~#  /etc/init.d/xinetd  restart

   (or)

   root@yours~#  service xinetd restart

check xinetd has enabled your new service ::

    you@yours~$  netstat -tan | grep 9999
   
    you@yours~$  nc 10.1.1.10 9999

        "get foo"

        HTTP/1.1 400 Bad Request
        Server: approx/5.10 Ocamlnet/4.1.2


(yes that *nc get*  is deliberately broken)

    you@yours~$  CTL c
  
to close the "conversation".

5. check default logging (and any errors like the one above)

::
  you@yours~$  tail -f /var/log/daemonlog

something like: *Nethttpd: Bad request line*

6. simple test from the configured client

::
   root@yours~# apt-get update

see:    /var/cache/approx/

next:
The Long Version includes more options and some alternatives.

References:

Devuan
Xinetd
Approx

  A Local Devuan Package Mirror  (( with Xinetd and approx ))   Verbose Version A shorter simpler version is also available (one ...