How to install lancache on debian

post updated 20-06-2016, now including Playstation Network and Epic games caching.

Inspired from
https://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/
and
https://github.com/ti-mo/ansible-lanparty

On the above sites, i’ve found interesting information to build a LAN-Cache. A LAN-Cache caches all your client’s Steam, Blizzard, Origin, Windows updates , Epic Games, Playstation Network and League of Legends downloads transparent like a reverse proxy. It also caches simultaneous viewed Twitch streams. There is no client configuration required, only the clients have to use your local DNS server that resolves the steam, blizzard, etc. DNS names to your local nginx server IP. (except Epic Games)
Very useful for WAN bandwidth saving on a LAN Party.

Try yourself:

Get all the required packages:
apt-get install build-essential libpcre3-dev zlib1g-dev libssl-dev

Download the zip and upload the contents to your debian machine:
sregex library

Download the zip and upload the contents to your debian machine:
replace-filter-nginx-module

compile the downloaded library and nginx:
cd regexblib
make
make install
wget http://nginx.org/download/nginx-1.9.10.tar.gz
tar xf nginx-1.9.10.tar.gz
cd ../nginx-1.9.10
./configure --with-http_sub_module --with-http_slice_module --add-module=/root/replace-filter-nginx-module --with-http_ssl_module
make
make install

On ubuntu 14.04 error at launch, resolve this with the following compile argument:
--with-ld-opt="-Wl,-rpath,/usr/local/lib"

 

so i my case:

./configure --with-http_sub_module --with-http_slice_module --add-module=/root/replace-filter-nginx-module --with-ld-opt="-Wl,-rpath,/usr/local/lib" --with-http_ssl_module

 

Create the required directories:

mkdir -p /var/lib/nginx/blizzard
mkdir -p /var/log/depot/blizzard
mkdir /var/log/depot/steam
mkdir /var/log/depot/riot
mkdir /var/log/depot/windows
mkdir /var/log/depot/epic
mkdir /var/log/depot/sony
mkdir /var/log/depot/origin
mkdir /var/log/twitch
mkdir /var/log/nginx

Download the zip and upload the contents to your debian machine in folder /usr/local/nginx/conf:
lancache-config-0.2.zip

Start the nginx server:
/usr/local/nginx/sbin/nginx

 

Remember to let you local DNS server forward all the required addresses to your nginx server.
DNS Adresses:

#Blizzard

dist.blizzard.com
llnw.blizzard.com
blzddist1-a.akamaihd.net
blzddist2-a.akamaihd.net
dist.blizzard.com.edgesuite.net
blizzard.vo.llnwd.net

# Windows Update
download.windowsupdate.com

# Origin
origin-a.akamaihd.net
akamai.cdn.ea.com
lvlt.cdn.ea.com

# League of Legends
l3cdn.riotgames.com

# Steam
content1.steampowered.com
content2.steampowered.com
content3.steampowered.com
content4.steampowered.com
content5.steampowered.com
content6.steampowered.com
content7.steampowered.com
content8.steampowered.com
cs.steampowered.com
clientconfig.akamai.steamstatic.com
hsar.steampowered.com.edgesuite.net
*.cs.steampowered.com
*.hsar.steampowered.com.edgesuite.net
*.akamai.steamstatic.com

# Twitch
usher.justin.tv
usher.twitch.tv
hls.twitch.tv
hls.ttvnw.net
*.hls.twitch.tv
*.hls.ttvnw.net

#Epic
download.epicgames.com
download1.epicgames.com
download2.epicgames.com
download3.epicgames.com
download4.epicgames.com

#Sony
*.dl.playstation.net
*.dl.playstation.net.edgesuite.net
dl.playstation.net
dl.playstation.net.edgesuite.net

a DNS resolver configfile for PfSense can be found here (replace the ip with your debian machine’s IP)

 

Updated 21-03-2016: make syntax and –with-ld-opt argument for nginx ./configure (Thanks to ju2256)

Updated 31-03-2016: Embedded configure command in <code> HTML tags (Thanks to Matt Davies)

Updated 20-06-2016: New config that supports Epic Games and Playstation network caching

Categories: Uncategorized

62 Comments

  • ju2256 says:

    Nice work, thanks a lot.
    I followed this tutorial and found some pb:

    1)
    […]
    compile the downloaded library and nginx:
    cd regexblib
    make
    install
    […]
    install -> make install

    2)
    […]
    cd ../nginx-1.9.10
    ./configure –with-http_sub_module –with-http_slice_module –add-module=/root/replace-filter-nginx-module

    on ubuntu 14.04 error at launch, works when adding
    –with-ld-opt=”-Wl,-rpath,/usr/local/lib”

    so i my case:

    ./configure –with-http_sub_module –with-http_slice_module –add-module=/root/replace-filter-nginx-module –with-ld-opt=”-Wl,-rpath,/usr/local/lib”

    Conclusion:
    With those minor changings, it works on ubuntu 14.04 lts.
    I’m currently testing with steam and riot

    Thanks a lot, i can’t wait testing blizzard updater working on lancache

  • Taffer says:

    Thanks, worked great on Ubuntu 14.04 LTS. Very much appreciated! There’s a lot of information and different ways to set this up and this was a nice walkthrough. Thanks ju2256 for the ubuntu details-it was helpful.

  • Taffer says:

    Any chance you could add some instructions on how to get this working for Epic Games domains? The new Unreal Tournament becoming a favorite at my LANs because it is free so everyone can play.

    • Eddie says:

      Hi Taffer,

      Unfortunately, the Epic Games Launcher is using HTTPS for its download traffic. That’s making it a bit difficult to cache its downloaded contents, but there is a(n) (experimental ) way, but it needs some workaround at the client:

      SERVER SIDE:
      create the config file for the epic games intance(an example can be downloaded here: http://blog.yolocation.pro/wp-content/uploads/2016/01/epic.zip)

      Create a self signed certificate (SSC) and use the following common names:
      download.epicgames.com
      download1.epicgames.com
      download2.epicgames.com
      download3.epicgames.com
      download4.epicgames.com

      You can find more info about SSC here:
      https://doc.pfsense.org/index.php/Certificate_Management
      You need to refer to the cert in your site config.

      Install the libssl-dev package through apt-get (apt-get install libssl-dev)
      You need to rebuild nginx with the --with-http_ssl_module flag added.

      Configure your dns server to resolve the above A records to your nginx cache IP.

      CLIENT SIDE:

      Install your SSC CA to your client PC.
      Start the epic launcher with the flag -http=wininet
      (without this flag, the Epic games Launcher wil use curl to process the downloads and wil reject your SSC, because it uses its own CA bundle)

      Enjoy cached downloads! But be warned, hasn’t been tested thoroughly yet!

      If you’ve any questions, please let me know.

  • Eddie says:

    Bad news for League of Legends.
    The LoL Patcher uses HTTPS to connect to l3cdn.riotgames.com
    Unfortunately the curl based LoL client only accepts SSL certificates from its own CA. It’s doesn’t inherit the Trusted CA certificates from the OS(Unlike the Epic Games Launcher).
    The trusted CA certificated are compiled within the client.

    As far as I’ve looked, there is no fallback option to HTTP or to alter the curl-ca-bundle in the client.

    I’m afraid it’s not possible with >6.4 client patcher to download from your cache with a self signed SSL certificate.

    The only very nasty fix is to rewrite the versions configuration file, in order to keep the patcher version below 6.4, that still downloads with HTTP. For an example config file, check http://blog.yolocation.pro/wp-content/uploads/2016/01/riot.site

  • Matt Davies says:

    I’m about to set this up and just wanted to confirm since i cant find confirmation anywhere; if the game is not yet cached, when you click install and it downloads, it goes to “the web”. after that, all other downloads come from the server?

    • Eddie says:

      What your saying is correct;
      The first request is downloaded from the internet to the nginx server cache and serves the client’s request from the cache.
      The following requests are directly served from the nginx server cache directly until the cache expires.

      • Matt Davies says:

        Awesome!! Thanks for the quick reply!! I’ve been trying for a while to set this up through bsd (ugh!!) but much prefer to work in debian, just found your guide and I’m made up that i can follow it 🙂

        thanks again!!!!

  • idiot says:

    Hi,

    I’m getting the error: invalid option “-with-http_sub_module”

    I think its to do with where I extracted the stuff. where am I supposed to extract it to?

    • Eddie says:

      WordPress automatically substitutes double dashes to strange ascii:
      I’ll update the post. In the mean time, see below the correct configure parameters:

      ./configure --with-http_sub_module --with-http_slice_module --add-module=/root/replace-filter-nginx-module --with-ld-opt="-Wl,-rpath,/usr/local/lib"

      The http_sub_module does not have to be extracted. The only module that needs to be extracted is the replace-filter-nginx-module
      In the example above, the module is extracted in: /root/replace-filter-nginx-module

  • idiot says:

    Hi, are there any additional settings in the DNS resolver section that need to be checked? when I set it up, I couldn’t access the internet…

  • Eddie says:

    The config provided in my blog post is meant for dnsmasq. Dnsmasq is used for the DNS forwarder in PfSense.

    The Unbound (used for dns resolver in PfSense) requires a different config syntax.
    The Unbound server might not like the syntax in the advanced config and refuses to start. It uses local-zone and local-data directives.

  • Matt Davies says:

    Hi, I’ve got it working thanks to all your help 🙂

    One thing I was wondering about though; if I wanted to change the location of where the cache is stored to say, an external hard disk, which files would i need to change/edit?

    • Eddie says:

      Glad to hear 🙂

      Those paths are defined in the proxy_cache_path in the /conf/depot/depot.conf file.
      Nginx writes the cached file first to the temp directory that is specified in the proxy_temp_path.

      Keep in mind: After the temp files are written, Nginx will copy the temp files to the proxy cache path. If the both locations are on the same filesystem, it will just rename the file paths, instead of copying. Renaming file paths will save harddisk I/O.

  • Jack says:

    Hi, I am really stuggling with the Lancache setup. Our venue has a room full of PS4’s and downloading new PS4+ games takes days…. So I am trying to setup LANCache for SONY downloads. IIS and all servers on the windows platform is a walk in the park, however, my pace has grinded to a holt with Ubuntu 16 and nginx….. I have installed Ubuntu and nginx and view the standard welcome to nginx if I browse to the IP-Address of the Ubuntu 16 server. What ever I try causes the welcome screen to stop working and nothing to respond in its place. I have downloaded the lancache-386.tgz file and can’t get past what I am supposed to do as opposed to what I am doing…. Is anyone able to through me a lifeline and tell me where I am going wrong? The directories mentioned in the above blog seem to be different to where nginx lives on Ubuntu 16

  • Eddie says:

    Hi Jack,

    The lancache-386.tgz you mentioned might be from the multiplay blog. This package has config files meant for the FreeBSD distribution of Linux. Directives like http-filter and kqueue are included that are not supported in Ubuntu Linux.

    You should either use FreeBSD and the multiplay configs or use Ubuntu/Debian and the walktrough of this blog post.

  • Joel says:

    All installed well and ngix is up and running.

    Im using pfsense and have forwarded as required. All is working well there.

    I receive a 404 error in the access log and the game wont download.

    Any ideas?

    Thanks

    Joel

    • Eddie says:

      Hi Joel,

      This might involve some investigation.
      Which game/platform are you trying to download?
      It might be so that the cdn (content download network) has changed.
      Can you provide the log entry?

  • Joel says:

    Thanks Eddie.

    Steam i need to update to include some additional servers as its finding away around the dns forward etc as my net provider mirrors steam content.

    Blizzard / Starcraft is the game / playform that halts.

    192.168.10.211 – – [05/Jun/2016:14:23:43 +0930] “GET /tpr/sc2/config/57/b5/57b5376b967ecfd0bdfa0058dc5cfcc8 HTTP/1.1” 404 168 “-” “bndl/1.0 (Blizzard Downloader)”

    • Eddie says:

      If your ISP is also mirroring content, that will make it a bit harder.
      Try to use Google DNS (8.8.8.8) in the config to get around this, as it may give you problems.

      I’ve just tested the config, no problems appear.
      It has always been that Blizzard games don’t download fast at the first time.

      I use to preload my cache before going to a LAN.

  • Freddy says:

    Hi Eddie! Thanks in advance for your work, I will try this today. Before to begin; I have been using free bsd and the multiplay blog configuration, it works great for Steam and Riot, I have problems with Blizzard (you may already know the problems with that), how did you solve those problems?, in the multiplay blog some says that using range_cache module solves the problem. I am not an expert (you may notice). How is the blizzard updates working with your configuration?
    Regards
    Freddy

    • Eddie says:

      Hi Freddy,
      Thanks for trying.
      I use the slice module, integrated in Nginx version 1.9.0.
      The first Blizzard download may be a bit slower than expexted (3MB/s at my 10MB/s internet connection)
      Downloading from the cache (content that has been downloaded before) is around 77MB/s using sata HHD’s in ZFS.
      I use to preload most of the blizzard content before going to a lan.

      • Freddy says:

        Hi, folowing the complete guide I get the error:
        /usr/local/nginx/sbin/nginx: error while loading shared libraries: libsregex.so.0: cannot open shared object file: No such file or directory
        when trying to start nginx
        Using Debian 8
        donwloaded the extra libs (also installed)

        thanks

      • Freddy says:

        Eddie, also I am getting the error
        [emerg] “proxy_max_temp_file_size” directive invalid value in /usr/local/nginx/conf/proxy/proxy-cache.conf:27
        this value is set for 40G.

        thanks in advance

        • Eddie says:

          Could you check your free diskspace? (df -h), if it’s lower then 40G it could trow this error.

          • Freddy says:

            Hi, this a test machine, it has a 160G HD, the error continues:

            root@debian:/# /usr/local/nginx/sbin/nginx
            nginx: [emerg] “proxy_max_temp_file_size” directive invalid value in /usr/local/ nginx/conf/proxy/proxy-cache.conf:27

            root@obrajes:/# df -h
            Filesystem Size Used Avail Use% Mounted on
            /dev/sda4 128G 1.4G 120G 2% /
            udev 10M 0 10M 0% /dev
            tmpfs 405M 5.6M 400M 2% /run
            tmpfs 1012M 0 1012M 0% /dev/shm
            tmpfs 5.0M 0 5.0M 0% /run/lock
            tmpfs 1012M 0 1012M 0% /sys/fs/cgroup
            tmpfs 203M 0 203M 0% /run/user/1000

            thanks in advance

  • Freddy says:

    Hi Eddie, sorry for asking again, any ideas on to solve this:
    root@debian:/# /usr/local/nginx/sbin/nginx
    nginx: [emerg] “proxy_max_temp_file_size” directive invalid value in /usr/local/ nginx/conf/proxy/proxy-cache.conf:27

    My machine has 160 GB HD, when I remove this line Nginx can start normally, but I think this is very important right?

    thanks in advance for your help.

  • Itow says:

    Hi Eddie, thanks for your hard work 😉
    It runs like a charm
    For the DNS-Stuff i use opnsense

    Would it be possible to add the following networks:

    GoG
    Wargaming (EU)
    uplay

    Grüße Itow

    • Eddie says:

      Appreciate your reply.
      I will look into the cdn’s (content distribution network) soon.
      I’ll keep you posted.

  • Verb0se says:

    Thanks for making this guide, content-caching is wicked cool! 😀
    Have you considered making all of this available on github? Might make it easier in the long run if you intend to publish updated cfg for different sites etc. Regardless, I really appreciate the time you put into making this setup available!

    I’m seeing this issue where steam content is getting cached, but not to the correct directory. It ends up in /var/lib/nginx/blizzard instead. If i remove the blizzard.site and remove it from depot.conf, it starts caching steam to /var/lib/nginx/origin instead.
    So it seems that the caching path gets overwritten or something, or that perhaps it uses the first caching path in the alphabet (i removed epic and origin, so that would explain why it jumps from blizzard to origin).

    Any clues? Are anyone else experiencing this?

    • Verb0se says:

      (typo: It should have said that I removed/unconfigured epic and riot)

      • Itow says:

        Same here
        i have create uplay.site and he write always in the origins cache
        but if i rename uplay.site to auplay.site all is fine :confused

        • Itow says:

          now with the rename of uplay.site to auplay.site
          he write Steam > Origin chache
          I can not find any mistakes

        • Eddie says:

          Hi Itow,

          The sites are given a key in their config:
          example blizzard
          # Cache Location
          proxy_cache blizzard;

          the proxy_cache is matched in depot.conf:
          keys_zone=blizzard:500m

          Content location should not be controlled with file names.
          Can you check above directives in your config file?

  • Itow says:

    Hi Eddie

    I have set:

    uplay.site =

    location / {
    # Cache location
    proxy_cache uplay;

    depote.conf=

    proxy_cache_path /media/lancache/uplay levels=2:2 keys_zone=uplay:500m..

    but he writes the uplay files in the origin chache folder
    :\

    • Itow says:

      Hi Eddie

      i have upload my settings =

      h t t p s://goo.gl/F9A0Cl (without whitespace)

      i use nginx/1.11.3

      Hope it helps

      • Eddie says:

        Hi Itow,

        I can’t reproduce this situation, using the config files from the blog.
        Have you found a solution in the meantime?

        • Itow says:

          Hi Eddie,

          Yes, the solution is to delete the entire cache after having added a new site
          (gog, uplay …)

          btw

          The DNS-Settings are outdated

          #blizzard
          dist.blizzard.com llnw.blizzard.com blzddist1-a.akamaihd.net blzddist2-a.akamaihd.net dist.blizzard.com.edgesuite.net blizzard.vo.llnwd.net
          #origin
          origin-a.akamaihd.net akamai.cdn.ea.com lvlt.cdn.ea.com
          #steam
          cs.steampowered.com *.cs.steampowered.com content1.steampowered.com content2.steampowered.com content3.steampowered.com content4.steampowered.com content5.steampowered.com content6.steampowered.com content7.steampowered.com content8.steampowered.com *.hsar.steampowered.com.edgesuite.net *.akamai.steamstatic.com content-origin.steampowered.com client-download.steampowered.com *.steamcontent.com
          #uplay
          cdn.ubi.com
          #windows
          *.download.windowsupdate.com download.windowsupdate.com dlassets.xboxlive.com *.xboxone.loris.llnwd.net xboxone.vo.llnwd.net images-eds.xboxlive.com xbox-mbr.xboxlive.com assets1.xboxlive.com.nsatc.net assets1.xboxlive.com

          Grüße Itow

  • Jak says:

    Hello, when I launch nginx I get a BIO_new_file error.

    nginx: [emerg] BIO_new_file(“/usr/local/nginx/conf/epic.crt”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/usr/local/nginx/conf/epic.crt’,’r’) error:2006D080:BIO routines:BIO_new_file:no such file)

    Do I just have to create an ssl certificate for Epic?

    • Eddie says:

      If you want to use Epic caching (eg. Unreal Tournament 4) Then you should use it with a ssl certificate.
      The epic downloader only accepts HTTPS transfers from the cache.

  • Freddy says:

    Hi, today I have tried to update WOW (from Blizzard) and the cache is not working, looked in the DNS requests and there is anew entry:level3.blizzard.com
    this redirects to CNAME: download.blizzard.com.c.footprint.net

    I have configured the first one: level3.blizzard.com to the nGINx Server, this is working but very slow, and after some time it stops completely.

    Has someone else tried this?, is it possible that Blizzard began to use SSL (I don’t know how to check that)… if that is the problem how to fix?

    regards

    Freddy

    • Eddie says:

      You could check your traffic with wireshard and check the destination ports.
      If it’s 443 it’s SSL, if it’s 80 it’s non-ssl (HTTP)

      • FReddy says:

        Thanks Eddie, its HTTP, I have reinstalled the server and now its working, I think it was some change in the configuration files…
        Anyway, level3.blizzard.com its working fine !

  • Choque says:

    So at first I got the following error

    ~~~nginx: [emerg] “proxy_max_temp_file_size” directive invalid value in /usr/local/nginx/conf/proxy/proxy-cache.conf:27
    ~~~

    Which I resolved by playing around with the value until it worked. I’ve got a 1TB HDD hooked up to it and changed all the paths in each and every *.site file inside the /depots/ folder so I’m not sure why it would complain about not enough space.

    Afterwards I got the following error

    ~~~nginx: [emerg] BIO_new_file(“/usr/local/nginx/conf/epic.crt”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/usr/local/nginx/conf/epic.crt’,’r’) error:2006D080:BIO routines:BIO_new_file:no such file)
    ~~~

    Fair enough. Just deleted the epic.site file in the /depots/ folder since I didn’t plan on using caching for EPIC Games anyway.

    And now I get the following error

    ~~~nginx: [emerg] BIO_new_file(“/usr/local/nginx/conf/apache.crt”) failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen(‘/usr/local/nginx/conf/apache.crt’,’r’) error:2006D080:BIO routines:BIO_new_file:no such file)
    ~~~

    Any ideas?

    • Eddie says:

      Hi Choque,

      The apache.crt files are referenced in the riot.site config file. I’m not sure if riot is using ssl nowadays again, i’ll check for it soon.
      In the meanwhile you can out-comment these.

  • Kaytoz says:

    Greetings , I’m having problems with my lancache , mainly trying to cache riot and blizzard atm. I’m using pfsense and DNS Resolver with the config settings you gave. I can ping l3cdn.riotgames.com and my lancache @ 10.0.0.2 responds correctly. But when i turn on the launcher i get a HTTP error. Same thing goes for anything with blizzard. any help will be appreciated.

    • Eddie says:

      Hi Kaytoz,

      I’m not sure, but i’ve heard that Riot is using SSL again, so that means you cant cache it.
      There is a nasty workaround to use a proxy ssl, but then you need to generate a self signed cert, and install that on every client computer.
      Not ideal at big LAN’s.

      • Andicnock says:

        Hey guys riot use new dns worldwide.l3cdn.riotgames.com for me it works so far.
        I do not know if I change to https if it is so share the solution thanks.

  • HtHNightwolf says:

    I am confused. I set up my lancache months ago but it stopped working. So I wanted to give your setup a try.

    I cannot download libpcre3-dev zlib1g-dev, they both do not exist.

    And where please is the folder “regexblib” in that I am supposed to compile?
    Lubuntu 15,04 here

Leave a Reply