Ad-block… ad-block everything (OpenVPN + Pi-hole setup)

So, just like a lot of other people, I am fighting my fight with adblocking. This post will not go into ethics of it, morality of it or what are alternatives – I might cover that with some other post. This time, I want to explain (what I think is) my ultimate setup for ad blocking! With it, you will get best adblocking experience wherever you are and on whatever device you are. This might not be best setup for you – it all depends how technical you want to get and how determine you are against ads. What you will get with this:

• Blocking of most ads on all your devices
• Blocking of most trackers on all your devices
• This setup will save you data (really handy if you have limited data plan). Depending what site you visit, it can be up to 30% of your data (yes, I saw this).
• This setup will secure all your communications (maybe not secure enough against government agencies, but good enough against unsecured wireless endpoints and your provider)

• This setup will allow you to share it easily with others (friends don’t let friends watch ads)

  What are downsides of this approach:

• You need to have spare machine, with one UDP port open

• Due to VPN use, all websites will see you as if you are coming from country where VPN is (it didn’t end up as a problem for me, but I have to point that out now)

My madness progression

This all started rather innocent (just like all horrid stuff do start) with simple ad-blocker in browser. BTW, if you want to stop reading, now is the time and I recommend Firefox browser with holy trinity of uBlock Origin, Privacy Badger and Decentraleyes – this should be good enough for all non-geek people.

But… this ad-blocker setup is not silver bullet, it will not save you from other forms of tracking, like ads inside applications or when generally application do phone home. So, as much as ads were getting smarter and smarter (if you like South Park, please do watch season 19 where this is analyzed much thoroughly), we also had to get smarter and smarter. My journey looked something like this:

I was satisfied with this setup until Viber once got rogue and slowed down my machine to the point it was unusable – ads were autoplaying, mouse was stuttering and there was no way to stop this. I even recorded that: So, that was last drop. Turns out, I was not the only one with this problem and searching internet, people were recommending to put "ads.viber.com" to hosts file, so that it points to 127.0.0.1. This way, any request going to ads.viber.com will be resolved to local address and effectively blocked. But you all knew that already. (Besides getting rid of Viber) I started toying around with /etc/hosts file, but that was not that pleasant experience and (from my experience) Windows is choking on large /etc/hosts file, so I wasn’t long on this setup. For the sake of completeness, here is excellent starting point to craft your own hosts file, if you are into that.

This is when I found out about Pi-hole – this is one of the best pieces of software ever created (not the idea, but whole execution). Pi-hole is complex software with simple idea – point your DNS to it and it will resolve all domains of ads and trackers as 127.0.0.1 and you will never even start a request with them. Installation and upgrade and usage – everything with it is just so smooth and everything "just works". I don’t often donate money to free software, but in this case, I opened my wallet. And so I loaded Pi-hole to my Raspberry Pi and suddenly everyone in whole household (local LAN network) started benefiting from blocked ads and trackers. Life was good again. Except… when I was outside of home and it wasn’t.

This was unsolvable problem for me. Until I accidentally bumped to Streisand software that automates VPN setup with ansible recipes. And this awesome blog post that explains how to setup Streisand with Pi-hole. This is one beautiful idea – use VPN to always connect to local network wherever you are and put Pi-hole in that network and use it as your DNS resolver on-the-go. Not to mention benefits of VPN itself. I was sold! At the end, I didn’t use these advises (and rolled my own which are very similar) because:

• I don’t know ansible
• This is much more software than I asked for
• I don’t like when complexity is huge (and I tend to try to understand what I am installing and why)
• I already have VM and I didn’t know how to point ansible there and was afraid what horrible things will happen on my prod machine

That being said, if you want ultimate setup, please read mentioned blog, as it will use best practices, including WireGuard, socks5 and whatsnot, and will surely be more secure than my Frankenstein. Hack, even Pi-hole itself have nice explanation to integrate it in VPN. But if you just want to dip a toe and don’t care about world-class security, here is 5 minute guide (no, really, it is literally 5 minutes to set this up).

Steps to setup OpenVPN with Pi-hole

• First you will need one machine with some newer Ubuntu (I have 18.04 LTS) – it can be any cloud VM or local machine, as long as you can open UDP port on it for whole world. This is port where VPN will listen. Any cloud provider will do, and any VM size will do. Do note that all your traffic will go through this machine, so check network pricing for your provider.

Alternatively, this can all be done in your home, on Raspberry Pi, but watch out your upload speed (mine is extremely slow), as your download speed while roaming will be bound by your upload speed at home. If you don’t want your traffic to go through this VM (or through your home), this is nice way to fix it, but this is outside of the scope of this tutorial.

• SSH to that machine and install Pi-hole first (but it can done with installing OpenVPN first, doesn’t matter):

curl -sSL https://install.pi-hole.net | bash

Accept all default values, nothing special here. For upstream DNS providers, you can put Cloudflare’s 1.1.1.1 and 1.0.0.1, but you can also use Google ones – 8.8.8.8, or OpenDNS at 208.67.222.222.

• Install OpenVPN (we are using here awesome PiVPN packaging):

curl -L https://install.pivpn.io | bash

When asked for address of DNS server, provide "127.0.0.1" (this is where Pi-hole is). Second DNS can be either empty, or provide some existing one, I generally use "1.1.1.1". You can accept all defaults for other values, nothing special here.

• Change interface on which Pi-hole works and restart it

Installation of OpenVPN gave us new interface (it should be named "tun0"). Check new interface you see with ifconfig command. You should also get address that VPN is set up with command:

ifconfig tun0 | grep "inet " | awk '{print $2}'

For me, it is 10.8.0.1. This is interface and address we want Pi-hole to listen on. Open up "/etc/pihole/setupVars.conf" with your favorite editor and make sure those two lines looks like this:

    PIHOLE_INTERFACE=tun0
    IPV4_ADDRESS=10.8.0.1/24

and restart Pi-hole with sudo service pihole-FTL restart.

• Create your first VPN profile

You are now ready to create your first profile (profile is file with .ovpn extension). You will need new profile for each device you want to connect to VPN (do not use same profile from multiple devices!). To do this, execute:

pivpn add nopass

Give some recognizable name (I use "<person>-<device>" naming scheme). You can use version with password (omit nopass), but I generally find it really sub-optimal to type password whenever I want to connect (it might be once every couple of days, but still…). After this step, you will get your profile in ovpns directory. Download it to device where you will use it and that’s it. OpenVPN client is generally available for all platforms, so you can download it for Windows, Linux, Android (even from F-Droid), iPhone…Just load .ovpn file in client and you will connect.

• Customize Pi-hole experience

If you don’t have web port open to access Pi-hole web interface from outside (which I recommend), you can use pi.hole as URI to access it when you are in VPN. Now, visit https://firebog.net and pick lists you want blocked. Paste them in "Settings"->"Blocklists".

• Share your setup with others

Each time you fire "pivpn add nopass", you will generate new profile. You can share this profile with whomever you want, but beware that they will also use same traffic as you and also, you are now in position to see what sites they visit – you should not abuse their trust on this.

That’s it, enjoy your new setup!

tags: ads, adblocking, openvpn, vpn, pihole, pi-hole

Access SSH behind NAT

Introduction

I self-host a lot of stuff at home. 2x Raspberry PIs, 1x QNAP, and there are lot of software there. I do want access to my home from outside (example: rogue telegram bot starts spamming about air quality in home because my air quality hardware module got busted – don’t you just hate when that happens). Anyway, I don’t feel very comfortable exposing SSH to public. So, what else to do?

One option was to install OpenVPN, and enable access only from that network. That would work. In fact, that is probably 10x more secure solution that what I am about to present, but well… nobody’s perfect.

Reverse SSH tunn…what?

Maybe I should be ashamed, but I really didn’t know about this until 2 months ago (and I use Linux for decade or more). There is this thing called "reverse SSH tunneling" and this blog post is all about that (BTW, I purposely didn’t just named this blog post like that as I wanted to convey intent better; FWIW I do have public IP, so NAT is not even a problem:D). So, after 2 months of never letting me down, I can finally write about it, as this thing is fu$#%k robust.

So, how this works? Behind this cryptic name is actually just simple inversion of control, similar to how any software from 90s exposed your port to internet (anyone remember TeamViewer or similar software) – it just used SEC (somebody’s else computer) to connect to it from inside NAT and proxies (or forwards, if you will) all traffic back to you. But in our case, we don’t just want anyone to be able to peak into our traffic, so we don’t want to use any third-party service (is there actually reverse-SSH-as-a-service offer?:). So, we will build it ourselves. What do we need:

• One public machine (A) with SSH exposed over the internet. It can be any cloud VM, DO droplet or anything you have (root) access to
• One private machine (B) in a secure network that you want to access

So, how to do it, Here is step by step guide.

Step 1 – make sure regular SSH work

Make sure you can do regular SSH from B to A (`ssh usernameOnA@A). If this doesn’t work, no point in proceeding:)

Step 2 – create reverse SSH

So, pick any port (I will use 10022 here), expose that port from machine A to internet (make sure it is not under firewall or expose it in portal of your favorite cloud provider). On machine A make sure you have line GatewayPorts yes in /etc/ssh/sshd_config. Now, from machine B, do:

ssh -fN -R 10022:localhost:22 usernameOnA@A

This will create a tunnel B<===>A. Now, from anywhere, try to connect to machine A, but not on regular port 22, but on port 10022:

ssh usernameOnB@A -p 10022

This is moment when you want to digest this whole thing:) You just connected to newly exposed port 10022 on machine A that will "tunnel" you straight to machine B! If this doesn’t blow your mind, you are soulless person and I hope you ever find love in your life. On the other hand, if this doesn’t work for you – tough luck, try searching online, I am here to expose you to new concepts, I am not here to be your support.

Step 3 – paswordless login

OK, same thing, but without passwords. Generate new key on machine B and add it to authorized keys on machine A (just duckduckgo for "ssh-keygen ssh without password" or something along those lines). Now, you should be able to issue something like this:

ssh -i ~/.ssh/path-to-your-key -fN -R 10022:localhost:22 usernameOnA@A -o "PasswordAuthentication=no"

You should not get asked for password! You should, however, be asked for password when you ssh to B machine (on port 10022). Now, for last step.

Step 4 – auto-login on boot

To wrap it up, you want this connection to be always alive, always available and available upon boot. So, first you need to install autossh tool, it will make your life easier. After that, you should place these lines somewhere that will executed upon boot, like /etc/rc.local or whatever init system rocks your boat:

autossh -i /path/to/your/key -fN -M 10022 -o "PubkeyAuthentication=yes" -o "StrictHostKeyChecking=false" -o "PasswordAuthentication=no" -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R A:10022:localhost:22 usernameOnA@A

There are some new lines here, but I left it as exercise to reader to figure out what do they mean. I also left as exercise to reader another cool thing: do passwordless login directly from any machine to B on port 10022. So, basically, generate key pair on machine C, expose it to authorized_keys on machine B and try to fully login without passwords (C to A to B). Trust me, you do want this, if you are already exposing your SSH to internet!

Conclusion

Self-hosting is sometimes not easy, especially when you have a lot of things. That’s why I like robust things that do not break. This little guy still didn’t let me down and I do use it from time to time – connection is always there and is never broken (and when I need to use, it is probably when I need it the most, so I would be really pissed if it doesn’t work:).

Now, I know what some of you would say – "but, hey duuude, isn’t it just as insecure as regular SSH?". Well, I am not security expert, but I would say it almost isn’t – you are protected with another layer, but it is so thin that you can also assume that it is not there, so no – this is not your FBI/script-kiddy protection, for that – OpenVPN is better suited and I might talk about it soon.

My basic point is – I did it, this is freaking cool, I use it daily, it just works and you just learned something new… or you didn’t, you snarky smartass:) Anyway, I am geek for writing this, you are geek for reading this (you just got to this point, didn’t ya!) – being geek is awesome:)

tags: ssh, autossh, linux, nat, tunneling

Extract top blocked domain from pi-hole using bash

So, I wanted to look around and see what are my top blocked domains on my local network. As far as I know, web interface of pi-hole is offering to just show you top N blocked domains. I wanted them all.

So, once I figured out it is all in log files in /var/log/, it all boiled down to figure out how to detect blocked domains and then it was just one simple script:

cat pihole.log | grep "0.0.0.0$" | sed -e "s/.*\s\(.*\)\sis 0.0.0.0/\1/g" | sort | uniq -c | sort -nr

Output of the script for me looks like:

790 mobile.pipe.aria.microsoft.com
116 winatp-gw-eus.microsoft.com
 52 us-v20.events.data.microsoft.com
 50 tracker.grepler.com
 50 ads.viber.com
 47 v10c.events.data.microsoft.com
 46 settings-win.data.microsoft.com
 40 tracker.trackerfix.com
 16 nexusrules.officeapps.live.com
  6 browser.pipe.aria.microsoft.com
  4 watson.telemetry.microsoft.com
  3 v10.events.data.microsoft.com
  2 nexus.officeapps.live.com
  2 az416426.vo.msecnd.net
  1 vortex.data.microsoft.com

Yes, there are lot of Windows machines on my network:) But since I use ad-tracker, there you cannot see lot of ads blocked.

Here is another example from another of mine pihole servers. This one is:

• outside of my network,
• has lot more of lists obtained from this awesome aggregator,
• family members are not the only users here, and
• is used in conjuction with openVPN, so I am covered on my mobile too

So, here it is:

2516 mobile.pipe.aria.microsoft.com
  54 app-measurement.com
  31 reports.crashlytics.com
   8 www.googletagmanager.com
   8 www.google-analytics.com
   4 www.googletagservices.com
   4 static.chartbeat.com
   4 js-agent.newrelic.com
   4 googleads.g.doubleclick.net
   4 dev.visualwebsiteoptimizer.com
   3 sb.scorecardresearch.com
   3 cdn.optimizely.com
   2 static.doubleclick.net
   2 settings.crashlytics.com
   2 secure-us.imrworldwide.com
   2 s.webtrends.com
   2 realtime.services.disqus.com
   2 pagead2.googlesyndication.com
   2 logs-01.loggly.com
   2 load.sumome.com
   2 d1z2jf7jlzjs58.cloudfront.net
   2 cdn.segment.io
   2 c1.rfihub.net
   2 c.amazon-adsystem.com
   2 ads.mopub.com
   1 www.zergnet.com
   1 video.adaptv.advertising.com
   1 stats.wp.com
   1 ssl.google-analytics.com
   1 secure.quantserve.com
   1 s.skimresources.com
   1 s.adroll.com
   1 referrer.disqus.com
   1 platform.tumblr.com
   1 pixel.wp.com
   1 p.typekit.net
   1 nexusrules.officeapps.live.com
   1 mads.amazon-adsystem.com
   1 live.sekindo.com
   1 hello.myfonts.net
   1 experience.contextly.com
   1 events.redditmedia.com
   1 engine.adzerk.net
   1 device-metrics-us.amazon.com
   1 d3ezl4ajpp2zy8.cloudfront.net
   1 cx.atdmt.com
   1 collector-medium.lightstep.com
   1 cdn.simpleanalytics.io
   1 cdn.onesignal.com
   1 api.branch.io
   1 ads.adthrive.com
   1 ad-delivery.net

Stay safe online, friends;)

tags: bash, bash-magic, pihole, pi-hole

My first blog post on tilde

Yey!:) Once I am done with bb rebuild and this baby, maybe I finally collect some time to create some fabulous 90s web page. Just watch me!

tags: first