Routing tables

Routing tables are useful to set specific routes via specific targets. So what we need is ensure that we have to different routing tables: 1 for the default GW of my ISP so I can play games etc with least latency and another one pointing to the tunnel interface. Sadly, you cannot use eth0 as the next hop-interface for your default route.

protocols {
 static {
     table 1 {
         route 0.0.0.0/0 {
             next-hop 10.0.0.1 {
             }
         }
     }
     table 2 {
         interface-route 0.0.0.0/0 {
             next-hop-interface vtun0 {
               }
           }
       }
   }
}

Rule for internal traffic

First rule is pretty simple and simply indicates that all internal traffic should be internal.

set firewall modify PBR rule 10 description "Internal routing"
set firewall modify PBR rule 10 modify table main
set firewall modify PBR rule 10 action modify
set firewall modify PBR rule 10 source group network-group INTERNAL
set firewall modify PBR rule 10 destination group network-group INTERNAL

modify PBR {
  description "Main routing"
  enable-default-log
  rule 10 {
    action modify
    destination {
      group {
        network-group INTERNAL
      }
    }
    modify {
      table main
      }
    source {
      group {
        network-group INTERNAL
        }
      }
  }

Rule to route LAN-PC traffic via the ISP GW

set firewall modify PBR rule 20 description LAN-to-ISP
set firewall modify PBR rule 20 action modify
set firewall modify PBR rule 20 modify table 1
set firewall modify PBR rule 20 source group network-group LAN-PC

Rule to route LAN-IOT traffic via the VPN

set firewall modify PBR rule 21 description LAN-to-ISP
set firewall modify PBR rule 21 action modify
set firewall modify PBR rule 21 modify table 1
set firewall modify PBR rule 21 source group network-group LAN-IOT

Rule to route traffic from IoT to VPN

set firewall modify PBR rule 30 description IoT-to-VPN
set firewall modify PBR rule 30 action modify
set firewall modify PBR rule 30 modify table 2
set firewall modify PBR rule 30 source group network-group IOT

Alternatively, you could use the network address range of your VLAN. Here, assuming the VLAN 107 for IOT is on port eth1

set firewall modify PBR rule 30 source group address-group ADDRv4_eth1.107

Great. Our routes are in place but we have not defined any NATing as we still need to masquerade ourselves behind the router when getting out to the internet.

Setting up our NATs

nat {
  rule 5000 {
    description IOT-to-PIA
    log disable
    outbound-interface vtun0
      protocol all
      source {
        group {
          network-group IOT
          }
        }
    type masquerade
  }
  rule 5001 {
    description LAN-IOT-to-PIA
    log disable
    outbound-interface vtun0
      protocol all
      source {
        group {
          network-group LAN-IOT
          }
        }
    type masquerade
  }
  rule 5002 {
    description LAN-PC-to-ISP
    log disable
    outbound-interface eth0
      protocol all
      source {
        group {
          network-group LAN-PC
          }
        }
    type masquerade
  }

curl ifconfig.me

If you are connected to your IOT network, you should get the public IP address of your VPN. If you are connected to the LAN-PC network and within 192.168.0.0/25 then you should get your ISP address (of course, the public IP, not 10.0.0.1) and if you are in the 192.168.0.128/25 range, you should also get the VPN public IP address.

Specifically re-route DNS via tunnel

Well so very similarly as before, we are going to indicate to our router that we want the DNS traffic to be routed via our VPN. We don't need to do it for our IOT subnet, as all traffic goes through there anyway, but, we want to do that for all our hosts in the LAN (192.168.0.0/24) subnet.

set firewall modify PBR rule 11 description DNS-to-VPN
set firewall modify PBR rule 11 action modify
set firewall modify PBR rule 11 modify table 2
set firewall modify PBR rule 11 destination port 53
set firewall modify PBR rule 11 protocol udp
set firewall modify PBR rule 11 source group network-group LAN-ALL

Similarly to before, we need to get a NAT rule to get traffic through.

rule 5003 {
        description DNS-TO-PIA
        destination {
            address 0.0.0.0/0
            port 53
        }
        log disable
        outbound-interface vtun0
        protocol udp
        source {
            group {
                network-group LAN-ALL
            }
        }
        type masquerade
    }

How to check?

So how are we going to check that this works? I created a tiny EC2 instance on Amazon, setup bind with logging to file for all requests, and forward from there to CloudFlare etc.

Then from there you will be able to see what IP address the client used to make the DNS requests. If you see your ISP IP address, you forgot something along the lines to make sure that DNS requests are sent through.

Do the same with Pi-Hole.

Pi-Hole is awesome and most of readers using Ubiquit probably also use Pi-Hole somewhere. If you setup your Pi-Hole in a different LAN etc (as I did), all you'd need to do is either route all traffic of your Pi-Hole server via VPN or apply the same rules as above to route the DNS requests of your server LAN to your VPN.

Enable HugePages

I would not be the best person to explain the reasons behind using huge pages (which for a guy who had to reimplement malloc at school is pretty bad :/) but there are plenty of guides that explain the benefit of using huge pages.

I picked up these numbers myself, they are probably not the best ones but they do work. I always go for ^2 numbers because it is always easier / faster for a computer to go with ^2 numbers.

/etc/sysctl.conf

# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).

vm.nr_hugepages=8192
vm.hugetlb_shm_group=64

sysctl -p

/etc/security/limits.conf

# /etc/security/limits.conf
#
#This file sets the resource limits for the users logged in via PAM.
#It does not affect resource limits of the system services.
#
#Also note that configuration files in /etc/security/limits.d directory,
#which are read in alphabetical order, override the settings in this
#file in case the domain is the same or more specific.
#That means for example that setting a limit for wildcard domain here
#can be overriden with a wildcard setting in a config file in the
#subdirectory, but a user specific setting here can be overriden only
#with a user specific setting in the subdirectory.
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - a user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open file descriptors
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# End of file

soft memlock 16777216
hard memlock 16777216

VirtIO or no VirtIO

Before we jump right into this, you might want to prepare a couple things if you intend to use VirtIO drivers for your Machine. There are no particular reasons that I can think of today to do this. However, in the past from using virtio drivers everywhere, I had really good performances. So in this guide I will add the extra optional steps to use a VirtIO disk for the OS and for the network card.

More information here

 sudo wget https://fedorapeople.org/groups/virt/virtio-win/virtio-win.repo -O /etc/yum.repos.d/virtio-win.repo
 sudo dnf install virtio-win

To ensure that you have the drivers available as you install Windows, create a second CD Rom storage and mount the drivers ISO. Once in windows setup, bottom left, you will find a

1. Create our Win10 VM

Requires:

• Windows 10 ISO

• Windows License (I mean, you can run it without it but I can't be held responsible for you hacking it).

• VirtIO drivers ISO (if you are using Virtio for HDD too).

• Your GPU to have been isolated to attach it.

Virt-Manager makes it super easy to do that. So let's do the domain creation quickly.

1. Select source -> ISO -> Windows ISO

2. Set CPU/RAM -> I recommend 4 vCPU at least and 8GB of RAM. Depends on your available capacity.

3. Set Storage Size and pool -> I went for 128GB disk in a new pool that I created in my home folder.

4. Set network to the bridge -> br0 for me.

Boot options

Optional - Virtio Disk

Optional - Virtio Network

virsh edit win10 # where win10 is the name of the VM you created in VirtManager

Optional - Windows 10 find VirtioDrivers at install

Click on Load Drivers -> Navigate to the Secondary Disk.

5. Modify domain to include mode schemas

<domain type='kvm' id='3' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

6. Memory tuning

Adding the huge page as the backend for our VM.

  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>

7. CPU Pinning and tuning

CPU pinning is optional. It allows to bind a vCPU to a CPU core/thread of your machine.

  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='6'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <emulatorpin cpuset='0-1'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
    <vcpusched vcpus='0' scheduler='fifo' priority='1'/>
    <vcpusched vcpus='1' scheduler='fifo' priority='1'/>
    <vcpusched vcpus='2' scheduler='fifo' priority='1'/>
    <vcpusched vcpus='3' scheduler='fifo' priority='1'/>
  </cputune>
  <cpu mode='host-passthrough' check='partial'/>

H. Fake NVIDIA

So NVIDIA driver figures out that there is some information that tells it is not running bare metal but in a VM. Greedy as they are, they want you to get expensive Quadro cards etc. for this usage... So we are going to cheat on NVIDIA.

• Add a vendor_id to the hyperv

• Instruct KVM to hide itself from the OS

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='off'/>
      <vapic state='off'/>
      <spinlocks state='off'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  </features>

1. Install Windows

Don't forget to turn off as much $h17 as possible from MS' services and otherwise-wares...

( I could not decide which GIF was best to illustrate my thinking)

10. Install NVIDIA drivers

From within your VM, downlaod the drivers and install them. Link for drivers

Once the installation is done, reboot the VM.

11. The moment of truth

Boot your VM if not just reboot from the NVIDIA drivers install. If all went well, your GPU will turn on, you will see the UEFI display come up, and a few seconds later, you should have the Windows login page show up before your eyes on your second monitor / TV.

If it doesn't go back to the Device Manager to figure out if the card is properly setup. If you have another Error code 43, I would recommend buying an ATI GPU... Joke aside, using the EFI boot for Windows is what my Error Code 43 from NVIDIA, everything else was done but that...