apt get life

Life around technology

You are here: Home / Archives for Technology / Linux

KVM converting virtual disks from raw img files to qcow2

by Leave a Comment

If you’re running qemu KVM on Ubuntu and want to take advantage of the qcow2 file format’s snapshotting capabilities and sparse disk population you can easily convert using the command line tool

1
qemu-img convert

First, make sure your virtual machine is turned off! Then you can navigate to the directory your virtual disks are stored in (usually /var/lib/libvirt). It’s probably a good idea to be a root user or otherwise sudo the following command

1
qemu-img convert -f raw -O qcow2 vm_hdd.img vm_hdd.qcow2

The

1
-f

flag tells convert what format it’s reading. If you don’t provide it then it’ll guess based on the file extension.

the

1
-O

flag tells convert what file format to output to, again if not provided it’ll guess based on the file extension.

Now you’ve got a qcow2 file, you’ll need to edit the VM configuration

1
virsh edit vm_name

this will open up an editor for your VM configuration. It’s an XML file, so it’s reasonably easy to follow. What you’re looking for is a disk section so you can change the file extension and disk type

1
2
3
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/var/lib/libvirt/images/rhel62-2.img'/>

1
2
3
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source file='/var/lib/libvirt/images/rhel62-2.qcow2'/>

Note both “raw” and “img” have been changed to “qcow2” for this disk. Make sure you’ve picked the right disk to edit in the XML. It may be a good idea to take a backup first so you can fall back to the img file if needed!

That should be it, your VM should now boot with the new disk file. Once you’re sure it’s working you can delete the original (or keep it safe somewhere).

 

More information about KVM can be found on the RedHat website: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/index or the Ubuntu wiki https://help.ubuntu.com/community/KVM/Installation

Optimising Nginx for PHP & WordPress (Time To First Byte)

by

When running page speed insights, it seems that TTFB (Time To First Byte) is something that it really doesn’t like when checking performance.

To solve this, we can use nginx’s caching of compiled PHP pages. Even better, the cache can be a RAM disk, making it very responsive.

First, create a directory for the RAM disk:

1
sudo mkdir -p /mnt/nginx-cache

Now create an entry in the fstab file so it’s mounted to the RAM disk on boot:

1
sudo nano /etc/fstab

1
tmpfs /mnt/nginx-cache tmpfs rw,size=2048M 0 0

This creates a 2GB RAM disk. Edit the size as appropriate for your server. Then mount it:

1
sudo mount /mnt/nginx-cache

Now, create a cache configuration file for Nginx:

1
sudo nano /etc/nginx/conf.d/cache.conf

/etc/nginx/conf.d/cache.conf
1
2
fastcgi_cache_path /etc/nginx-cache levels=1:2 keys_zone=phpcache:512m inactive=2h max_size=1024m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";

This creates a cache of 1GB with a default time of 2 hours. Next update the config files for your website – change your config file name where appropriate.

1
/etc/nginx/sites-enabled/mysite.conf

Inside of the

1
location ~ "^(.+\.php)($|/)" {

section, add:

1
2
3
4
5
6
7
8
9
10
11
# ----------------------------------------------
# Caching
# ----------------------------------------------
# This defines which cache to use (defined in /etc/nginx/cache.conf)
fastcgi_cache phpcache;
# Cache only 200 Okay responses for 2 hours
fastcgi_cache_valid 200 2h;
# Don't cache POST requests, only GET
fastcgi_cache_methods GET HEAD;
# Optional. Add a header to prove it works
add_header X-Fastcgi-Cache $upstream_cache_status;

now you should be able to restart nginx

1
sudo service nginx restart

and access the site via a web browser. Then you can use something like developer tools access the headers of the web requests. You should find a header:

1
X-Fastcgi-Cache: HIT

 

Setup Fail2ban for NextCloud

by 2 Comments

Running NextCloud or OwnCloud online comes with some risk, as with any online service. It’s important that your installation remains secure against hackers (or at least as secure as it can be). I’ve opted to implement fail2ban in order to help secure it using some custom rules. It’s worth noting that NextCloud does block unwanted login attempts itself through the application, but you’re having to trust application level security. I feel far safer having fail2ban implement firewall rules to prevent access to anyone probing the server.

First thing to do is create the NextCloud filter configuration file. This file will contain the regex that’s used to scan the logs for anything we don’t like the look of in order to block attacking hosts. My understanding is that this file can remain the same for OwnCloud, although I do not currently have a running instance of it to check.

1
sudo nano /etc/fail2ban/filter.d/nextcloud.conf

Add the following to the file:

1
2
3
4
[Definition]
failregex=^{"reqId":".<em>","remoteAddr":".</em>","app":"core","message":"Login failed: '.<em>' &#40;Remote IP: '<HOST>'&#41;","level":2,"time":".</em>"}$
        ^{"reqId":".<em>","level":2,"time":".</em>","remoteAddr":".<em>","app":"core".</em>","message":"Login failed: '.<em>' &#40;Remote IP: '<HOST>'&#41;".</em>}$
        ^.<em>\"remoteAddr\":\"<HOST>\".</em>Trusted domain error.*$

There are three regular expressions included here. The first and second checks for login failures, and flags the source IP. The third checks for trusted domain errors – which are usually a result of bots accessing your installation via it’s IP, not via it’s domain (thus, suspicious and I wanted to block them).

Once the file is saved, you can test what the filter would report by running the following command. This is entirely optional (although would help identify issues) and isn’t required for the rest of the steps to work.

1
sudo fail2ban-regex /var/nextcloud/data/nextcloud.log /etc/fail2ban/filter.d/nextcloud.conf -v

Next, the configuration file needs setup to activate the configurations we’ve just created. Never edit the Fail2ban jail.conf file, it’s likely to be overridden on upgrades. Always create a “.local” file, ideally a separate one for each application or rule you’re setting up (why? because it makes things more organised and easier to manage one rule over another!) inside the jail.d directory. With this in mind, create a nextcloud (or owncloud) file:

1
sudo nano /etc/fail2ban/jail.d/nextcloud.local

And add the following to it:

1
2
3
4
5
6
7
8
9
10
11
[nextcloud]
ignoreip = 192.168.1.0/24
backend = auto
enabled = true
port = 80,443
protocol = tcp
filter = nextcloud
maxretry = 3
bantime = 36000
findtime = 36000
logpath = /var/nextcloud/data/nextcloud.log

Make sure your ignoreip is your local subnet or IP address. I opted to allow my whole LAN to access it without being auto-blocked. I’ve enabled the rule, set the ports to 80 (HTTP) and 443 (HTTPS) and configured ban times, etc. The most important things are the filter which should match the name of the file that was created inside the filter.d directory (excluding extension), and the log path, which may vary by installation. This path is the default for Ubuntu.

Once done, run the following command to restart nextcloud:

1
sudo service fail2ban restart

You can check the status of the jail by running:

1
sudo fail2ban-client status nextcloud

You’ll see something similar to this:

1
2
3
4
5
6
7
8
9
Status for the jail: nextcloud
|- Filter
|  |- Currently failed: 13
|  |- Total failed: 82
|  `- File list:    /var/nextcloud/data/nextcloud.log
`- Actions
   |- Currently banned: 0
   |- Total banned: 5
   `- Banned IP list:

Setting up a KVM host on Ubuntu 18.04 with bridged networking

by Leave a Comment

Since the launch of Ubuntu server 18.04, I’ve had a few people ask me how to setup bridged networking since the changeover to netplan for networking. The setup’s actually quite straight forwards once you know how. Start out by installing KVM and associated packages (I’m assuming you’re on a fresh Ubuntu 18.04 box):

1
sudo apt install bridge-utils qemu-kvm libvirt-bin

You should now be able to access virsh, to test it just run the command to list the virtual machines on the host (this will of course output an empty list. We just want to make sure the command hasn’t failed before going further):

1
virsh list --all

1
<h2> Id    Name                           State</h2>

Now edit the netplan file for the networking configuration of the host system:

1
sudo nano /etc/netplan/50-cloud-init.yaml

The configuration should be edited to look similar to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
network:
    ethernets:
        eno1:
            addresses: []
            dhcp4: true
            optional: true
    version: 2
    bridges:
        br0:
            interfaces: [eno1]
            dhcp4: true
            parameters:
                stp: false
                forward-delay: 0

We’re basically defining a single ethernet port (eno1) and assigning it to a bride. Note that you can check your ethernet port name using the ‘ip address’ command on the command line and it will return your network adapters and IP addresses. It should be resonably obvious which one is your ethernet adapter from the list!

It’s worth noting that in this setup, we’re allowing DHCP to allocate addresses to the machine. If this isn’t what you want, take a look at the netplan examples here: https://netplan.io/examples

You can apply these change using:

1
sudo netplan apply

And check it’s working:

1
ip address

or

1
networkctl list

If you’ve gotten this far without a network issue, you should be able to create machines using your favorite tool. I use Virtual Machine Manager (https://virt-manager.org/) which provides a reasonable user interface as well as remote management via SSH.

Setting up a Bond and Bridge in Netplan on Ubuntu 18.04

by 2 Comments

For some of my Ubuntu 18.04 servers, I need to run KVM virtual machines which require a bridge to the network so the machines get public LAN IP addresses and aren’t hidden behind NAT. With the server configuration for both my co-location and servers at work the network interfaces are all bonded for fail-over. This means I need a bond to have a bridge ontop of it for the virtual machines to get public IP addresses, while still allowing for failover of the network connection in the event of a network failure.

Research

There are some good examples of setting up netplan here: https://netplan.io/examples

They have a bridge example:

Netplan bridge example
1
2
3
4
5
6
7
8
network:
  version: 2
  renderer: networkd
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - enp3s0

And a bond example:

Netplan bond example
1
2
3
4
5
6
7
8
9
10
11
12
network:
  version: 2
  renderer: networkd
  bonds:
    bond0:
      dhcp4: yes
      interfaces:
        - enp3s0
        - enp4s0
      parameters:
        mode: active-backup
        primary: enp3s0

But there’s not a clear indication of how to amalgamate the two.

A bond and a bridge

Here’s what I’ve ended up with in ‘/etc/netplan/50-cloud-init.yaml’:

Netplan bridge and bond
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
network:
    bridges:
        br0:
            addresses:
            - 192.168.10.30
            dhcp4: false
            gateway4: 192.168.10.1
            nameservers:
                addresses:
                - 192.168.10.1
                - 192.168.10.2
                search: []
            interfaces:
                - bond0
    bonds:
        bond0:
            interfaces:
            - eno1
            - eno2
            parameters:
                mode: active-backup
    ethernets:
        eno1:
            addresses: []
            dhcp4: false
            dhcp6: false
        eno2:
            addresses: []
            dhcp4: false
            dhcp6: false

Note that I’ve obviously defined static IP addresses, but this isn’t a requirement. Just set ‘dhcp4: true’ and remove the ‘address’, ‘gateway’ and ‘nameserver’ sections if you’re using DHCP.

Once the file’s got that setup in it, it’s possible to run:

1
sudo netplan apply

and you should be able to run ‘networkctl list’ to check the bridge and bond are setup.