apt get life

Life around technology

You are here: Home / Archives for Linux

System Hang on Ubuntu 24.04 “e1000_print_hw_hang”

by Leave a Comment

I recently ran into an issue on an Ubuntu 24.04 server where the system would intermittently hang and become unresponsive. Checking /var/log/syslog, it seems the onboard Intel network card was the problem.

May 14 01:38:09 server-1 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:
...
May 14 01:38:25 server-1 kernel: workqueue: e1000_print_hw_hang [e1000e] hogged CPU for >13333us 4 times, consider switching to WQ_UNBOUND

This repeats every 2 seconds, filling the logs and hogging CPU time. The NIC in question is using the e1000e driver for the onboard Intel Ethernet controller. This is a known issue with certain Intel NICs and the e1000e driver. The kernel repeatedly reports a “Hardware Unit Hang” when the NIC’s transmit queue stalls. Apparently, it is more often see this after the system has been up for a while, usually under I/O or network load. Power-saving features and offloads like ASPM and TSO seem to trigger or worsen it.

Fixing e1000_print_hw_hang

1. Disable TSO (TCP Segmentation Offload)

This stops the NIC from offloading TCP segmentation — which can misbehave on this driver.

sudo ethtool -K eno1 tso off

But this won’t persist after reboot, so we’ll make it stick with a systemd service…

Make TSO Setting Persistent with systemd

Create a service file:

sudo nano /etc/systemd/system/disable-tso.service

Paste this in:

[Unit]
Description=Disable TSO on eno1
After=network.target

[Service]
Type=oneshot
ExecStart=/sbin/ethtool -K eno1 tso off
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable it:

sudo systemctl daemon-reexec
sudo systemctl enable disable-tso.service

Optional: start it now without rebooting:

sudo systemctl start disable-tso.service

2. Disable PCIe ASPM via GRUB

Edit the GRUB config:

sudo nano /etc/default/grub

Find this line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

And change it to:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off"

Then update grub:

sudo update-grub

After these changes, the e1000e driver should became stable again.

Install Jetbrains Toolbox on Ubuntu 22.04

by Leave a Comment

If you have access to more than one of the JetBrains products I have found using the toolbox to install and manage them can be more convenient than the snap packages in Ubuntu itself. I often find it quite interesting to load up JetBrains Toolbox and see what new applications are available (since i don’t get email updates from them).

To install the JetBrains Toolbox on Ubuntu 22.04, first the following package will need to be installed:

sudo apt install libfuse2

Now download the JetBrains Toolbox from their website: https://www.jetbrains.com/toolbox-app/

Once downloaded extract the zip/tar.gz changing the version number as appropriate

tar xvzf ~/Downloads/jetbrains-toolbox-1.27.2.13801.tar.gz

run it either via command line ./jetbrains-toolbox or via the UI. It should install the toolbox and it should appear in the system tray.

Setup Fail2ban for NextCloud

by 3 Comments

Running NextCloud or OwnCloud online comes with some risk, as with any online service. It’s important that your installation remains secure against hackers (or at least as secure as it can be). I’ve opted to implement fail2ban in order to help secure it using some custom rules. It’s worth noting that NextCloud does block unwanted login attempts itself through the application, but you’re having to trust application level security. I feel far safer having fail2ban implement firewall rules to prevent access to anyone probing the server.

First thing to do is create the NextCloud filter configuration file. This file will contain the regex that’s used to scan the logs for anything we don’t like the look of in order to block attacking hosts. My understanding is that this file can remain the same for OwnCloud, although I do not currently have a running instance of it to check.

sudo nano /etc/fail2ban/filter.d/nextcloud.conf

Add the following to the file:


[Definition]
failregex=^{"reqId":".","remoteAddr":".","app":"core","message":"Login failed: '.' (Remote IP: '')","level":2,"time":"."}$
^{"reqId":".","level":2,"time":".","remoteAddr":".","app":"core".","message":"Login failed: '.' (Remote IP: '')".}$
^.\"remoteAddr\":\"\".Trusted domain error.*$

There are three regular expressions included here. The first and second checks for login failures, and flags the source IP. The third checks for trusted domain errors – which are usually a result of bots accessing your installation via it’s IP, not via it’s domain (thus, suspicious and I wanted to block them).

Once the file is saved, you can test what the filter would report by running the following command. This is entirely optional (although would help identify issues) and isn’t required for the rest of the steps to work.

sudo fail2ban-regex /var/nextcloud/data/nextcloud.log /etc/fail2ban/filter.d/nextcloud.conf -v

Next, the configuration file needs setup to activate the configurations we’ve just created. Never edit the Fail2ban jail.conf file, it’s likely to be overridden on upgrades. Always create a “.local” file, ideally a separate one for each application or rule you’re setting up (why? because it makes things more organised and easier to manage one rule over another!) inside the jail.d directory. With this in mind, create a nextcloud (or owncloud) file:

sudo nano /etc/fail2ban/jail.d/nextcloud.local

And add the following to it:


[nextcloud]
ignoreip = 192.168.1.0/24
backend = auto
enabled = true
port = 80,443
protocol = tcp
filter = nextcloud
maxretry = 3
bantime = 36000
findtime = 36000
logpath = /var/nextcloud/data/nextcloud.log

Make sure your ignoreip is your local subnet or IP address. I opted to allow my whole LAN to access it without being auto-blocked. I’ve enabled the rule, set the ports to 80 (HTTP) and 443 (HTTPS) and configured ban times, etc. The most important things are the filter which should match the name of the file that was created inside the filter.d directory (excluding extension), and the log path, which may vary by installation. This path is the default for Ubuntu.

Once done, run the following command to restart nextcloud:

sudo service fail2ban restart

You can check the status of the jail by running:

sudo fail2ban-client status nextcloud

You’ll see something similar to this:

Status for the jail: nextcloud
|- Filter
|  |- Currently failed: 13
|  |- Total failed: 82
|  `- File list:    /var/nextcloud/data/nextcloud.log
`- Actions
   |- Currently banned: 0
   |- Total banned: 5
   `- Banned IP list:

Setting up a Bond and Bridge in Netplan on Ubuntu 18.04

by 2 Comments

For some of my Ubuntu 18.04 servers, I need to run KVM virtual machines which require a bridge to the network so the machines get public LAN IP addresses and aren’t hidden behind NAT. With the server configuration for both my co-location and servers at work the network interfaces are all bonded for fail-over. This means I need a bond to have a bridge ontop of it for the virtual machines to get public IP addresses, while still allowing for failover of the network connection in the event of a network failure.

Research

There are some good examples of setting up netplan here: https://netplan.io/examples

They have a bridge example:

network:
  version: 2
  renderer: networkd
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - enp3s0

And a bond example:

network:
  version: 2
  renderer: networkd
  bonds:
    bond0:
      dhcp4: yes
      interfaces:
        - enp3s0
        - enp4s0
      parameters:
        mode: active-backup
        primary: enp3s0

But there’s not a clear indication of how to amalgamate the two.

A bond and a bridge

Here’s what I’ve ended up with in ‘/etc/netplan/50-cloud-init.yaml’:

network:
    bridges:
        br0:
            addresses:
            - 192.168.10.30
            dhcp4: false
            gateway4: 192.168.10.1
            nameservers:
                addresses:
                - 192.168.10.1
                - 192.168.10.2
                search: []
            interfaces:
                - bond0
    bonds:
        bond0:
            interfaces:
            - eno1
            - eno2
            parameters:
                mode: active-backup
    ethernets:
        eno1:
            addresses: []
            dhcp4: false
            dhcp6: false
        eno2:
            addresses: []
            dhcp4: false
            dhcp6: false

Note that I’ve obviously defined static IP addresses, but this isn’t a requirement. Just set ‘dhcp4: true’ and remove the ‘address’, ‘gateway’ and ‘nameserver’ sections if you’re using DHCP.

Once the file’s got that setup in it, it’s possible to run:
sudo netplan apply
and you should be able to run ‘networkctl list’ to check the bridge and bond are setup.

Linux servers – using ClamAV to find malware

by

ClamAV is an open source anti-virus program that can be run from the command line, making it incredibly useful for locating any viruses and malware on Linux based servers. Recently someone I’ve previously worked with reported that they’d had reports of abuse originating form one of their servers. Given the quantity of sites, it was difficult to locate any potential vulnerabilities.

grep -RPl --include=*.{php,txt} "(passthru|shell_exec|system|phpinfo|base64_decode|chmod|mkdir|fopen|fclose|readfile) *\(" /var/www/

 

Blindly grepping for potentially malicious strings such as “base64_decode” and “exec” was getting tired fast, as these can be legitimately used for some applications. I stumbled across reports that ClamAV works well for locating potential threats

nice -n 19 clamscan ./ -r -i | grep " FOUND" >> possible_exploits.txt

You can then review these files as you see fit, editing the file to remove ones that are false positives. I then run a command to delete the infected files:

while read f; do rm $f ; done<$possible_exploits.txt