apt get life - Page 2 of 11 - Life around technology

Optimising Nginx for PHP & WordPress (Time To First Byte)

2019/04/06 by sudo

When running page speed insights, it seems that TTFB (Time To First Byte) is something that it really doesn’t like when checking performance.

To solve this, we can use nginx’s caching of compiled PHP pages. Even better, the cache can be a RAM disk, making it very responsive.

First, create a directory for the RAM disk:

sudo mkdir -p /mnt/nginx-cache

Now create an entry in the fstab file so it’s mounted to the RAM disk on boot:

sudo nano /etc/fstab

tmpfs /mnt/nginx-cache tmpfs rw,size=2048M 0 0

This creates a 2GB RAM disk. Edit the size as appropriate for your server. Then mount it:

sudo mount /mnt/nginx-cache

Now, create a cache configuration file for Nginx:

sudo nano /etc/nginx/conf.d/cache.conf

fastcgi_cache_path /etc/nginx-cache levels=1:2 keys_zone=phpcache:512m inactive=2h max_size=1024m;
fastcgi_cache_key "$scheme$request_method$host$request_uri";

This creates a cache of 1GB with a default time of 2 hours. Next update the config files for your website – change your config file name where appropriate.

/etc/nginx/sites-enabled/mysite.conf

Inside of the location ~ "^(.+\.php)($|/)" { section, add:

# ----------------------------------------------
# Caching
# ----------------------------------------------
# This defines which cache to use (defined in /etc/nginx/cache.conf)
fastcgi_cache phpcache;
# Cache only 200 Okay responses for 2 hours
fastcgi_cache_valid 200 2h;
# Don't cache POST requests, only GET
fastcgi_cache_methods GET HEAD;
# Optional. Add a header to prove it works
add_header X-Fastcgi-Cache $upstream_cache_status;

now you should be able to restart nginx sudo service nginx restart and access the site via a web browser. Then you can use something like developer tools access the headers of the web requests. You should find a header:

X-Fastcgi-Cache: HIT

Backing up email using offlineimap

2019/03/06 by sudo

I’ve been asked to help backup a friends email from their existing provider. Since we’re not sure what’ll happen to the existing email after the request to change service providers has been made, it seems reasonable to backup her existing email automatically.

Since I run my own mail server, I thought it’d be quite easy to run something like offlineimap to backup the existing service. One caveat to that has been the existing server prefixes “INBOX” to the mail directory, which my setup doesn’t support. It looks like this is a difference between Courier and Dovecot. I also want to rename “Sent Items” to “.Sent” to follow my Dovecot setup.

Fortunately, offlineimap offers the ability to translate folders on the fly.

.offlineimaprc

[general]
accounts = alice

[Account alice]
localrepository = Local
remoterepository = Remote

[Repository Local]
type = Maildir
localfolders = ~/Mail/Alice

[Repository Remote]
type = IMAP
readonly = true
remotehost = mail.example.com
remoteuser = alice@example.com
remotepass = secret
# Necessary as of OfflineIMAP 6.5.4
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
# Necessary to work around https://github.com/OfflineIMAP/offlineimap/issues/573 (versions 7.0.12, 7.2.1)
ssl_version = tls1_2
nametrans = lambda foldername: re.sub('^INBOX\.*', '.',
                               re.sub('^Sent Items\.*', '.Sent', foldername))

Once I’ve got this setup, I can use crontabs to automatically sync it every x minutes.

Note that after pulling the email, Dovecot may need the quota recalculated manually:

doveadm quota get -u alice@example.com
doveadm quota recalc -u alice@example.com

https://wiki2.dovecot.org/Tools/Doveadm/Quota

Here are some good resources:

https://github.com/OfflineIMAP/offlineimap/issues/417

[general]

metadata = $XDG_DATA_HOME/offlineimap
accounts = darnirmlist
maxsyncaccounts = 1
ui = basic
pythonfile = $XDG_DATA_HOME/offlineimap/passkey.py

socktimeout = 20
fsync = false

[DEFAULT]
localrepositoryhome = ~/.local/var/lib/mail

[Account darnirmlist]
localrepository = darnirmlist-local
remoterepository = darnirmlist-remote
status_backend = sqlite
autorefresh = 1
quick = 20
synclabels = yes
ignorelabels = \Inbox, \Starred, \Sent, \Draft, \Spam, \Trash, \Important

[Repository darnirmlist-local]
type = GmailMaildir
localfolders = %(localrepositoryhome)s/darnirmlist

[Repository darnirmlist-remote]
type = Gmail
remoteuser = darnirmlist@gmail.com

auth_mechanisms = XOAUTH2
oauth2_client_id_eval = get_client_id('darnirmlist@gmail.com')
oauth2_client_secret_eval = get_client_secret('darnirmlist@gmail.com')
oauth2_refresh_token_eval = get_refresh_token('darnirmlist@gmail.com')
oauth2_request_url = https://accounts.google.com/o/oauth2/token
nametrans = lambda folder: re.sub('.*Spam$', 'Spam',
                           re.sub('.*INBOX$', 'Inbox',
                           re.sub('.*Drafts$', 'Drafts',
                           re.sub('.*Sent Mail$', 'Sent',
                           re.sub('.*All Mail', 'Archive',
                           re.sub('.*Trash$', 'Trash', folder))))))
folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail',
                                                     '[Gmail]/Drafts',
                                                     '[Gmail]/Sent Mail',
                                                     '[Gmail]/Spam']
createfolders = False
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
realdelete = no
holdconnectionopen = yes
keepalive = 50

http://www.offlineimap.org/doc/use_cases.html

https://gist.githubusercontent.com/dabrahams/3030332/raw/6e0cfab9982a97d0921d774d9f0d070dccb3b061/gistfile1.cfg

# -*- mode: conf; -*-
# 
# NOTE: Settings generally support python interpolation. This means
# values can contain python format strings which refer to other values
# in the same section, or values in a special DEFAULT section. This
# allows you for example to use common settings for multiple accounts:
#
# [Repository Gmail1]
# trashfolder: %(gmailtrashfolder)s
#
# [Repository Gmail2]
# trashfolder: %(gmailtrashfolder)s
#
# [DEFAULT]
# gmailtrashfolder = [Google Mail]/Papierkorb
#
# would set the trashfolder setting for your German gmail accounts.

# NOTE2: This implies that any '%' needs to be encoded as '%%'

##################################################
# General definitions
##################################################

[general]

# This specifies where offlineimap is to store its metadata.
# This directory will be created if it does not already exist.

#metadata = ~/.offlineimap

# This variable specifies which accounts are defined.  Separate them
# with commas.  Account names should be alphanumeric only.
# You will need to specify one section per account below.  You may
# not use "general" for an account name.

accounts = BoostPro

# Offlineimap can synchronize more than one account at a time.  If you
# want to enable this feature, set the below value to something
# greater than 1.  To force it to synchronize only one account at a
# time, set it to 1.
#
# Note: if you are using autorefresh and have more than one account,
# you must set this number to be >= to the number of accounts you have;
# since any given sync run never "finishes" due to a timer, you will never
# sync your additional accounts if this is 1.

#maxsyncaccounts = 1

# You can specify one or more user interface modules for OfflineIMAP
# to use.  OfflineIMAP will try the first in the list, and if it
# fails, the second, and so forth.
#
# The pre-defined options are:
# Blinkenlights -- A fancy (terminal) interface
# TTYUI         -- a text-based (terminal) interface
# Basic         -- Noninteractive interface suitable for cron'ing
# Quiet         -- Noninteractive interface, generates no output
#                  except for errors.
# MachineUI     -- Interactive interface suitable for machine
#                  parsing.
#
# You can override this with a command-line option -u.

#ui = basic

# If you try to synchronize messages to a folder which the IMAP server
# considers read-only, OfflineIMAP will generate a warning.  If you want
# to suppress these warnings, set ignore-readonly to yes.  Read-only
# IMAP folders allow reading but not modification, so if you try to
# change messages in the local copy of such a folder, the IMAP server
# will prevent OfflineIMAP from propagating those changes to the IMAP
# server.  Note that ignore-readonly is unrelated to the "readonly"
# setting which prevents a repository from being modified at all.

#ignore-readonly = no

########## Advanced settings

# You can give a Python source filename here and all config file
# python snippets will be evaluated in the context of that file.
# This allows you to e.g. define helper functions in the Python
# source file and call them from this config file.  You can find
# an example of this in the manual.
#
# pythonfile = ~/.offlineimap.py
pythonfile = ~/.offlineimap/offlineimap_config.py

# By default, OfflineIMAP will not exit due to a network error until
# the operating system returns an error code.  Operating systems can sometimes
# take forever to notice this.  Here you can activate a timeout on the
# socket.  This timeout applies to individual socket reads and writes,
# not to an overall sync operation.  You could perfectly well have a 30s
# timeout here and your sync still take minutes.
#
# Values in the 30-120 second range are reasonable.
#
# The default is to have no timeout beyond the OS.  Times are given in seconds.
#
# socktimeout = 60

# By default, OfflineIMAP will use fsync() to force data out to disk at
# opportune times to ensure consistency.  This can, however, reduce
# performance.  Users where /home is on SSD (Flash) may also wish to reduce
# write cycles.  Therefore, you can disable OfflineIMAP's use of fsync().
# Doing so will come at the expense of greater risk of message duplication
# in the event of a system crash or power loss.  Default is fsync = true.
# Set fsync = false ot disable fsync.
#
# fsync = true

##################################################
# Mailbox name recorder
##################################################

[mbnames]

# offlineimap can record your mailbox names in a format you specify.
# You can define the header, each mailbox item, the separator,
# and the footer.  Here is an example for Mutt.
# If enabled is yes, all six setting must be specified, even if they
# are just the empty string "".
#
# The header, peritem, sep, and footer are all Python expressions passed
# through eval, so you can (and must) use Python quoting.

enabled = no
filename = ~/Mutt/muttrc.mailboxes
header = "mailboxes "
peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"

# You can also specify a folderfilter.  It will apply to the
# *translated* folder name here, and it takes TWO arguments:
# accountname and foldername.  In all other ways, it will
# behave identically to the folderfilter for accounts.  Please see
# that section for more information and examples.
#
# Note that this filter can be used only to further restrict mbnames
# to a subset of folders that pass the account's folderfilter.


##################################################
# Accounts
##################################################

# This is an account definition clause.  You'll have one of these
# for each account listed in general/accounts above.

[Account BoostPro]
########## Basic settings

# These settings specify the two folders that you will be syncing.
# You'll need to have a "Repository ..." section for each one.

localrepository = BoostProLocal
remoterepository = BoostProRemote

########## Advanced settings

# You can have offlineimap continue running indefinitely, automatically
# syncing your mail periodically.  If you want that, specify how
# frequently to do that (in minutes) here.  You can also specify
# fractional minutes (ie, 3.25).

autorefresh = 5

# OfflineImap can replace a number of full updates by quick
# synchronizations.  It only synchronizes a folder if 1) a Maildir
# folder has changed, or 2) if an IMAP folder has received new messages
# or had messages deleted, ie it does not update if only IMAP flags have
# changed.  Full updates need to fetch ALL flags for all messages, so
# this makes quite a performance difference (especially if syncing
# between two IMAP servers).
# Specify 0 for never, -1 for always (works even in non-autorefresh
# mode), or a positive integer <n> to do <n> quick updates before doing
# another full synchronization (requires autorefresh).  Updates are
# always performed after <autorefresh> minutes, be they quick or full.

quick = 24

# You can specify a pre and post sync hook to execute a external command.
# In this case a call to imapfilter to filter mail before the sync process
# starts and a custom shell script after the sync completes.
# The pre sync script has to complete before a sync to the account will
# start. 

# presynchook = imapfilter
# postsynchook = notifysync.sh
postsynchook = /Users/dave/bin/imap-postsync localhost dave --soft-errors

# You can also specify parameters to the commands
# presynchook = imapfilter -c someotherconfig.lua

# OfflineImap caches the state of the synchronisation to e.g. be able to
# determine if a mail has been deleted on one side or added on the
# other.
#
# The default and historical backend is 'plain' which writes out the
# state in plain text files. On Repositories with large numbers of
# mails, the performance might not be optimal, as we write out the
# complete file for each change.  Another new backend 'sqlite' is
# available which stores the status in sqlite databases.
#
# If you switch the backend, you may want to delete the old cache
# directory in ~/.offlineimap/Account-<account>/LocalStatus manually
# once you are sure that things work.
#
status_backend = sqlite

# If you have a limited amount of bandwidth available you can exclude larger
# messages (e.g. those with large attachments etc).  If you do this it
# will appear to offlineimap that these messages do not exist at all.  They
# will not be copied, have flags changed etc.  For this to work on an IMAP
# server the server must have server side search enabled.  This works with gmail
# and most imap servers (e.g. cyrus etc)
# The maximum size should be specified in bytes - e.g. 2000000 for approx 2MB

# maxsize = 2000000


# When you are starting to sync an already existing account you can tell
# offlineimap to sync messages from only the last x days.  When you do
# this messages older than x days will be completely ignored.  This can
# be useful for importing existing accounts when you do not want to
# download large amounts of archive email.
#
# Messages older than maxage days will not be synced, their flags will
# not be changed, they will not be deleted etc.  For offlineimap it will
# be like these messages do not exist.  This will perform an IMAP search
# in the case of IMAP or Gmail and therefor requires that the server
# support server side searching.  This will calculate the earliest day
# that would be included in the search and include all messages from
# that day until today.  e.g. maxage = 3 to sync only the last 3 days
# mail
#
# maxage =
maxage = 365

# Maildir file format uses colon (:) separator between uniq name and info.
# Unfortunatelly colon is not allowed character in windows file name. If you
# enable maildir-windows-compatible option, offlineimap will be able to store
# messages on windows drive, but you will probably loose compatibility with
# other programs working with the maildir
#
#maildir-windows-compatible = no


[Repository BoostProLocal]

# Each repository requires a "type" declaration. The types supported for
# local repositories are Maildir and IMAP.

type = IMAP

# Specify local repository.  Your IMAP folders will be synchronized
# to maildirs created under this path.  OfflineIMAP will create the
# maildirs for you as needed.

# localfolders = ~/Test

# You can specify the "folder separator character" used for your Maildir
# folders.  It is inserted in-between the components of the tree. If you
# want your folders to be nested directories, set it to "/". 'sep' is
# ignored for IMAP repositories, as it is queried automatically.
#
#sep = .

# Some users may not want the atime (last access time) of folders to be
# modified by OfflineIMAP.  If 'restoreatime' is set to yes, OfflineIMAP
# will restore the atime of the "new" and "cur" folders in each maildir
# folder to their original value after each sync.
#
# In nearly all cases, the default should be fine.
#
#restoreatime = no

# This setting will be forced to "yes" by the fact because we have
# enabled IDLE support
#
# holdconnectionopen = yes
preauthtunnel = /usr/local/libexec/dovecot/imap

[Repository BoostProRemote]
# And this is the remote repository.  We only support IMAP or Gmail here.

type = Gmail

# The following can fetch the account credentials via a python expression that
# is parsed from the pythonfile parameter. For example, a function called
# "getcredentials" that parses a file "filename" and returns the account
# details for "hostname".
# remotehosteval = getcredentials("filename", "hostname", "hostname")
# remoteusereval = getcredentials("filename", "hostname", "user")
# remotepasseval = getcredentials("filename", "hostname", "passwd")
remotepasseval = osxkeychain('imap.gmail.com', account='dave@boostpro.com')

# Specify the remote hostname.
# remotehost = examplehost

# Whether or not to use SSL.
ssl = yes

# SSL Client certificate (optional)
# sslclientcert = /path/to/file.crt

# SSL Client key (optional)
# sslclientkey = /path/to/file.key

# SSL CA Cert(s) to verify the server cert against (optional).
# No SSL verification is done without this option. If it is
# specified, the CA Cert(s) need to verify the Server cert AND
# match the hostname (* wildcard allowed on the left hand side)
# The certificate should be in PEM format.
# sslcacertfile = /path/to/cacertfile.crt

# If you connect via SSL/TLS (ssl=true) and you have no CA certificate
# specified, offlineimap will refuse to sync as it connects to a server
# with an unknown "fingerprint". If you are sure you connect to the
# correct server, you can then configure the presented server
# fingerprint here. OfflineImap will verify that the server fingerprint
# has not changed on each connect and refuse to connect otherwise.
# You can also configure this in addition to CA certificate validation
# above and it will check both ways.

#cert_fingerprint = <SHA1_of_server_certificate_here>
cert_fingerprint = f3043dd689a2e7dddfbef82703a6c65ea9b634c1

# Specify the port.  If not specified, use a default port.
# remoteport = 993

# Specify the remote user name.
remoteuser = dave@boostpro.com

# There are six ways to specify the password for the IMAP server:
#
# 1. No password at all specified in the config file.
#    If a matching entry is found in ~/.netrc (see netrc (5) for
#    information) this password will be used. Do note that netrc only
#    allows one entry per hostname. If there is no ~/.netrc file but
#    there is an /etc/netrc file, the password will instead be taken
#    from there. Otherwise you will be prompted for the password when
#    OfflineIMAP starts when using a UI that supports this.
#
# 2. The remote password stored in this file with the remotepass
#    option. Any '%' needs to be encoded as '%%'. Example:
#    remotepass = mypassword
#
# 3. The remote password stored as a single line in an external
#    file, which is referenced by the remotefile option.  Example:
#    remotepassfile = ~/Password.IMAP.Account1
#
# 4. With a preauth tunnel.  With this method, you invoke an external
#    program that is guaranteed *NOT* to ask for a password, but rather
#    to read from stdin and write to stdout an IMAP procotol stream that
#    begins life in the PREAUTH state.  When you use a tunnel, you do
#    NOT specify a user or password (if you do, they'll be ignored.)
#    Instead, you specify a preauthtunnel, as this example illustrates
#    for Courier IMAP on Debian:
#    preauthtunnel = ssh -q imaphost '/usr/bin/imapd ./Maildir'
#
# 5. If you are using Kerberos and have the Python Kerberos package
#    installed, you should not specify a remotepass.  If the user has a
#    valid Kerberos TGT, OfflineIMAP will figure out the rest all by
#    itself, and fall back to password authentication if needed.
#
# 6. Using arbitrary python code.  With this method, you invoke a
#    function from your pythonfile.  To use this method assign the name
#    of the function to the variable 'remotepasseval'.  Example:
#    remotepasseval = get_password("imap.example.net")
#    You can also query for the username:
#    remoteusereval = get_username("imap.example.net")
#    This method can be used to design more elaborate setups, e.g. by
#    querying the gnome-keyring via its python bindings.

########## Advanced settings

# Some IMAP servers need a "reference" which often refers to the "folder
# root".  This is most commonly needed with UW IMAP, where you might
# need to specify the directory in which your mail is stored. The
# 'reference' value will be prefixed to all folder paths refering to
# that repository. E.g. accessing folder 'INBOX' with reference = Mail
# will try to access Mail/INBOX. Note that the nametrans and
# folderfilter functions will still apply the full path including the
# reference prefix.  Most users will not need this.
#
# reference = Mail

# In between synchronisations, OfflineIMAP can monitor mailboxes for new
# messages using the IDLE command. If you want to enable this, specify here
# the folders you wish to monitor. Note that the IMAP protocol requires a
# separate connection for each folder monitored in this way, so setting
# this option will force settings for:
#     maxconnections - to be at least the number of folders you give
#     holdconnectionopen - to be true
#     keepalive - to be 29 minutes unless you specify otherwise
#
# This feature isn't complete and may well have problems. See the manual
# for more details.
#
# This option should return a Python list. For example
#
# idlefolders = ['INBOX', 'INBOX.Alerts']
#
idlefolders = %(folderincludes)s

# OfflineIMAP can use multiple connections to the server in order
# to perform multiple synchronization actions simultaneously.
# This may place a higher burden on the server.  In most cases,
# setting this value to 2 or 3 will speed up the sync, but in some
# cases, it may slow things down.  The safe answer is 1.  You should
# probably never set it to a value more than 5.

#maxconnections = 2

# OfflineIMAP normally closes IMAP server connections between refreshes if
# the global option autorefresh is specified.  If you wish it to keep the
# connection open, set this to true.  If not specified, the default is
# false.  Keeping the connection open means a faster sync start the
# next time and may use fewer server resources on connection, but uses
# more server memory.  This setting has no effect if autorefresh is not set.
#
#holdconnectionopen = no

# If you want to have "keepalives" sent while waiting between syncs,
# specify the amount of time IN SECONDS between keepalives here.  Note that
# sometimes more than this amount of time might pass, so don't make it
# tight.  This setting has no effect if autorefresh and holdconnectionopen
# are not both set.
#
# keepalive = 60

# Normally, OfflineIMAP will expunge deleted messages from the server.
# You can disable that if you wish.  This means that OfflineIMAP will
# mark them deleted on the server, but not actually delete them.
# You must use some other IMAP client to delete them if you use this
# setting; otherwise, the messgaes will just pile up there forever.
# Therefore, this setting is definitely NOT recommended.
#
#expunge = no

# Specify whether to process all mail folders on the server, or only
# those listed as "subscribed".
#
#subscribedonly = no

# You can specify a folder translator.  This must be a eval-able
# Python expression that takes a foldername arg and returns the new
# value.  I suggest a lambda.  This example below will remove "INBOX." from
# the leading edge of folders (great for Courier IMAP users)
#
# See the user documentation for details and use cases. They are also
# online at:
# http://docs.offlineimap.org/en/latest/nametrans.html
#
# WARNING: you MUST construct this such that it NEVER returns
# the same value for two folders, UNLESS the second values are
# filtered out by folderfilter below.  Failure to follow this rule
# will result in undefined behavior
#
# nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)

# Using Courier remotely and want to duplicate its mailbox naming
# locally?  Try this:
#
# nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)

# You can specify which folders to sync using the folderfilter
# setting. You can provide any python function (e.g. a lambda function)
# which will be invoked for each foldername. If the filter function
# returns True, the folder will be synced, if it returns False, it. The
# folderfilter operates on the *UNTRANSLATED* name (before any nametrans
# translation takes place).
#
# Example 1: synchronizing only INBOX and Sent.
#
# folderfilter = lambda foldername: foldername in ['INBOX', 'Sent']
#
# Example 2: synchronizing everything except Trash.
#
# folderfilter = lambda foldername: foldername not in ['Trash']
#
# Example 3: Using a regular expression to exclude Trash and all folders
# containing the characters "Del".
#
# folderfilter = lambda foldername: not re.search('(^Trash$|Del)', foldername)
#
# If folderfilter is not specified, ALL remote folders will be
# synchronized.
#
# You can span multiple lines by indenting the others.  (Use backslashes
# at the end when required by Python syntax)  For instance:
#
# folderfilter = lambda foldername: foldername in
#        ['INBOX', 'Sent Mail', 'Deleted Items',
#         'Received']
folderfilter = lambda foldername: False

# You can specify folderincludes to include additional folders.  It
# should return a Python list.  This might be used to include a folder
# that was excluded by your folderfilter rule, to include a folder that
# your server does not specify with its LIST option, or to include a
# folder that is outside your basic reference. The 'reference' value
# will not be prefixed to this folder name, even if you have specified
# one.  For example:
# folderincludes = ['debian.user', 'debian.personal']
folderincludes = ['INBOX', '[Gmail]/All Mail']

# If you do not want to have any folders created on this repository,
# set the createfolders variable to False, the default is True. Using
# this feature you can e.g. disable the propagation of new folders to
# the new repository.
#createfolders = True


# You can specify 'foldersort' to determine how folders are sorted.
# This affects order of synchronization and mbnames. The expression
# should return -1, 0, or 1, as the default Python cmp() does.  The two
# arguments, x and y, are strings representing the names of the folders
# to be sorted.  The sorting is applied *AFTER* nametrans, if any.  The
# default is to sort IMAP folders alphabetically
# (case-insensitive). Usually, you should never have to modify this.  To
# eg. reverse the sort:
#
# foldersort = lambda x, y: -cmp(x, y)

# Enable 1-way synchronization. When setting 'readonly' to True, this
# repository will not be modified during synchronization. Use to
# e.g. backup an IMAP server. The readonly setting can be applied to any
# type of Repository (Maildir, Imap, etc).
#
#readonly = False

# [Repository GmailExample]

# A repository using Gmail's IMAP interface.  Any configuration
# parameter of `IMAP` type repositories can be used here.  Only
# `remoteuser` (or `remoteusereval` ) is mandatory.  Default values
# for other parameters are OK, and you should not need fiddle with
# those.
#
# The Gmail repository will use hard-coded values for `remotehost`,
# `remoteport`, `tunnel` and `ssl`.  (See
# http://mail.google.com/support/bin/answer.py?answer=78799&topic=12814)
# Any attempt to set those parameters will be silently ignored.

# type = Gmail

# Specify the Gmail user name. This is the only mandatory parameter.
# remoteuser = username@gmail.com

# The trash folder name may be different from [Gmail]/Trash
# for example on german googlemail, this setting should be
#
# trashfolder = [Google Mail]/Papierkorb
#
# The same is valid for the spam folder
#
# spamfolder = [Google Mail]/Spam

# Enable 1-way synchronization. See above for explanation.
#
#readonly = False

Setup Fail2ban for NextCloud

2018/10/05 by sudo 2 Comments

Running NextCloud or OwnCloud online comes with some risk, as with any online service. It’s important that your installation remains secure against hackers (or at least as secure as it can be). I’ve opted to implement fail2ban in order to help secure it using some custom rules. It’s worth noting that NextCloud does block unwanted login attempts itself through the application, but you’re having to trust application level security. I feel far safer having fail2ban implement firewall rules to prevent access to anyone probing the server.

First thing to do is create the NextCloud filter configuration file. This file will contain the regex that’s used to scan the logs for anything we don’t like the look of in order to block attacking hosts. My understanding is that this file can remain the same for OwnCloud, although I do not currently have a running instance of it to check.

sudo nano /etc/fail2ban/filter.d/nextcloud.conf

Add the following to the file:

[Definition] failregex=^{"reqId":".","remoteAddr":".","app":"core","message":"Login failed: '.' (Remote IP: '')","level":2,"time":"."}$ ^{"reqId":".","level":2,"time":".","remoteAddr":".","app":"core".","message":"Login failed: '.' (Remote IP: '')".}$ ^.\"remoteAddr\":\"\".Trusted domain error.*$

There are three regular expressions included here. The first and second checks for login failures, and flags the source IP. The third checks for trusted domain errors – which are usually a result of bots accessing your installation via it’s IP, not via it’s domain (thus, suspicious and I wanted to block them).

Once the file is saved, you can test what the filter would report by running the following command. This is entirely optional (although would help identify issues) and isn’t required for the rest of the steps to work.

sudo fail2ban-regex /var/nextcloud/data/nextcloud.log /etc/fail2ban/filter.d/nextcloud.conf -v

Next, the configuration file needs setup to activate the configurations we’ve just created. Never edit the Fail2ban jail.conf file, it’s likely to be overridden on upgrades. Always create a “.local” file, ideally a separate one for each application or rule you’re setting up (why? because it makes things more organised and easier to manage one rule over another!) inside the jail.d directory. With this in mind, create a nextcloud (or owncloud) file:

sudo nano /etc/fail2ban/jail.d/nextcloud.local

And add the following to it:

[nextcloud] ignoreip = 192.168.1.0/24 backend = auto enabled = true port = 80,443 protocol = tcp filter = nextcloud maxretry = 3 bantime = 36000 findtime = 36000 logpath = /var/nextcloud/data/nextcloud.log

Make sure your ignoreip is your local subnet or IP address. I opted to allow my whole LAN to access it without being auto-blocked. I’ve enabled the rule, set the ports to 80 (HTTP) and 443 (HTTPS) and configured ban times, etc. The most important things are the filter which should match the name of the file that was created inside the filter.d directory (excluding extension), and the log path, which may vary by installation. This path is the default for Ubuntu.

Once done, run the following command to restart nextcloud:

sudo service fail2ban restart

You can check the status of the jail by running:

sudo fail2ban-client status nextcloud

You’ll see something similar to this:

Status for the jail: nextcloud
|- Filter
|  |- Currently failed: 13
|  |- Total failed: 82
|  `- File list:    /var/nextcloud/data/nextcloud.log
`- Actions
   |- Currently banned: 0
   |- Total banned: 5
   `- Banned IP list:

Setting up a KVM host on Ubuntu 18.04 with bridged networking

2018/09/16 by sudo Leave a Comment

Since the launch of Ubuntu server 18.04, I’ve had a few people ask me how to setup bridged networking since the changeover to netplan for networking. The setup’s actually quite straight forwards once you know how. Start out by installing KVM and associated packages (I’m assuming you’re on a fresh Ubuntu 18.04 box):

sudo apt install bridge-utils qemu-kvm libvirt-bin

You should now be able to access virsh, to test it just run the command to list the virtual machines on the host (this will of course output an empty list. We just want to make sure the command hasn’t failed before going further):

virsh list --all


 Id    Name                           State

Now edit the netplan file for the networking configuration of the host system:
sudo nano /etc/netplan/50-cloud-init.yaml

The configuration should be edited to look similar to this:

network: ethernets: eno1: addresses: [] dhcp4: true optional: true version: 2 bridges: br0: interfaces: [eno1] dhcp4: true parameters: stp: false forward-delay: 0

We’re basically defining a single ethernet port (eno1) and assigning it to a bride. Note that you can check your ethernet port name using the ‘ip address’ command on the command line and it will return your network adapters and IP addresses. It should be resonably obvious which one is your ethernet adapter from the list!

It’s worth noting that in this setup, we’re allowing DHCP to allocate addresses to the machine. If this isn’t what you want, take a look at the netplan examples here: https://netplan.io/examples

You can apply these change using:

sudo netplan apply

And check it’s working:

ip address

networkctl list

If you’ve gotten this far without a network issue, you should be able to create machines using your favorite tool. I use Virtual Machine Manager (https://virt-manager.org/) which provides a reasonable user interface as well as remote management via SSH.

Setting up a Bond and Bridge in Netplan on Ubuntu 18.04

2018/09/12 by sudo 2 Comments

For some of my Ubuntu 18.04 servers, I need to run KVM virtual machines which require a bridge to the network so the machines get public LAN IP addresses and aren’t hidden behind NAT. With the server configuration for both my co-location and servers at work the network interfaces are all bonded for fail-over. This means I need a bond to have a bridge ontop of it for the virtual machines to get public IP addresses, while still allowing for failover of the network connection in the event of a network failure.

Research

There are some good examples of setting up netplan here: https://netplan.io/examples

They have a bridge example:

network:
  version: 2
  renderer: networkd
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - enp3s0

And a bond example:

network:
  version: 2
  renderer: networkd
  bonds:
    bond0:
      dhcp4: yes
      interfaces:
        - enp3s0
        - enp4s0
      parameters:
        mode: active-backup
        primary: enp3s0

But there’s not a clear indication of how to amalgamate the two.

A bond and a bridge

Here’s what I’ve ended up with in ‘/etc/netplan/50-cloud-init.yaml’:

network:
    bridges:
        br0:
            addresses:
            - 192.168.10.30
            dhcp4: false
            gateway4: 192.168.10.1
            nameservers:
                addresses:
                - 192.168.10.1
                - 192.168.10.2
                search: []
            interfaces:
                - bond0
    bonds:
        bond0:
            interfaces:
            - eno1
            - eno2
            parameters:
                mode: active-backup
    ethernets:
        eno1:
            addresses: []
            dhcp4: false
            dhcp6: false
        eno2:
            addresses: []
            dhcp4: false
            dhcp6: false

Note that I’ve obviously defined static IP addresses, but this isn’t a requirement. Just set ‘dhcp4: true’ and remove the ‘address’, ‘gateway’ and ‘nameserver’ sections if you’re using DHCP.

Once the file’s got that setup in it, it’s possible to run:
sudo netplan apply
and you should be able to run ‘networkctl list’ to check the bridge and bond are setup.

Share this:

Share this:

Share this:

Id Name State

Share this:

Research

A bond and a bridge

Share this: