apt get life - Life around technology

Backing up email using offlineimap

2019/03/06 by sudo

I’ve been asked to help backup a friends email from their existing provider. Since we’re not sure what’ll happen to the existing email after the request to change service providers has been made, it seems reasonable to backup her existing email automatically.

Since I run my own mail server, I thought it’d be quite easy to run something like offlineimap to backup the existing service. One caveat to that has been the existing server prefixes “INBOX” to the mail directory, which my setup doesn’t support. It looks like this is a difference between Courier and Dovecot. I also want to rename “Sent Items” to “.Sent” to follow my Dovecot setup.

Fortunately, offlineimap offers the ability to translate folders on the fly.

.offlineimaprc

[general]
accounts = alice

[Account nina]
localrepository = Local
remoterepository = Remote

[Repository Local]
type = Maildir
localfolders = ~/Mail/Alice

[Repository Remote]
type = IMAP
readonly = true
remotehost = mail.example.com
remoteuser = alice@example.com
remotepass = secret
# Necessary as of OfflineIMAP 6.5.4
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
# Necessary to work around https://github.com/OfflineIMAP/offlineimap/issues/573 (versions 7.0.12, 7.2.1)
ssl_version = tls1_2
nametrans = lambda foldername: re.sub('^INBOX\.*', '.',
                               re.sub('^Sent Items\.*', '.Sent', foldername))

[general]

accounts = alice

[Account nina]

localrepository = Local

remoterepository = Remote

[Repository Local]

type = Maildir

localfolders = ~/Mail/Alice

[Repository Remote]

type = IMAP

readonly = true

remotehost = mail.example.com

remoteuser = alice@example.com

remotepass = secret

# Necessary as of OfflineIMAP 6.5.4

sslcacertfile = /etc/ssl/certs/ca-certificates.crt

# Necessary to work around https://github.com/OfflineIMAP/offlineimap/issues/573 (versions 7.0.12, 7.2.1)

ssl_version = tls1_2

nametrans = lambda foldername: re.sub('^INBOX\.*', '.',

re.sub('^Sent Items\.*', '.Sent', foldername))

Once I’ve got this setup, I can use crontabs to automatically sync it every x minutes.

Note that after pulling the email, Dovecot may need the quota recalculated manually:

doveadm quota get -u alice@example.com
doveadm quota recalc -u alice@example.com

1 2	doveadm quota get -u alice@example.com doveadm quota recalc -u alice@example.com

https://wiki2.dovecot.org/Tools/Doveadm/Quota

Here are some good resources:

https://github.com/OfflineIMAP/offlineimap/issues/417

[general]

metadata = $XDG_DATA_HOME/offlineimap
accounts = darnirmlist
maxsyncaccounts = 1
ui = basic
pythonfile = $XDG_DATA_HOME/offlineimap/passkey.py

socktimeout = 20
fsync = false

[DEFAULT]
localrepositoryhome = ~/.local/var/lib/mail

[Account darnirmlist]
localrepository = darnirmlist-local
remoterepository = darnirmlist-remote
status_backend = sqlite
autorefresh = 1
quick = 20
synclabels = yes
ignorelabels = \Inbox, \Starred, \Sent, \Draft, \Spam, \Trash, \Important

[Repository darnirmlist-local]
type = GmailMaildir
localfolders = %(localrepositoryhome)s/darnirmlist

[Repository darnirmlist-remote]
type = Gmail
remoteuser = darnirmlist@gmail.com

auth_mechanisms = XOAUTH2
oauth2_client_id_eval = get_client_id('darnirmlist@gmail.com')
oauth2_client_secret_eval = get_client_secret('darnirmlist@gmail.com')
oauth2_refresh_token_eval = get_refresh_token('darnirmlist@gmail.com')
oauth2_request_url = https://accounts.google.com/o/oauth2/token
nametrans = lambda folder: re.sub('.*Spam$', 'Spam',
                           re.sub('.*INBOX$', 'Inbox',
                           re.sub('.*Drafts$', 'Drafts',
                           re.sub('.*Sent Mail$', 'Sent',
                           re.sub('.*All Mail', 'Archive',
                           re.sub('.*Trash$', 'Trash', folder))))))
folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail',
                                                     '[Gmail]/Drafts',
                                                     '[Gmail]/Sent Mail',
                                                     '[Gmail]/Spam']
createfolders = False
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
realdelete = no
holdconnectionopen = yes
keepalive = 50

[general]

metadata = $XDG_DATA_HOME/offlineimap

accounts = darnirmlist

maxsyncaccounts = 1

ui = basic

pythonfile = $XDG_DATA_HOME/offlineimap/passkey.py

socktimeout = 20

fsync = false

[DEFAULT]

localrepositoryhome = ~/.local/var/lib/mail

[Account darnirmlist]

localrepository = darnirmlist-local

remoterepository = darnirmlist-remote

status_backend = sqlite

autorefresh = 1

quick = 20

synclabels = yes

ignorelabels = \Inbox, \Starred, \Sent, \Draft, \Spam, \Trash, \Important

[Repository darnirmlist-local]

type = GmailMaildir

localfolders = %(localrepositoryhome)s/darnirmlist

[Repository darnirmlist-remote]

type = Gmail

remoteuser = darnirmlist@gmail.com

auth_mechanisms = XOAUTH2

oauth2_client_id_eval = get_client_id('darnirmlist@gmail.com')

oauth2_client_secret_eval = get_client_secret('darnirmlist@gmail.com')

oauth2_refresh_token_eval = get_refresh_token('darnirmlist@gmail.com')

oauth2_request_url = https://accounts.google.com/o/oauth2/token

nametrans = lambda folder: re.sub('.*Spam$', 'Spam',

re.sub('.*INBOX$', 'Inbox',

re.sub('.*Drafts$', 'Drafts',

re.sub('.*Sent Mail$', 'Sent',

re.sub('.*All Mail', 'Archive',

re.sub('.*Trash$', 'Trash', folder))))))

folderfilter = lambda foldername: foldername not in ['[Gmail]/All Mail',

'[Gmail]/Drafts',

'[Gmail]/Sent Mail',

'[Gmail]/Spam']

createfolders = False

sslcacertfile = /etc/ssl/certs/ca-certificates.crt

realdelete = no

holdconnectionopen = yes

keepalive = 50

http://www.offlineimap.org/doc/use_cases.html

https://gist.githubusercontent.com/dabrahams/3030332/raw/6e0cfab9982a97d0921d774d9f0d070dccb3b061/gistfile1.cfg

# -*- mode: conf; -*-
# 
# NOTE: Settings generally support python interpolation. This means
# values can contain python format strings which refer to other values
# in the same section, or values in a special DEFAULT section. This
# allows you for example to use common settings for multiple accounts:
#
# [Repository Gmail1]
# trashfolder: %(gmailtrashfolder)s
#
# [Repository Gmail2]
# trashfolder: %(gmailtrashfolder)s
#
# [DEFAULT]
# gmailtrashfolder = [Google Mail]/Papierkorb
#
# would set the trashfolder setting for your German gmail accounts.

# NOTE2: This implies that any '%' needs to be encoded as '%%'

##################################################
# General definitions
##################################################

[general]

# This specifies where offlineimap is to store its metadata.
# This directory will be created if it does not already exist.

#metadata = ~/.offlineimap

# This variable specifies which accounts are defined.  Separate them
# with commas.  Account names should be alphanumeric only.
# You will need to specify one section per account below.  You may
# not use "general" for an account name.

accounts = BoostPro

# Offlineimap can synchronize more than one account at a time.  If you
# want to enable this feature, set the below value to something
# greater than 1.  To force it to synchronize only one account at a
# time, set it to 1.
#
# Note: if you are using autorefresh and have more than one account,
# you must set this number to be >= to the number of accounts you have;
# since any given sync run never "finishes" due to a timer, you will never
# sync your additional accounts if this is 1.

#maxsyncaccounts = 1

# You can specify one or more user interface modules for OfflineIMAP
# to use.  OfflineIMAP will try the first in the list, and if it
# fails, the second, and so forth.
#
# The pre-defined options are:
# Blinkenlights -- A fancy (terminal) interface
# TTYUI         -- a text-based (terminal) interface
# Basic         -- Noninteractive interface suitable for cron'ing
# Quiet         -- Noninteractive interface, generates no output
#                  except for errors.
# MachineUI     -- Interactive interface suitable for machine
#                  parsing.
#
# You can override this with a command-line option -u.

#ui = basic

# If you try to synchronize messages to a folder which the IMAP server
# considers read-only, OfflineIMAP will generate a warning.  If you want
# to suppress these warnings, set ignore-readonly to yes.  Read-only
# IMAP folders allow reading but not modification, so if you try to
# change messages in the local copy of such a folder, the IMAP server
# will prevent OfflineIMAP from propagating those changes to the IMAP
# server.  Note that ignore-readonly is unrelated to the "readonly"
# setting which prevents a repository from being modified at all.

#ignore-readonly = no

########## Advanced settings

# You can give a Python source filename here and all config file
# python snippets will be evaluated in the context of that file.
# This allows you to e.g. define helper functions in the Python
# source file and call them from this config file.  You can find
# an example of this in the manual.
#
# pythonfile = ~/.offlineimap.py
pythonfile = ~/.offlineimap/offlineimap_config.py

# By default, OfflineIMAP will not exit due to a network error until
# the operating system returns an error code.  Operating systems can sometimes
# take forever to notice this.  Here you can activate a timeout on the
# socket.  This timeout applies to individual socket reads and writes,
# not to an overall sync operation.  You could perfectly well have a 30s
# timeout here and your sync still take minutes.
#
# Values in the 30-120 second range are reasonable.
#
# The default is to have no timeout beyond the OS.  Times are given in seconds.
#
# socktimeout = 60

# By default, OfflineIMAP will use fsync() to force data out to disk at
# opportune times to ensure consistency.  This can, however, reduce
# performance.  Users where /home is on SSD (Flash) may also wish to reduce
# write cycles.  Therefore, you can disable OfflineIMAP's use of fsync().
# Doing so will come at the expense of greater risk of message duplication
# in the event of a system crash or power loss.  Default is fsync = true.
# Set fsync = false ot disable fsync.
#
# fsync = true

##################################################
# Mailbox name recorder
##################################################

[mbnames]

# offlineimap can record your mailbox names in a format you specify.
# You can define the header, each mailbox item, the separator,
# and the footer.  Here is an example for Mutt.
# If enabled is yes, all six setting must be specified, even if they
# are just the empty string "".
#
# The header, peritem, sep, and footer are all Python expressions passed
# through eval, so you can (and must) use Python quoting.

enabled = no
filename = ~/Mutt/muttrc.mailboxes
header = "mailboxes "
peritem = "+%(accountname)s/%(foldername)s"
sep = " "
footer = "\n"

# You can also specify a folderfilter.  It will apply to the
# *translated* folder name here, and it takes TWO arguments:
# accountname and foldername.  In all other ways, it will
# behave identically to the folderfilter for accounts.  Please see
# that section for more information and examples.
#
# Note that this filter can be used only to further restrict mbnames
# to a subset of folders that pass the account's folderfilter.


##################################################
# Accounts
##################################################

# This is an account definition clause.  You'll have one of these
# for each account listed in general/accounts above.

[Account BoostPro]
########## Basic settings

# These settings specify the two folders that you will be syncing.
# You'll need to have a "Repository ..." section for each one.

localrepository = BoostProLocal
remoterepository = BoostProRemote

########## Advanced settings

# You can have offlineimap continue running indefinitely, automatically
# syncing your mail periodically.  If you want that, specify how
# frequently to do that (in minutes) here.  You can also specify
# fractional minutes (ie, 3.25).

autorefresh = 5

# OfflineImap can replace a number of full updates by quick
# synchronizations.  It only synchronizes a folder if 1) a Maildir
# folder has changed, or 2) if an IMAP folder has received new messages
# or had messages deleted, ie it does not update if only IMAP flags have
# changed.  Full updates need to fetch ALL flags for all messages, so
# this makes quite a performance difference (especially if syncing
# between two IMAP servers).
# Specify 0 for never, -1 for always (works even in non-autorefresh
# mode), or a positive integer <n> to do <n> quick updates before doing
# another full synchronization (requires autorefresh).  Updates are
# always performed after <autorefresh> minutes, be they quick or full.

quick = 24

# You can specify a pre and post sync hook to execute a external command.
# In this case a call to imapfilter to filter mail before the sync process
# starts and a custom shell script after the sync completes.
# The pre sync script has to complete before a sync to the account will
# start. 

# presynchook = imapfilter
# postsynchook = notifysync.sh
postsynchook = /Users/dave/bin/imap-postsync localhost dave --soft-errors

# You can also specify parameters to the commands
# presynchook = imapfilter -c someotherconfig.lua

# OfflineImap caches the state of the synchronisation to e.g. be able to
# determine if a mail has been deleted on one side or added on the
# other.
#
# The default and historical backend is 'plain' which writes out the
# state in plain text files. On Repositories with large numbers of
# mails, the performance might not be optimal, as we write out the
# complete file for each change.  Another new backend 'sqlite' is
# available which stores the status in sqlite databases.
#
# If you switch the backend, you may want to delete the old cache
# directory in ~/.offlineimap/Account-<account>/LocalStatus manually
# once you are sure that things work.
#
status_backend = sqlite

# If you have a limited amount of bandwidth available you can exclude larger
# messages (e.g. those with large attachments etc).  If you do this it
# will appear to offlineimap that these messages do not exist at all.  They
# will not be copied, have flags changed etc.  For this to work on an IMAP
# server the server must have server side search enabled.  This works with gmail
# and most imap servers (e.g. cyrus etc)
# The maximum size should be specified in bytes - e.g. 2000000 for approx 2MB

# maxsize = 2000000


# When you are starting to sync an already existing account you can tell
# offlineimap to sync messages from only the last x days.  When you do
# this messages older than x days will be completely ignored.  This can
# be useful for importing existing accounts when you do not want to
# download large amounts of archive email.
#
# Messages older than maxage days will not be synced, their flags will
# not be changed, they will not be deleted etc.  For offlineimap it will
# be like these messages do not exist.  This will perform an IMAP search
# in the case of IMAP or Gmail and therefor requires that the server
# support server side searching.  This will calculate the earliest day
# that would be included in the search and include all messages from
# that day until today.  e.g. maxage = 3 to sync only the last 3 days
# mail
#
# maxage =
maxage = 365

# Maildir file format uses colon (:) separator between uniq name and info.
# Unfortunatelly colon is not allowed character in windows file name. If you
# enable maildir-windows-compatible option, offlineimap will be able to store
# messages on windows drive, but you will probably loose compatibility with
# other programs working with the maildir
#
#maildir-windows-compatible = no


[Repository BoostProLocal]

# Each repository requires a "type" declaration. The types supported for
# local repositories are Maildir and IMAP.

type = IMAP

# Specify local repository.  Your IMAP folders will be synchronized
# to maildirs created under this path.  OfflineIMAP will create the
# maildirs for you as needed.

# localfolders = ~/Test

# You can specify the "folder separator character" used for your Maildir
# folders.  It is inserted in-between the components of the tree. If you
# want your folders to be nested directories, set it to "/". 'sep' is
# ignored for IMAP repositories, as it is queried automatically.
#
#sep = .

# Some users may not want the atime (last access time) of folders to be
# modified by OfflineIMAP.  If 'restoreatime' is set to yes, OfflineIMAP
# will restore the atime of the "new" and "cur" folders in each maildir
# folder to their original value after each sync.
#
# In nearly all cases, the default should be fine.
#
#restoreatime = no

# This setting will be forced to "yes" by the fact because we have
# enabled IDLE support
#
# holdconnectionopen = yes
preauthtunnel = /usr/local/libexec/dovecot/imap

[Repository BoostProRemote]
# And this is the remote repository.  We only support IMAP or Gmail here.

type = Gmail

# The following can fetch the account credentials via a python expression that
# is parsed from the pythonfile parameter. For example, a function called
# "getcredentials" that parses a file "filename" and returns the account
# details for "hostname".
# remotehosteval = getcredentials("filename", "hostname", "hostname")
# remoteusereval = getcredentials("filename", "hostname", "user")
# remotepasseval = getcredentials("filename", "hostname", "passwd")
remotepasseval = osxkeychain('imap.gmail.com', account='dave@boostpro.com')

# Specify the remote hostname.
# remotehost = examplehost

# Whether or not to use SSL.
ssl = yes

# SSL Client certificate (optional)
# sslclientcert = /path/to/file.crt

# SSL Client key (optional)
# sslclientkey = /path/to/file.key

# SSL CA Cert(s) to verify the server cert against (optional).
# No SSL verification is done without this option. If it is
# specified, the CA Cert(s) need to verify the Server cert AND
# match the hostname (* wildcard allowed on the left hand side)
# The certificate should be in PEM format.
# sslcacertfile = /path/to/cacertfile.crt

# If you connect via SSL/TLS (ssl=true) and you have no CA certificate
# specified, offlineimap will refuse to sync as it connects to a server
# with an unknown "fingerprint". If you are sure you connect to the
# correct server, you can then configure the presented server
# fingerprint here. OfflineImap will verify that the server fingerprint
# has not changed on each connect and refuse to connect otherwise.
# You can also configure this in addition to CA certificate validation
# above and it will check both ways.

#cert_fingerprint = <SHA1_of_server_certificate_here>
cert_fingerprint = f3043dd689a2e7dddfbef82703a6c65ea9b634c1

# Specify the port.  If not specified, use a default port.
# remoteport = 993

# Specify the remote user name.
remoteuser = dave@boostpro.com

# There are six ways to specify the password for the IMAP server:
#
# 1. No password at all specified in the config file.
#    If a matching entry is found in ~/.netrc (see netrc (5) for
#    information) this password will be used. Do note that netrc only
#    allows one entry per hostname. If there is no ~/.netrc file but
#    there is an /etc/netrc file, the password will instead be taken
#    from there. Otherwise you will be prompted for the password when
#    OfflineIMAP starts when using a UI that supports this.
#
# 2. The remote password stored in this file with the remotepass
#    option. Any '%' needs to be encoded as '%%'. Example:
#    remotepass = mypassword
#
# 3. The remote password stored as a single line in an external
#    file, which is referenced by the remotefile option.  Example:
#    remotepassfile = ~/Password.IMAP.Account1
#
# 4. With a preauth tunnel.  With this method, you invoke an external
#    program that is guaranteed *NOT* to ask for a password, but rather
#    to read from stdin and write to stdout an IMAP procotol stream that
#    begins life in the PREAUTH state.  When you use a tunnel, you do
#    NOT specify a user or password (if you do, they'll be ignored.)
#    Instead, you specify a preauthtunnel, as this example illustrates
#    for Courier IMAP on Debian:
#    preauthtunnel = ssh -q imaphost '/usr/bin/imapd ./Maildir'
#
# 5. If you are using Kerberos and have the Python Kerberos package
#    installed, you should not specify a remotepass.  If the user has a
#    valid Kerberos TGT, OfflineIMAP will figure out the rest all by
#    itself, and fall back to password authentication if needed.
#
# 6. Using arbitrary python code.  With this method, you invoke a
#    function from your pythonfile.  To use this method assign the name
#    of the function to the variable 'remotepasseval'.  Example:
#    remotepasseval = get_password("imap.example.net")
#    You can also query for the username:
#    remoteusereval = get_username("imap.example.net")
#    This method can be used to design more elaborate setups, e.g. by
#    querying the gnome-keyring via its python bindings.

########## Advanced settings

# Some IMAP servers need a "reference" which often refers to the "folder
# root".  This is most commonly needed with UW IMAP, where you might
# need to specify the directory in which your mail is stored. The
# 'reference' value will be prefixed to all folder paths refering to
# that repository. E.g. accessing folder 'INBOX' with reference = Mail
# will try to access Mail/INBOX. Note that the nametrans and
# folderfilter functions will still apply the full path including the
# reference prefix.  Most users will not need this.
#
# reference = Mail

# In between synchronisations, OfflineIMAP can monitor mailboxes for new
# messages using the IDLE command. If you want to enable this, specify here
# the folders you wish to monitor. Note that the IMAP protocol requires a
# separate connection for each folder monitored in this way, so setting
# this option will force settings for:
#     maxconnections - to be at least the number of folders you give
#     holdconnectionopen - to be true
#     keepalive - to be 29 minutes unless you specify otherwise
#
# This feature isn't complete and may well have problems. See the manual
# for more details.
#
# This option should return a Python list. For example
#
# idlefolders = ['INBOX', 'INBOX.Alerts']
#
idlefolders = %(folderincludes)s

# OfflineIMAP can use multiple connections to the server in order
# to perform multiple synchronization actions simultaneously.
# This may place a higher burden on the server.  In most cases,
# setting this value to 2 or 3 will speed up the sync, but in some
# cases, it may slow things down.  The safe answer is 1.  You should
# probably never set it to a value more than 5.

#maxconnections = 2

# OfflineIMAP normally closes IMAP server connections between refreshes if
# the global option autorefresh is specified.  If you wish it to keep the
# connection open, set this to true.  If not specified, the default is
# false.  Keeping the connection open means a faster sync start the
# next time and may use fewer server resources on connection, but uses
# more server memory.  This setting has no effect if autorefresh is not set.
#
#holdconnectionopen = no

# If you want to have "keepalives" sent while waiting between syncs,
# specify the amount of time IN SECONDS between keepalives here.  Note that
# sometimes more than this amount of time might pass, so don't make it
# tight.  This setting has no effect if autorefresh and holdconnectionopen
# are not both set.
#
# keepalive = 60

# Normally, OfflineIMAP will expunge deleted messages from the server.
# You can disable that if you wish.  This means that OfflineIMAP will
# mark them deleted on the server, but not actually delete them.
# You must use some other IMAP client to delete them if you use this
# setting; otherwise, the messgaes will just pile up there forever.
# Therefore, this setting is definitely NOT recommended.
#
#expunge = no

# Specify whether to process all mail folders on the server, or only
# those listed as "subscribed".
#
#subscribedonly = no

# You can specify a folder translator.  This must be a eval-able
# Python expression that takes a foldername arg and returns the new
# value.  I suggest a lambda.  This example below will remove "INBOX." from
# the leading edge of folders (great for Courier IMAP users)
#
# See the user documentation for details and use cases. They are also
# online at:
# http://docs.offlineimap.org/en/latest/nametrans.html
#
# WARNING: you MUST construct this such that it NEVER returns
# the same value for two folders, UNLESS the second values are
# filtered out by folderfilter below.  Failure to follow this rule
# will result in undefined behavior
#
# nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)

# Using Courier remotely and want to duplicate its mailbox naming
# locally?  Try this:
#
# nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)

# You can specify which folders to sync using the folderfilter
# setting. You can provide any python function (e.g. a lambda function)
# which will be invoked for each foldername. If the filter function
# returns True, the folder will be synced, if it returns False, it. The
# folderfilter operates on the *UNTRANSLATED* name (before any nametrans
# translation takes place).
#
# Example 1: synchronizing only INBOX and Sent.
#
# folderfilter = lambda foldername: foldername in ['INBOX', 'Sent']
#
# Example 2: synchronizing everything except Trash.
#
# folderfilter = lambda foldername: foldername not in ['Trash']
#
# Example 3: Using a regular expression to exclude Trash and all folders
# containing the characters "Del".
#
# folderfilter = lambda foldername: not re.search('(^Trash$|Del)', foldername)
#
# If folderfilter is not specified, ALL remote folders will be
# synchronized.
#
# You can span multiple lines by indenting the others.  (Use backslashes
# at the end when required by Python syntax)  For instance:
#
# folderfilter = lambda foldername: foldername in
#        ['INBOX', 'Sent Mail', 'Deleted Items',
#         'Received']
folderfilter = lambda foldername: False

# You can specify folderincludes to include additional folders.  It
# should return a Python list.  This might be used to include a folder
# that was excluded by your folderfilter rule, to include a folder that
# your server does not specify with its LIST option, or to include a
# folder that is outside your basic reference. The 'reference' value
# will not be prefixed to this folder name, even if you have specified
# one.  For example:
# folderincludes = ['debian.user', 'debian.personal']
folderincludes = ['INBOX', '[Gmail]/All Mail']

# If you do not want to have any folders created on this repository,
# set the createfolders variable to False, the default is True. Using
# this feature you can e.g. disable the propagation of new folders to
# the new repository.
#createfolders = True


# You can specify 'foldersort' to determine how folders are sorted.
# This affects order of synchronization and mbnames. The expression
# should return -1, 0, or 1, as the default Python cmp() does.  The two
# arguments, x and y, are strings representing the names of the folders
# to be sorted.  The sorting is applied *AFTER* nametrans, if any.  The
# default is to sort IMAP folders alphabetically
# (case-insensitive). Usually, you should never have to modify this.  To
# eg. reverse the sort:
#
# foldersort = lambda x, y: -cmp(x, y)

# Enable 1-way synchronization. When setting 'readonly' to True, this
# repository will not be modified during synchronization. Use to
# e.g. backup an IMAP server. The readonly setting can be applied to any
# type of Repository (Maildir, Imap, etc).
#
#readonly = False

# [Repository GmailExample]

# A repository using Gmail's IMAP interface.  Any configuration
# parameter of `IMAP` type repositories can be used here.  Only
# `remoteuser` (or `remoteusereval` ) is mandatory.  Default values
# for other parameters are OK, and you should not need fiddle with
# those.
#
# The Gmail repository will use hard-coded values for `remotehost`,
# `remoteport`, `tunnel` and `ssl`.  (See
# http://mail.google.com/support/bin/answer.py?answer=78799&topic=12814)
# Any attempt to set those parameters will be silently ignored.

# type = Gmail

# Specify the Gmail user name. This is the only mandatory parameter.
# remoteuser = username@gmail.com

# The trash folder name may be different from [Gmail]/Trash
# for example on german googlemail, this setting should be
#
# trashfolder = [Google Mail]/Papierkorb
#
# The same is valid for the spam folder
#
# spamfolder = [Google Mail]/Spam

# Enable 1-way synchronization. See above for explanation.
#
#readonly = False

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

# -*- mode: conf; -*-

# NOTE: Settings generally support python interpolation. This means

# values can contain python format strings which refer to other values

# in the same section, or values in a special DEFAULT section. This

# allows you for example to use common settings for multiple accounts:

# [Repository Gmail1]

# trashfolder: %(gmailtrashfolder)s

# [Repository Gmail2]

# trashfolder: %(gmailtrashfolder)s

# [DEFAULT]

# gmailtrashfolder = [Google Mail]/Papierkorb

# would set the trashfolder setting for your German gmail accounts.

# NOTE2: This implies that any '%' needs to be encoded as '%%'

##################################################

# General definitions

##################################################

[general]

# This specifies where offlineimap is to store its metadata.

# This directory will be created if it does not already exist.

#metadata = ~/.offlineimap

# This variable specifies which accounts are defined. Separate them

# with commas. Account names should be alphanumeric only.

# You will need to specify one section per account below. You may

# not use "general" for an account name.

accounts = BoostPro

# Offlineimap can synchronize more than one account at a time. If you

# want to enable this feature, set the below value to something

# greater than 1. To force it to synchronize only one account at a

# time, set it to 1.

# Note: if you are using autorefresh and have more than one account,

# you must set this number to be >= to the number of accounts you have;

# since any given sync run never "finishes" due to a timer, you will never

# sync your additional accounts if this is 1.

#maxsyncaccounts = 1

# You can specify one or more user interface modules for OfflineIMAP

# to use. OfflineIMAP will try the first in the list, and if it

# fails, the second, and so forth.

# The pre-defined options are:

# Blinkenlights -- A fancy (terminal) interface

# TTYUI -- a text-based (terminal) interface

# Basic -- Noninteractive interface suitable for cron'ing

# Quiet -- Noninteractive interface, generates no output

# except for errors.

# MachineUI -- Interactive interface suitable for machine

# parsing.

# You can override this with a command-line option -u.

#ui = basic

# If you try to synchronize messages to a folder which the IMAP server

# considers read-only, OfflineIMAP will generate a warning. If you want

# to suppress these warnings, set ignore-readonly to yes. Read-only

# IMAP folders allow reading but not modification, so if you try to

# change messages in the local copy of such a folder, the IMAP server

# will prevent OfflineIMAP from propagating those changes to the IMAP

# server. Note that ignore-readonly is unrelated to the "readonly"

# setting which prevents a repository from being modified at all.

#ignore-readonly = no

########## Advanced settings

# You can give a Python source filename here and all config file

# python snippets will be evaluated in the context of that file.

# This allows you to e.g. define helper functions in the Python

# source file and call them from this config file. You can find

# an example of this in the manual.

# pythonfile = ~/.offlineimap.py

pythonfile = ~/.offlineimap/offlineimap_config.py

# By default, OfflineIMAP will not exit due to a network error until

# the operating system returns an error code. Operating systems can sometimes

# take forever to notice this. Here you can activate a timeout on the

# socket. This timeout applies to individual socket reads and writes,

# not to an overall sync operation. You could perfectly well have a 30s

# timeout here and your sync still take minutes.

# Values in the 30-120 second range are reasonable.

# The default is to have no timeout beyond the OS. Times are given in seconds.

# socktimeout = 60

# By default, OfflineIMAP will use fsync() to force data out to disk at

# opportune times to ensure consistency. This can, however, reduce

# performance. Users where /home is on SSD (Flash) may also wish to reduce

# write cycles. Therefore, you can disable OfflineIMAP's use of fsync().

# Doing so will come at the expense of greater risk of message duplication

# in the event of a system crash or power loss. Default is fsync = true.

# Set fsync = false ot disable fsync.

# fsync = true

##################################################

# Mailbox name recorder

##################################################

[mbnames]

# offlineimap can record your mailbox names in a format you specify.

# You can define the header, each mailbox item, the separator,

# and the footer. Here is an example for Mutt.

# If enabled is yes, all six setting must be specified, even if they

# are just the empty string "".

# The header, peritem, sep, and footer are all Python expressions passed

# through eval, so you can (and must) use Python quoting.

enabled = no

filename = ~/Mutt/muttrc.mailboxes

header = "mailboxes "

peritem = "+%(accountname)s/%(foldername)s"

sep = " "

footer = "\n"

# You can also specify a folderfilter. It will apply to the

# *translated* folder name here, and it takes TWO arguments:

# accountname and foldername. In all other ways, it will

# behave identically to the folderfilter for accounts. Please see

# that section for more information and examples.

# Note that this filter can be used only to further restrict mbnames

# to a subset of folders that pass the account's folderfilter.

##################################################

# Accounts

##################################################

# This is an account definition clause. You'll have one of these

# for each account listed in general/accounts above.

[Account BoostPro]

########## Basic settings

# These settings specify the two folders that you will be syncing.

# You'll need to have a "Repository ..." section for each one.

localrepository = BoostProLocal

remoterepository = BoostProRemote

########## Advanced settings

# You can have offlineimap continue running indefinitely, automatically

# syncing your mail periodically. If you want that, specify how

# frequently to do that (in minutes) here. You can also specify

# fractional minutes (ie, 3.25).

autorefresh = 5

# OfflineImap can replace a number of full updates by quick

# synchronizations. It only synchronizes a folder if 1) a Maildir

# folder has changed, or 2) if an IMAP folder has received new messages

# or had messages deleted, ie it does not update if only IMAP flags have

# changed. Full updates need to fetch ALL flags for all messages, so

# this makes quite a performance difference (especially if syncing

# between two IMAP servers).

# Specify 0 for never, -1 for always (works even in non-autorefresh

# mode), or a positive integer <n> to do <n> quick updates before doing

# another full synchronization (requires autorefresh). Updates are

# always performed after <autorefresh> minutes, be they quick or full.

quick = 24

# You can specify a pre and post sync hook to execute a external command.

# In this case a call to imapfilter to filter mail before the sync process

# starts and a custom shell script after the sync completes.

# The pre sync script has to complete before a sync to the account will

# start.

# presynchook = imapfilter

# postsynchook = notifysync.sh

postsynchook = /Users/dave/bin/imap-postsync localhost dave --soft-errors

# You can also specify parameters to the commands

# presynchook = imapfilter -c someotherconfig.lua

# OfflineImap caches the state of the synchronisation to e.g. be able to

# determine if a mail has been deleted on one side or added on the

# other.

# The default and historical backend is 'plain' which writes out the

# state in plain text files. On Repositories with large numbers of

# mails, the performance might not be optimal, as we write out the

# complete file for each change. Another new backend 'sqlite' is

# available which stores the status in sqlite databases.

# If you switch the backend, you may want to delete the old cache

# directory in ~/.offlineimap/Account-<account>/LocalStatus manually

# once you are sure that things work.

status_backend = sqlite

# If you have a limited amount of bandwidth available you can exclude larger

# messages (e.g. those with large attachments etc). If you do this it

# will appear to offlineimap that these messages do not exist at all. They

# will not be copied, have flags changed etc. For this to work on an IMAP

# server the server must have server side search enabled. This works with gmail

# and most imap servers (e.g. cyrus etc)

# The maximum size should be specified in bytes - e.g. 2000000 for approx 2MB

# maxsize = 2000000

# When you are starting to sync an already existing account you can tell

# offlineimap to sync messages from only the last x days. When you do

# this messages older than x days will be completely ignored. This can

# be useful for importing existing accounts when you do not want to

# download large amounts of archive email.

# Messages older than maxage days will not be synced, their flags will

# not be changed, they will not be deleted etc. For offlineimap it will

# be like these messages do not exist. This will perform an IMAP search

# in the case of IMAP or Gmail and therefor requires that the server

# support server side searching. This will calculate the earliest day

# that would be included in the search and include all messages from

# that day until today. e.g. maxage = 3 to sync only the last 3 days

# mail

# maxage =

maxage = 365

# Maildir file format uses colon (:) separator between uniq name and info.

# Unfortunatelly colon is not allowed character in windows file name. If you

# enable maildir-windows-compatible option, offlineimap will be able to store

# messages on windows drive, but you will probably loose compatibility with

# other programs working with the maildir

#maildir-windows-compatible = no

[Repository BoostProLocal]

# Each repository requires a "type" declaration. The types supported for

# local repositories are Maildir and IMAP.

type = IMAP

# Specify local repository. Your IMAP folders will be synchronized

# to maildirs created under this path. OfflineIMAP will create the

# maildirs for you as needed.

# localfolders = ~/Test

# You can specify the "folder separator character" used for your Maildir

# folders. It is inserted in-between the components of the tree. If you

# want your folders to be nested directories, set it to "/". 'sep' is

# ignored for IMAP repositories, as it is queried automatically.

#sep = .

# Some users may not want the atime (last access time) of folders to be

# modified by OfflineIMAP. If 'restoreatime' is set to yes, OfflineIMAP

# will restore the atime of the "new" and "cur" folders in each maildir

# folder to their original value after each sync.

# In nearly all cases, the default should be fine.

#restoreatime = no

# This setting will be forced to "yes" by the fact because we have

# enabled IDLE support

# holdconnectionopen = yes

preauthtunnel = /usr/local/libexec/dovecot/imap

[Repository BoostProRemote]

# And this is the remote repository. We only support IMAP or Gmail here.

type = Gmail

# The following can fetch the account credentials via a python expression that

# is parsed from the pythonfile parameter. For example, a function called

# "getcredentials" that parses a file "filename" and returns the account

# details for "hostname".

# remotehosteval = getcredentials("filename", "hostname", "hostname")

# remoteusereval = getcredentials("filename", "hostname", "user")

# remotepasseval = getcredentials("filename", "hostname", "passwd")

remotepasseval = osxkeychain('imap.gmail.com', account='dave@boostpro.com')

# Specify the remote hostname.

# remotehost = examplehost

# Whether or not to use SSL.

ssl = yes

# SSL Client certificate (optional)

# sslclientcert = /path/to/file.crt

# SSL Client key (optional)

# sslclientkey = /path/to/file.key

# SSL CA Cert(s) to verify the server cert against (optional).

# No SSL verification is done without this option. If it is

# specified, the CA Cert(s) need to verify the Server cert AND

# match the hostname (* wildcard allowed on the left hand side)

# The certificate should be in PEM format.

# sslcacertfile = /path/to/cacertfile.crt

# If you connect via SSL/TLS (ssl=true) and you have no CA certificate

# specified, offlineimap will refuse to sync as it connects to a server

# with an unknown "fingerprint". If you are sure you connect to the

# correct server, you can then configure the presented server

# fingerprint here. OfflineImap will verify that the server fingerprint

# has not changed on each connect and refuse to connect otherwise.

# You can also configure this in addition to CA certificate validation

# above and it will check both ways.

#cert_fingerprint = <SHA1_of_server_certificate_here>

cert_fingerprint = f3043dd689a2e7dddfbef82703a6c65ea9b634c1

# Specify the port. If not specified, use a default port.

# remoteport = 993

# Specify the remote user name.

remoteuser = dave@boostpro.com

# There are six ways to specify the password for the IMAP server:

# 1. No password at all specified in the config file.

# If a matching entry is found in ~/.netrc (see netrc (5) for

# information) this password will be used. Do note that netrc only

# allows one entry per hostname. If there is no ~/.netrc file but

# there is an /etc/netrc file, the password will instead be taken

# from there. Otherwise you will be prompted for the password when

# OfflineIMAP starts when using a UI that supports this.

# 2. The remote password stored in this file with the remotepass

# option. Any '%' needs to be encoded as '%%'. Example:

# remotepass = mypassword

# 3. The remote password stored as a single line in an external

# file, which is referenced by the remotefile option. Example:

# remotepassfile = ~/Password.IMAP.Account1

# 4. With a preauth tunnel. With this method, you invoke an external

# program that is guaranteed *NOT* to ask for a password, but rather

# to read from stdin and write to stdout an IMAP procotol stream that

# begins life in the PREAUTH state. When you use a tunnel, you do

# NOT specify a user or password (if you do, they'll be ignored.)

# Instead, you specify a preauthtunnel, as this example illustrates

# for Courier IMAP on Debian:

# preauthtunnel = ssh -q imaphost '/usr/bin/imapd ./Maildir'

# 5. If you are using Kerberos and have the Python Kerberos package

# installed, you should not specify a remotepass. If the user has a

# valid Kerberos TGT, OfflineIMAP will figure out the rest all by

# itself, and fall back to password authentication if needed.

# 6. Using arbitrary python code. With this method, you invoke a

# function from your pythonfile. To use this method assign the name

# of the function to the variable 'remotepasseval'. Example:

# remotepasseval = get_password("imap.example.net")

# You can also query for the username:

# remoteusereval = get_username("imap.example.net")

# This method can be used to design more elaborate setups, e.g. by

# querying the gnome-keyring via its python bindings.

########## Advanced settings

# Some IMAP servers need a "reference" which often refers to the "folder

# root". This is most commonly needed with UW IMAP, where you might

# need to specify the directory in which your mail is stored. The

# 'reference' value will be prefixed to all folder paths refering to

# that repository. E.g. accessing folder 'INBOX' with reference = Mail

# will try to access Mail/INBOX. Note that the nametrans and

# folderfilter functions will still apply the full path including the

# reference prefix. Most users will not need this.

# reference = Mail

# In between synchronisations, OfflineIMAP can monitor mailboxes for new

# messages using the IDLE command. If you want to enable this, specify here

# the folders you wish to monitor. Note that the IMAP protocol requires a

# separate connection for each folder monitored in this way, so setting

# this option will force settings for:

# maxconnections - to be at least the number of folders you give

# holdconnectionopen - to be true

# keepalive - to be 29 minutes unless you specify otherwise

# This feature isn't complete and may well have problems. See the manual

# for more details.

# This option should return a Python list. For example

# idlefolders = ['INBOX', 'INBOX.Alerts']

idlefolders = %(folderincludes)s

# OfflineIMAP can use multiple connections to the server in order

# to perform multiple synchronization actions simultaneously.

# This may place a higher burden on the server. In most cases,

# setting this value to 2 or 3 will speed up the sync, but in some

# cases, it may slow things down. The safe answer is 1. You should

# probably never set it to a value more than 5.

#maxconnections = 2

# OfflineIMAP normally closes IMAP server connections between refreshes if

# the global option autorefresh is specified. If you wish it to keep the

# connection open, set this to true. If not specified, the default is

# false. Keeping the connection open means a faster sync start the

# next time and may use fewer server resources on connection, but uses

# more server memory. This setting has no effect if autorefresh is not set.

#holdconnectionopen = no

# If you want to have "keepalives" sent while waiting between syncs,

# specify the amount of time IN SECONDS between keepalives here. Note that

# sometimes more than this amount of time might pass, so don't make it

# tight. This setting has no effect if autorefresh and holdconnectionopen

# are not both set.

# keepalive = 60

# Normally, OfflineIMAP will expunge deleted messages from the server.

# You can disable that if you wish. This means that OfflineIMAP will

# mark them deleted on the server, but not actually delete them.

# You must use some other IMAP client to delete them if you use this

# setting; otherwise, the messgaes will just pile up there forever.

# Therefore, this setting is definitely NOT recommended.

#expunge = no

# Specify whether to process all mail folders on the server, or only

# those listed as "subscribed".

#subscribedonly = no

# You can specify a folder translator. This must be a eval-able

# Python expression that takes a foldername arg and returns the new

# value. I suggest a lambda. This example below will remove "INBOX." from

# the leading edge of folders (great for Courier IMAP users)

# See the user documentation for details and use cases. They are also

# online at:

# http://docs.offlineimap.org/en/latest/nametrans.html

# WARNING: you MUST construct this such that it NEVER returns

# the same value for two folders, UNLESS the second values are

# filtered out by folderfilter below. Failure to follow this rule

# will result in undefined behavior

# nametrans = lambda foldername: re.sub('^INBOX\.', '', foldername)

# Using Courier remotely and want to duplicate its mailbox naming

# locally? Try this:

# nametrans = lambda foldername: re.sub('^INBOX\.*', '.', foldername)

# You can specify which folders to sync using the folderfilter

# setting. You can provide any python function (e.g. a lambda function)

# which will be invoked for each foldername. If the filter function

# returns True, the folder will be synced, if it returns False, it. The

# folderfilter operates on the *UNTRANSLATED* name (before any nametrans

# translation takes place).

# Example 1: synchronizing only INBOX and Sent.

# folderfilter = lambda foldername: foldername in ['INBOX', 'Sent']

# Example 2: synchronizing everything except Trash.

# folderfilter = lambda foldername: foldername not in ['Trash']

# Example 3: Using a regular expression to exclude Trash and all folders

# containing the characters "Del".

# folderfilter = lambda foldername: not re.search('(^Trash$|Del)', foldername)

# If folderfilter is not specified, ALL remote folders will be

# synchronized.

# You can span multiple lines by indenting the others. (Use backslashes

# at the end when required by Python syntax) For instance:

# folderfilter = lambda foldername: foldername in

# ['INBOX', 'Sent Mail', 'Deleted Items',

# 'Received']

folderfilter = lambda foldername: False

# You can specify folderincludes to include additional folders. It

# should return a Python list. This might be used to include a folder

# that was excluded by your folderfilter rule, to include a folder that

# your server does not specify with its LIST option, or to include a

# folder that is outside your basic reference. The 'reference' value

# will not be prefixed to this folder name, even if you have specified

# one. For example:

# folderincludes = ['debian.user', 'debian.personal']

folderincludes = ['INBOX', '[Gmail]/All Mail']

# If you do not want to have any folders created on this repository,

# set the createfolders variable to False, the default is True. Using

# this feature you can e.g. disable the propagation of new folders to

# the new repository.

#createfolders = True

# You can specify 'foldersort' to determine how folders are sorted.

# This affects order of synchronization and mbnames. The expression

# should return -1, 0, or 1, as the default Python cmp() does. The two

# arguments, x and y, are strings representing the names of the folders

# to be sorted. The sorting is applied *AFTER* nametrans, if any. The

# default is to sort IMAP folders alphabetically

# (case-insensitive). Usually, you should never have to modify this. To

# eg. reverse the sort:

# foldersort = lambda x, y: -cmp(x, y)

# Enable 1-way synchronization. When setting 'readonly' to True, this

# repository will not be modified during synchronization. Use to

# e.g. backup an IMAP server. The readonly setting can be applied to any

# type of Repository (Maildir, Imap, etc).

#readonly = False

# [Repository GmailExample]

# A repository using Gmail's IMAP interface. Any configuration

# parameter of `IMAP` type repositories can be used here. Only

# `remoteuser` (or `remoteusereval` ) is mandatory. Default values

# for other parameters are OK, and you should not need fiddle with

# those.

# The Gmail repository will use hard-coded values for `remotehost`,

# `remoteport`, `tunnel` and `ssl`. (See

# http://mail.google.com/support/bin/answer.py?answer=78799&topic=12814)

# Any attempt to set those parameters will be silently ignored.

# type = Gmail

# Specify the Gmail user name. This is the only mandatory parameter.

# remoteuser = username@gmail.com

# The trash folder name may be different from [Gmail]/Trash

# for example on german googlemail, this setting should be

# trashfolder = [Google Mail]/Papierkorb

# The same is valid for the spam folder

# spamfolder = [Google Mail]/Spam

# Enable 1-way synchronization. See above for explanation.

#readonly = False

Setup Fail2ban for NextCloud

2018/10/05 by sudo

Running NextCloud or OwnCloud online comes with some risk, as with any online service. It’s important that your installation remains secure against hackers (or at least as secure as it can be). I’ve opted to implement fail2ban in order to help secure it using some custom rules. It’s worth noting that NextCloud does block unwanted login attempts itself through the application, but you’re having to trust application level security. I feel far safer having fail2ban implement firewall rules to prevent access to anyone probing the server.

First thing to do is create the NextCloud filter configuration file. This file will contain the regex that’s used to scan the logs for anything we don’t like the look of in order to block attacking hosts. My understanding is that this file can remain the same for OwnCloud, although I do not currently have a running instance of it to check.

sudo nano /etc/fail2ban/filter.d/nextcloud.conf

1	sudo nano /etc/fail2ban/filter.d/nextcloud.conf

Add the following to the file:

[Definition]
failregex=^{"reqId":".<em>","remoteAddr":".</em>","app":"core","message":"Login failed: '.<em>' &#40;Remote IP: '<HOST>'&#41;","level":2,"time":".</em>"}$
        ^{"reqId":".<em>","level":2,"time":".</em>","remoteAddr":".<em>","app":"core".</em>","message":"Login failed: '.<em>' &#40;Remote IP: '<HOST>'&#41;".</em>}$
        ^.<em>\"remoteAddr\":\"<HOST>\".</em>Trusted domain error.*$

[Definition]

failregex=^{"reqId":".","remoteAddr":".","app":"core","message":"Login failed: '.' (Remote IP: '<HOST>')","level":2,"time":"."}$

^{"reqId":".","level":2,"time":".","remoteAddr":".","app":"core".","message":"Login failed: '.' (Remote IP: '<HOST>')".}$

^.\"remoteAddr\":\"<HOST>\".Trusted domain error.*$

There are three regular expressions included here. The first and second checks for login failures, and flags the source IP. The third checks for trusted domain errors – which are usually a result of bots accessing your installation via it’s IP, not via it’s domain (thus, suspicious and I wanted to block them).

Once the file is saved, you can test what the filter would report by running the following command. This is entirely optional (although would help identify issues) and isn’t required for the rest of the steps to work.

sudo fail2ban-regex /var/nextcloud/data/nextcloud.log /etc/fail2ban/filter.d/nextcloud.conf -v

1	sudo fail2ban-regex /var/nextcloud/data/nextcloud.log /etc/fail2ban/filter.d/nextcloud.conf -v

Next, the configuration file needs setup to activate the configurations we’ve just created. Never edit the Fail2ban jail.conf file, it’s likely to be overridden on upgrades. Always create a “.local” file, ideally a separate one for each application or rule you’re setting up (why? because it makes things more organised and easier to manage one rule over another!) inside the jail.d directory. With this in mind, create a nextcloud (or owncloud) file:

sudo nano /etc/fail2ban/jail.d/nextcloud.local

1	sudo nano /etc/fail2ban/jail.d/nextcloud.local

And add the following to it:

[nextcloud]
ignoreip = 192.168.1.0/24
backend = auto
enabled = true
port = 80,443
protocol = tcp
filter = nextcloud
maxretry = 3
bantime = 36000
findtime = 36000
logpath = /var/nextcloud/data/nextcloud.log

[nextcloud]

ignoreip = 192.168.1.0/24

backend = auto

enabled = true

port = 80,443

protocol = tcp

filter = nextcloud

maxretry = 3

bantime = 36000

findtime = 36000

logpath = /var/nextcloud/data/nextcloud.log

Make sure your ignoreip is your local subnet or IP address. I opted to allow my whole LAN to access it without being auto-blocked. I’ve enabled the rule, set the ports to 80 (HTTP) and 443 (HTTPS) and configured ban times, etc. The most important things are the filter which should match the name of the file that was created inside the filter.d directory (excluding extension), and the log path, which may vary by installation. This path is the default for Ubuntu.

Once done, run the following command to restart nextcloud:

sudo service fail2ban restart

1	sudo service fail2ban restart

You can check the status of the jail by running:

sudo fail2ban-client status nextcloud

1	sudo fail2ban-client status nextcloud

You’ll see something similar to this:

Status for the jail: nextcloud
|- Filter
|  |- Currently failed: 13
|  |- Total failed: 82
|  `- File list:    /var/nextcloud/data/nextcloud.log
`- Actions
   |- Currently banned: 0
   |- Total banned: 5
   `- Banned IP list:

Status for the jail: nextcloud

|- Filter

| |- Currently failed: 13

| |- Total failed: 82

| `- File list: /var/nextcloud/data/nextcloud.log

`- Actions

|- Currently banned: 0

|- Total banned: 5

`- Banned IP list:

Setting up a KVM host on Ubuntu 18.04 with bridged networking

2018/09/16 by sudo

Since the launch of Ubuntu server 18.04, I’ve had a few people ask me how to setup bridged networking since the changeover to netplan for networking. The setup’s actually quite straight forwards once you know how. Start out by installing KVM and associated packages (I’m assuming you’re on a fresh Ubuntu 18.04 box):

sudo apt install bridge-utils qemu-kvm libvirt-bin

1	sudo apt install bridge-utils qemu-kvm libvirt-bin

You should now be able to access virsh, to test it just run the command to list the virtual machines on the host (this will of course output an empty list. We just want to make sure the command hasn’t failed before going further):

virsh list --all

1	virsh list --all

<h2> Id    Name                           State</h2>

1	<h2> Id Name State</h2>

Now edit the netplan file for the networking configuration of the host system:

sudo nano /etc/netplan/50-cloud-init.yaml

1	sudo nano /etc/netplan/50-cloud-init.yaml

The configuration should be edited to look similar to this:

network:
    ethernets:
        eno1:
            addresses: []
            dhcp4: true
            optional: true
    version: 2
    bridges:
        br0:
            interfaces: [eno1]
            dhcp4: true
            parameters:
                stp: false
                forward-delay: 0

network:

ethernets:

eno1:

addresses: []

dhcp4: true

optional: true

version: 2

bridges:

br0:

interfaces: [eno1]

dhcp4: true

parameters:

stp: false

forward-delay: 0

We’re basically defining a single ethernet port (eno1) and assigning it to a bride. Note that you can check your ethernet port name using the ‘ip address’ command on the command line and it will return your network adapters and IP addresses. It should be resonably obvious which one is your ethernet adapter from the list!

It’s worth noting that in this setup, we’re allowing DHCP to allocate addresses to the machine. If this isn’t what you want, take a look at the netplan examples here: https://netplan.io/examples

You can apply these change using:

sudo netplan apply

1	sudo netplan apply

And check it’s working:

ip address

1	ip address

networkctl list

1	networkctl list

If you’ve gotten this far without a network issue, you should be able to create machines using your favorite tool. I use Virtual Machine Manager (https://virt-manager.org/) which provides a reasonable user interface as well as remote management via SSH.

Setting up a Bond and Bridge in Netplan on Ubuntu 18.04

2018/09/12 by sudo

For some of my Ubuntu 18.04 servers, I need to run KVM virtual machines which require a bridge to the network so the machines get public LAN IP addresses and aren’t hidden behind NAT. With the server configuration for both my co-location and servers at work the network interfaces are all bonded for fail-over. This means I need a bond to have a bridge ontop of it for the virtual machines to get public IP addresses, while still allowing for failover of the network connection in the event of a network failure.

Research

There are some good examples of setting up netplan here: https://netplan.io/examples

They have a bridge example:

network:
  version: 2
  renderer: networkd
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - enp3s0

network:

version: 2

renderer: networkd

bridges:

br0:

dhcp4: yes

interfaces:

- enp3s0

And a bond example:

network:
  version: 2
  renderer: networkd
  bonds:
    bond0:
      dhcp4: yes
      interfaces:
        - enp3s0
        - enp4s0
      parameters:
        mode: active-backup
        primary: enp3s0

network:

version: 2

renderer: networkd

bonds:

bond0:

dhcp4: yes

interfaces:

- enp3s0

- enp4s0

parameters:

mode: active-backup

primary: enp3s0

But there’s not a clear indication of how to amalgamate the two.

A bond and a bridge

Here’s what I’ve ended up with in ‘/etc/netplan/50-cloud-init.yaml’:

network:
    bridges:
        br0:
            addresses:
            - 192.168.10.30
            dhcp4: false
            gateway4: 192.168.10.1
            nameservers:
                addresses:
                - 192.168.10.1
                - 192.168.10.2
                search: []
            interfaces:
                - bond0
    bonds:
        bond0:
            interfaces:
            - eno1
            - eno2
            parameters:
                mode: active-backup
    ethernets:
        eno1:
            addresses: []
            dhcp4: false
            dhcp6: false
        eno2:
            addresses: []
            dhcp4: false
            dhcp6: false

network:

bridges:

br0:

addresses:

- 192.168.10.30

dhcp4: false

gateway4: 192.168.10.1

nameservers:

addresses:

- 192.168.10.1

- 192.168.10.2

search: []

interfaces:

- bond0

bonds:

bond0:

interfaces:

- eno1

- eno2

parameters:

mode: active-backup

ethernets:

eno1:

addresses: []

dhcp4: false

dhcp6: false

eno2:

addresses: []

dhcp4: false

dhcp6: false

Note that I’ve obviously defined static IP addresses, but this isn’t a requirement. Just set ‘dhcp4: true’ and remove the ‘address’, ‘gateway’ and ‘nameserver’ sections if you’re using DHCP.

Once the file’s got that setup in it, it’s possible to run:

sudo netplan apply

1	sudo netplan apply

and you should be able to run ‘networkctl list’ to check the bridge and bond are setup.

Fail2ban does not enable some rules

2018/08/12 by sudo

While setting up a webserver, I noticed that

fail2ban-client status

1	fail2ban-client status

did not list all of the rules that I’d configured in

jail.d

jail.d

. Althought configuration files were added to the directory, the status wasn’t showing them listed, so it hadn’t activated the jails. Running

service fail2ban restart

1	service fail2ban restart

didn’t add them either.

After spending some time researching, some people suggest changing the backend used in fail2ban from auto to polling in jail.conf. Since you shouldn’t edit jail.conf on a debian based system, I tried creating a new file

nano /etc/fail2ban/jail.local

1	nano /etc/fail2ban/jail.local

and adding the line

backend = polling

1	backend = polling

. This actually prevented the fail2ban service from starting for me. The errors in

systemctl status fail2ban.service

1	systemctl status fail2ban.service

didn’t show anything useful as to the cause of the error. In the end I reverted this change.

running

fail2ban-client reload

1	fail2ban-client reload

did present error messages detailing which rule couldn’t be activated:

ERROR No file(s) found for glob /var/log/apache2/*.log
ERROR Failed during configuration: Have not found any log file for apache-xmlrpc jail

1 2	ERROR No file(s) found for glob /var/log/apache2/*.log ERROR Failed during configuration: Have not found any log file for apache-xmlrpc jail

This then meant I was able to investigate the

jail.d

jail.d

config file with the rule indicated as causing the problem and correct the log file path.

fail2ban-client status

1	fail2ban-client status

then shows the correct jails running.

Share this:

Share this:

Share this:

Research

A bond and a bridge

Share this:

Share this: