apt get life

Life around technology

  • Technology
    • Guides
    • Linux
    • Development
      • Laravel
    • Misc
    • Raspberry Pi
  • Writing
  • Crafts
    • Crochet
    • Model Making
    • Painting
  • Privacy Policy
You are here: Home / Technology / Linux / Solving Error on statfs() system call with cAdvisor and Docker

Solving Error on statfs() system call with cAdvisor and Docker

2021/01/14 by sudo Leave a Comment

With a docker setup running with Prometheus node exporter and a docker container running cAdvisor, I’ve been seeing error messages similar to the following repeatedly appearing in syslog:

Jan 14 15:15:25 dockerserver-live prometheus-node-exporter[603]: time="2021-01-14T15:15:25Z" level=error msg="Error on statfs() system call for \"/var/lib/docker/containers/719fe4c20d2d274bb034e914006ecfe6760d8aec98efdc8010c85a01cf4059aa/mounts/shm\": permission denied" source="filesystem_linux.go:57"
Jan 14 16:23:25 dockerserver-live prometheus-node-exporter[28623]: time="2021-01-14T16:23:25Z" level=error msg="Error on statfs() system call for \"/var/lib/docker/overlay2/8c25cb3049b4cfc9bebfd4df0ea6104560155bed2c18a9bd75d21323931570f4/merged\": permission denied" source="filesystem_linux.go:57"

These errors are being generated by a Prometheus node exporter process running with the following args:

      --collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$ \
      --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \
      --collector.netdev.ignored-devices=^lo$ \
      --collector.textfile.directory=/var/lib/prometheus/node-exporter

I’m not sure where this block originally came from, but in our case it’s in /etc/defaults/prometheus-node-exporter and easily edited to fix the regular expressions. Specifcally, because there’s problems with permissions on the shm and overlay they can be added to the ignored-mount-point regular expression as below:

      --collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$ \
      --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run|var\/lib\/docker\/containers\/.*\/mounts\/shm|var\/lib\/docker\/overlay2\/.*\/merged)($|/) \
      --collector.netdev.ignored-devices=^lo$ \
      --collector.textfile.directory=/var/lib/prometheus/node-exporter

following this change and restarting the prometheus-node-exporter process stopped new entries appearing in our syslog as it should be ignoring both troublesome directories.

Share this:

  • Twitter
  • Facebook
  • Reddit
  • Tumblr
  • Pinterest

Related

Filed Under: Docker, Linux, Technology, Uncategorized

Leave a Reply Cancel reply

Recent Posts

  • Solving Error on statfs() system call with cAdvisor and Docker
  • MySQL in Docker for Local Development
  • Laravel Docker Development Environment With PHP7
  • Docker WordPress Increase PHP Max File Size
  • Injecting inbound request headers with Traefik v2

Tags

7zip API auditing Courier MTA crochet data recovery debian debudding development Dingo API docker email Getting started with Laravel 5 & Dingo API hard drive health KVM Laravel larvel 5 lenovo Linux Mail Quota Minion mint netgear nas networking network shares php PHP development Postfix samba security SMART smartctl smartmontools smb smbfs testing traefik ubuntu ubuntu 18.04 ubuntu 20.04 ubuntu server vagrant Virtual machines xdebug xubuntu

© Copyright 2015 apt get life