Skip to content

Category: Linux

Mattermost and Linux system logging or:

How I learned to stop worrying about eventvwr and love journalctl.

I started writing this article to share how to fix a simple issue with Mattermost plugins. It happily morphed into a great opportunity for me to learn and write about Linux system logging. In my previous life on Windows, you would encounter an application error, scratch your head and then run eventvwr. (Oh, the many happy hours I spent in Event Viewer trying to decipher ADFS logs!). This article is intended to give you an introduction to the logging features in Mattermost and Linux. It is not an exhaustive history or guide to Linux system logging far from it; but practical guidance on how to quickly gain useful insights and troubleshoot a problem.

By sheer luck this error is a logging error, so ‘meta’.

If you are coming from the Windows world then you need to go cold turkey on your Event Viewer addiction. I’ll use a simple application error within Mattermost as my example, so…. come with me and lets find out if the mighty Event Viewer has a command line evil(?) twin in Linux….

The problem..

Plugins are a great way to extend the capabilities of Mattermost. They can be complex integrations which improve your DevOps workflow like the Jira and Jenkins plugins, or a simple productivity hack like Matterpoll or Remind. The Mattermost server ships with the Plugin Marketplace which can be used to install a collection of Mattermost developed plugins. Also, there is the https://integrations.mattermost.com website with all the Open Source community plugins.

I had downloaded the remind plugin from the Mattermost integrations website and installed it on my Mattermost HA Cluster. After some time, I had tried to Enable the plugin and found that it was reporting an error in the system console.

Screenshot of error, this plugin failed to start.
Oops

What’s going on..?

Mattermost Application Logs

I needed to check the logs to see what was going on under the hood. The Mattermost server logs can be configured to write to both a file and the console, with varying levels of detail (ERROR, INFO or DEBUG). I usually have my logs set to DEBUG in my lab and navigated to the System Console | Server Logs to find a more detailed error:

{"level":"error","ts":1565913590.4749634,"caller":"mlog/log.go:172","msg":"Unable to activate plugin","plugin_id":"com.github.scottleedavis.mattermost-plugin-remind","error":"unable to start plugin: com.github.scottleedavis.mattermost-plugin-remind: fork/exec plugins/com.github.scottleedavis.mattermost-plugin-remind/server/dist/plugin-linux-amd64: permission denied",

Instead of going through the UI to review the Server Logs I could also go to the command line and search the mattermost.log, usually stored in /opt/mattermost/logs. In this example; I know the name of the plugin so can use that as a search term, or search for the error term.

grep "remind" mattermost.log
grep "error" mattermost.log

Linux system logging

Another method to find the plugin error information is to use the tools in the operating system. Many Linux distributions have moved to using systemd which provides a suite of tools to manage your server. The systemd-journald.service collects and manages logs from kernel and user processes in a journal which can be accessed using journalctl.

In this example, similar to tail, you can follow the latest information in the log, or specify the number of the most recent events you want to see:

journalctl -f
journalctl -n 20

You can filter based on date and time; so if you know when the failure occurred do this:

journalctl --since today
journalctl --since "YYYY-MM-DD HH:MM:SS"
journalctl --since "2020-02-01 18:00:00"
journalctl --since "2020-02-05 18:00:00" --until 18:01

If I want to find the same information as the Mattermost server logs in journalctl I can filter based upon the mattermost.service unit:

journalctl -u mattermost.service

You can limit the output to just show the message field:

journalctl -o cat

Bringing it all together. If I want to find the last 5 logged messages just for Mattermost:

journalctl -u mattermost.service -n 5 -o cat

You can see the output from the journalctl command is the start of the same error message that is posted in the server log earlier in the article.

Love a bit of putty

The fix is in..

Getting back to the plugin problem in hand. The key bit in the logs was the permission denied statement. Because I am a GUI lovin’ Windows Admin I use WinSCP to manage my transferring of files from Windows desktop to Linux. It also can be used for permissions management on the Linux server. I logged on and could see that something wasn’t right:

Screenshot of Plugins Folder Permissions
Screenshot of incorrectly set permissions in the contents of the individual plugin folder

The parent mattermost/plugins folder had mattermost as the Owner. However, when you went into the actual plugin folder itself, root was still the owner on the contents.

I needed to do a simple reset of the ownership of the files and folders within the specific plugin directory. I highlighted the contents of the plugin folder in WinSCP, and set Group and Owner to mattermost.

Alternatively, you could also fix this by using the following Linux command. Again, ensure you navigate into the folder of the plugin with the issue. (Don’t miss off the ‘period’ at the end!)

chown -R mattermost:mattermost .
Screenshot of the correct permissions

This means mattermost is the assigned Group and Owner of the plugin folder and files and now the plugin should start.

Screenshot of message that 'this plugin is running'
Happy Days!

Conclusion

Hopefully that has given you a starting point into how to troubleshoot issues using the Mattermost server logs and journalctl. It also means Event Viewer is a thing of the past for me!

A big thank you to Mark P, who’s encyclopedic knowledge of Linux (and Windows) and obscure points on grammar were invaluable in my learning and the writing of the article.

Reference:
Linux logging concepts and tutorial – stdin, stdout, stderr:
https://www.howtogeek.com/435903/what-are-stdin-stdout-and-stderr-on-linux/
How to use journalctl:
https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs
A useful blog article I used to setup my WinSCP:
https://blog.jamesbayley.com/2014/01/28/how-to-connect-as-root-using-winscp/