Skip to content

Category: Mattermost

Mattermost and Linux system logging or:

How I learned to stop worrying about eventvwr and love journalctl.

I started writing this article to share how to fix a simple issue with Mattermost plugins. It happily morphed into a great opportunity for me to learn and write about Linux system logging. In my previous life on Windows, you would encounter an application error, scratch your head and then run eventvwr. (Oh, the many happy hours I spent in Event Viewer trying to decipher ADFS logs!). This article is intended to give you an introduction to the logging features in Mattermost and Linux. It is not an exhaustive history or guide to Linux system logging far from it; but practical guidance on how to quickly gain useful insights and troubleshoot a problem.

By sheer luck this error is a logging error, so ‘meta’.

If you are coming from the Windows world then you need to go cold turkey on your Event Viewer addiction. I’ll use a simple application error within Mattermost as my example, so…. come with me and lets find out if the mighty Event Viewer has a command line evil(?) twin in Linux….

The problem..

Plugins are a great way to extend the capabilities of Mattermost. They can be complex integrations which improve your DevOps workflow like the Jira and Jenkins plugins, or a simple productivity hack like Matterpoll or Remind. The Mattermost server ships with the Plugin Marketplace which can be used to install a collection of Mattermost developed plugins. Also, there is the website with all the Open Source community plugins.

I had downloaded the remind plugin from the Mattermost integrations website and installed it on my Mattermost HA Cluster. After some time, I had tried to Enable the plugin and found that it was reporting an error in the system console.

Screenshot of error, this plugin failed to start.

What’s going on..?

Mattermost Application Logs

I needed to check the logs to see what was going on under the hood. The Mattermost server logs can be configured to write to both a file and the console, with varying levels of detail (ERROR, INFO or DEBUG). I usually have my logs set to DEBUG in my lab and navigated to the System Console | Server Logs to find a more detailed error:

{"level":"error","ts":1565913590.4749634,"caller":"mlog/log.go:172","msg":"Unable to activate plugin","plugin_id":"com.github.scottleedavis.mattermost-plugin-remind","error":"unable to start plugin: com.github.scottleedavis.mattermost-plugin-remind: fork/exec plugins/com.github.scottleedavis.mattermost-plugin-remind/server/dist/plugin-linux-amd64: permission denied",

Instead of going through the UI to review the Server Logs I could also go to the command line and search the mattermost.log, usually stored in /opt/mattermost/logs. In this example; I know the name of the plugin so can use that as a search term, or search for the error term.

grep "remind" mattermost.log
grep "error" mattermost.log

Linux system logging

Another method to find the plugin error information is to use the tools in the operating system. Many Linux distributions have moved to using systemd which provides a suite of tools to manage your server. The systemd-journald.service collects and manages logs from kernel and user processes in a journal which can be accessed using journalctl.

In this example, similar to tail, you can follow the latest information in the log, or specify the number of the most recent events you want to see:

journalctl -f
journalctl -n 20

You can filter based on date and time; so if you know when the failure occurred do this:

journalctl --since today
journalctl --since "YYYY-MM-DD HH:MM:SS"
journalctl --since "2020-02-01 18:00:00"
journalctl --since "2020-02-05 18:00:00" --until 18:01

If I want to find the same information as the Mattermost server logs in journalctl I can filter based upon the mattermost.service unit:

journalctl -u mattermost.service

You can limit the output to just show the message field:

journalctl -o cat

Bringing it all together. If I want to find the last 5 logged messages just for Mattermost:

journalctl -u mattermost.service -n 5 -o cat

You can see the output from the journalctl command is the start of the same error message that is posted in the server log earlier in the article.

Love a bit of putty

The fix is in..

Getting back to the plugin problem in hand. The key bit in the logs was the permission denied statement. Because I am a GUI lovin’ Windows Admin I use WinSCP to manage my transferring of files from Windows desktop to Linux. It also can be used for permissions management on the Linux server. I logged on and could see that something wasn’t right:

Screenshot of Plugins Folder Permissions
Screenshot of incorrectly set permissions in the contents of the individual plugin folder

The parent mattermost/plugins folder had mattermost as the Owner. However, when you went into the actual plugin folder itself, root was still the owner on the contents.

I needed to do a simple reset of the ownership of the files and folders within the specific plugin directory. I highlighted the contents of the plugin folder in WinSCP, and set Group and Owner to mattermost.

Alternatively, you could also fix this by using the following Linux command. Again, ensure you navigate into the folder of the plugin with the issue. (Don’t miss off the ‘period’ at the end!)

chown -R mattermost:mattermost .
Screenshot of the correct permissions

This means mattermost is the assigned Group and Owner of the plugin folder and files and now the plugin should start.

Screenshot of message that 'this plugin is running'
Happy Days!


Hopefully that has given you a starting point into how to troubleshoot issues using the Mattermost server logs and journalctl. It also means Event Viewer is a thing of the past for me!

A big thank you to Mark P, who’s encyclopedic knowledge of Linux (and Windows) and obscure points on grammar were invaluable in my learning and the writing of the article.

Linux logging concepts and tutorial – stdin, stdout, stderr:
How to use journalctl:
A useful blog article I used to setup my WinSCP:

Mattermost Office 365 SSO and TLS

My goal was to test out the Office 365 SSO authentication to Mattermost. The limitation is that the Office 365 SSO (Azure AD) requires the OAuth Redirect URI to match whatever the SiteURL is configured on the Mattermost server. 

My Mattermost HA Cluster lab environment was setup with:
Mattermost SiteURL:
NGINX TLS config:

So in my example this will fail as Azure AD expects as the redirect URI endpoint whereas I only have the HTTPS endpoint published by NGINX to the internet and visible to Office 365/Azure AD.

The Mattermost service is secured with a certificate and domain name I do not own so I wasn’t able to change certs or DNS myself. So I could….
1) Configure the NGINX server as a load balancer to passthrough TLS to the Mattermost app server cluster
2) Remove the NGINX proxy and configure a load balancer in AWS to passthrough TLS to the Mattermost app server cluster

NGINX must be a version that has the Stream module to enable passthrough. First thing is to check the stream module is installed:

nginx -V 2>&1 | tr -- - '\n' | grep module

Once I knew NGINX could support passthrough I focused on getting the SSL certificate onto the Mattermost server. I was able to just copy the same certificate and private key file currently used by the NGINX reverse proxy to the Mattermost server. Making sure I changed the owner and permissions:

sudo chown -R mattermost:mattermost /opt/mattermost/config/fullchain.pem
sudo chown -R mattermost:mattermost /opt/mattermost/config/privekey.pem
sudo chmod 400 *.pem

I then followed the TLS configuration steps described in our docs. Leaving me with ServiceSettings similar to below:

"ServiceSettings": {
        "SiteURL": "",
        "WebsocketURL": "",
        "LicenseFileLocation": "",
        "ListenAddress": ":443",
        "ConnectionSecurity": "TLS",
        "TLSCertFile": "/opt/mattermost/config/fullchain.pem",
        "TLSKeyFile": "/opt/mattermost/config/privkey.pem",

NGINX needs to be told to look for a passthrough configuration. So edit the nginx.conf and at the end of the http configuration block add the include statement.

sudo nano /etc/nginx/nginx.conf 
http {
include /etc/nginx/passthrough.conf;

We now have to create the passthrough configuration file you have just referenced.

sudo nano /etc/nginx/passthrough.conf

The NGINX docs are really good here and I picked out a couple of key points. There are many settings you could use to determine the load balancing settings but I wanted to keep my configuration simple in my lab.

Load Balancing Method and Session Persistence
Firstly the Open Source version of the product only allows the Session persistence methods hash or ip_hash directive. I tried ip_hash first but ended up getting the following error:
nginx: [emerg] “ip_hash” directive is not allowed here in /etc/nginx/passthrough.conf:4
As far as the notes go, it should work but I didn’t have the patience to find the answer.
I selected for my configuration:

hash – The server to which a request is sent is determined from a user‑defined key
$remote_addr – client address
consistent – Requests are evenly distributed across all upstream servers

# LB https to 2 backend servers
stream {
    upstream mm_mydomain_com {
      hash $remote_addr consistent;

    server {
        listen 443;
        proxy_pass mm_mydomain_com;
        proxy_next_upstream error timeout;

Once I had a passthrough.conf that I thought would work, I removed the current NGINX https reverse proxy configuration.

cd /etc/nginx/sites-enabled
ls -l
sudo rm mattermost

I then ran a test against the NGINX configuration sudo nginx -t

The final test was to take my two Mattermost app servers and reboot testing connections to both, which finally worked!

This was the best way I had to quickly get a working configuration in place to test Office 365 SSO with Mattermost and give me a chance to learn a bit about NGINX. There are likely better ways I could have completed this but this works well for me for testing Mattermost Office 365 sign on in my lab.

If I was looking at this for a production instance then I would investigate the ip_hash configuration, health_check and proxy_next_upstream settings in more detail.

Thank you to those who have shared before and used as a basis for this: