Skip to content

John Thompson Posts

Featured Post

Mattermost and Linux system logging or:

How I learned to stop worrying about eventvwr and love journalctl.

I started writing this article to share how to fix a simple issue with Mattermost plugins. It happily morphed into a great opportunity for me to learn and write about Linux system logging. In my previous life on Windows, you would encounter an application error, scratch your head and then run eventvwr. (Oh, the many happy hours I spent in Event Viewer trying to decipher ADFS logs!). This article is intended to give you an introduction to the logging features in Mattermost and Linux. It is not an exhaustive history or guide to Linux system logging far from it; but practical guidance on how to quickly gain useful insights and troubleshoot a problem.

By sheer luck this error is a logging error, so ‘meta’.

If you are coming from the Windows world then you need to go cold turkey on your Event Viewer addiction. I’ll use a simple application error within Mattermost as my example, so…. come with me and lets find out if the mighty Event Viewer has a command line evil(?) twin in Linux….

The problem..

Plugins are a great way to extend the capabilities of Mattermost. They can be complex integrations which improve your DevOps workflow like the Jira and Jenkins plugins, or a simple productivity hack like Matterpoll or Remind. The Mattermost server ships with the Plugin Marketplace which can be used to install a collection of Mattermost developed plugins. Also, there is the https://integrations.mattermost.com website with all the Open Source community plugins.

I had downloaded the remind plugin from the Mattermost integrations website and installed it on my Mattermost HA Cluster. After some time, I had tried to Enable the plugin and found that it was reporting an error in the system console.

Screenshot of error, this plugin failed to start.
Oops

What’s going on..?

Mattermost Application Logs

I needed to check the logs to see what was going on under the hood. The Mattermost server logs can be configured to write to both a file and the console, with varying levels of detail (ERROR, INFO or DEBUG). I usually have my logs set to DEBUG in my lab and navigated to the System Console | Server Logs to find a more detailed error:

{"level":"error","ts":1565913590.4749634,"caller":"mlog/log.go:172","msg":"Unable to activate plugin","plugin_id":"com.github.scottleedavis.mattermost-plugin-remind","error":"unable to start plugin: com.github.scottleedavis.mattermost-plugin-remind: fork/exec plugins/com.github.scottleedavis.mattermost-plugin-remind/server/dist/plugin-linux-amd64: permission denied",

Instead of going through the UI to review the Server Logs I could also go to the command line and search the mattermost.log, usually stored in /opt/mattermost/logs. In this example; I know the name of the plugin so can use that as a search term, or search for the error term.

grep "remind" mattermost.log
grep "error" mattermost.log

Linux system logging

Another method to find the plugin error information is to use the tools in the operating system. Many Linux distributions have moved to using systemd which provides a suite of tools to manage your server. The systemd-journald.service collects and manages logs from kernel and user processes in a journal which can be accessed using journalctl.

In this example, similar to tail, you can follow the latest information in the log, or specify the number of the most recent events you want to see:

journalctl -f
journalctl -n 20

You can filter based on date and time; so if you know when the failure occurred do this:

journalctl --since today
journalctl --since "YYYY-MM-DD HH:MM:SS"
journalctl --since "2020-02-01 18:00:00"
journalctl --since "2020-02-05 18:00:00" --until 18:01

If I want to find the same information as the Mattermost server logs in journalctl I can filter based upon the mattermost.service unit:

journalctl -u mattermost.service

You can limit the output to just show the message field:

journalctl -o cat

Bringing it all together. If I want to find the last 5 logged messages just for Mattermost:

journalctl -u mattermost.service -n 5 -o cat

You can see the output from the journalctl command is the start of the same error message that is posted in the server log earlier in the article.

Love a bit of putty

The fix is in..

Getting back to the plugin problem in hand. The key bit in the logs was the permission denied statement. Because I am a GUI lovin’ Windows Admin I use WinSCP to manage my transferring of files from Windows desktop to Linux. It also can be used for permissions management on the Linux server. I logged on and could see that something wasn’t right:

Screenshot of Plugins Folder Permissions
Screenshot of incorrectly set permissions in the contents of the individual plugin folder

The parent mattermost/plugins folder had mattermost as the Owner. However, when you went into the actual plugin folder itself, root was still the owner on the contents.

I needed to do a simple reset of the ownership of the files and folders within the specific plugin directory. I highlighted the contents of the plugin folder in WinSCP, and set Group and Owner to mattermost.

Alternatively, you could also fix this by using the following Linux command. Again, ensure you navigate into the folder of the plugin with the issue. (Don’t miss off the ‘period’ at the end!)

chown -R mattermost:mattermost .
Screenshot of the correct permissions

This means mattermost is the assigned Group and Owner of the plugin folder and files and now the plugin should start.

Screenshot of message that 'this plugin is running'
Happy Days!

Conclusion

Hopefully that has given you a starting point into how to troubleshoot issues using the Mattermost server logs and journalctl. It also means Event Viewer is a thing of the past for me!

A big thank you to Mark P, who’s encyclopedic knowledge of Linux (and Windows) and obscure points on grammar were invaluable in my learning and the writing of the article.

Reference:
Linux logging concepts and tutorial – stdin, stdout, stderr:
https://www.howtogeek.com/435903/what-are-stdin-stdout-and-stderr-on-linux/
How to use journalctl:
https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs
A useful blog article I used to setup my WinSCP:
https://blog.jamesbayley.com/2014/01/28/how-to-connect-as-root-using-winscp/

Mattermost Office 365 SSO and TLS

PROBLEM:
My goal was to test out the Office 365 SSO authentication to Mattermost. The limitation is that the Office 365 SSO (Azure AD) requires the OAuth Redirect URI to match whatever the SiteURL is configured on the Mattermost server. 

My Mattermost HA Cluster lab environment was setup with:
Mattermost SiteURL: http://mm.mydomain.com:8065
NGINX TLS config:        https://mm.mydomain.com

So in my example this will fail as Azure AD expects http://mm.yourserver.com:8065 as the redirect URI endpoint whereas I only have the HTTPS endpoint published by NGINX to the internet and visible to Office 365/Azure AD.

OPTIONS
The Mattermost service is secured with a certificate and domain name I do not own so I wasn’t able to change certs or DNS myself. So I could….
1) Configure the NGINX server as a load balancer to passthrough TLS to the Mattermost app server cluster
2) Remove the NGINX proxy and configure a load balancer in AWS to passthrough TLS to the Mattermost app server cluster

SOLUTION
NGINX must be a version that has the Stream module to enable passthrough. First thing is to check the stream module is installed:

nginx -V 2>&1 | tr -- - '\n' | grep module
modules
path=/usr/lib/nginx/modules
stream_ssl_module

Once I knew NGINX could support passthrough I focused on getting the SSL certificate onto the Mattermost server. I was able to just copy the same certificate and private key file currently used by the NGINX reverse proxy to the Mattermost server. Making sure I changed the owner and permissions:

sudo chown -R mattermost:mattermost /opt/mattermost/config/fullchain.pem
sudo chown -R mattermost:mattermost /opt/mattermost/config/privekey.pem
sudo chmod 400 *.pem

I then followed the TLS configuration steps described in our docs. Leaving me with ServiceSettings similar to below:

"ServiceSettings": {
        "SiteURL": "https://mm.domain.com",
        "WebsocketURL": "",
        "LicenseFileLocation": "",
        "ListenAddress": ":443",
        "ConnectionSecurity": "TLS",
        "TLSCertFile": "/opt/mattermost/config/fullchain.pem",
        "TLSKeyFile": "/opt/mattermost/config/privkey.pem",

NGINX needs to be told to look for a passthrough configuration. So edit the nginx.conf and at the end of the http configuration block add the include statement.

sudo nano /etc/nginx/nginx.conf 
 
http {
|
|
}
include /etc/nginx/passthrough.conf;

We now have to create the passthrough configuration file you have just referenced.

sudo nano /etc/nginx/passthrough.conf

The NGINX docs are really good here and I picked out a couple of key points. There are many settings you could use to determine the load balancing settings but I wanted to keep my configuration simple in my lab.

Load Balancing Method and Session Persistence
Firstly the Open Source version of the product only allows the Session persistence methods hash or ip_hash directive. I tried ip_hash first but ended up getting the following error:
nginx: [emerg] “ip_hash” directive is not allowed here in /etc/nginx/passthrough.conf:4
As far as the notes go, it should work but I didn’t have the patience to find the answer.
I selected for my configuration:

hash – The server to which a request is sent is determined from a user‑defined key
$remote_addr – client address
consistent – Requests are evenly distributed across all upstream servers

# LB https to 2 backend servers
stream {
    upstream mm_mydomain_com {
      hash $remote_addr consistent;
      server 192.168.30.20:443;
      server 192.168.30.21:443;
    }

    server {
        listen 443;
        proxy_pass mm_mydomain_com;
        proxy_next_upstream error timeout;
    }
}

Once I had a passthrough.conf that I thought would work, I removed the current NGINX https reverse proxy configuration.

cd /etc/nginx/sites-enabled
ls -l
sudo rm mattermost

I then ran a test against the NGINX configuration sudo nginx -t

The final test was to take my two Mattermost app servers and reboot testing connections to both, which finally worked!

CONCLUSION
This was the best way I had to quickly get a working configuration in place to test Office 365 SSO with Mattermost and give me a chance to learn a bit about NGINX. There are likely better ways I could have completed this but this works well for me for testing Mattermost Office 365 sign on in my lab.

If I was looking at this for a production instance then I would investigate the ip_hash configuration, health_check and proxy_next_upstream settings in more detail.

REFERENCES
Thank you to those who have shared before and used as a basis for this:
https://www.cyberciti.biz/faq/configure-nginx-ssltls-passthru-with-tcp-load-balancing/
https://www.thegeekstuff.com/2017/01/nginx-loadbalancer/
https://www.digitalocean.com/community/tutorials/understanding-nginx-http-proxying-load-balancing-buffering-and-caching

Ansible and Windows a match made in…

…Windows Subsystem for Linux Heaven?

My creative colleague Christian Johannsen has developed an amazing script to deploy a demo environment of Mattermost with two of our DevOps integrations the Jira and Jenkins plugin. He hopes to open source the repo at some point.

It uses terraform to deploy the environment in AWS, bash and python scripts to configure the services and an Ansible playbook for the Jenkins install.

Christian created the instructions to run all the scripts but these were for Mac. I’m a rare breed at Mattermost as I am a Windows user so I had to go through the environment setup myself and modify it for Windows.

I was aiming to run this all natively in Windows or at least in Git Bash but got beaten in the end by Ansible. As they point out Ansible cannot run on a Windows host natively, though it can run under the Windows Subsystem for Linux (WSL). They then go on to say that they do not support the WSL and it shouldn’t be used in production.

Installing WSL on Windows 10
Installing Ansible on WSL

I raised  a query on the Ansible Google forum and a helpful Redhat engineer stated that there are also no plans to support this natively on Windows. “It will work but in the odd case it doesn’t you won’t get any help from GitHub unless you can replicate it on an actual Linux platform.”

So it looks like Windows is an outlier in the Ansible world. Just like me and my Surface Laptop at Mattermost!

intro to the mattermost CLI

At work we have been building out our demo environments and looking to create a re-usable Github repo for deploying and configuring Mattermost. The aim is to automate as much as possible of the mattermost deployment. Mattermost can be configured using the API or the CLI. These are my adventures with the CLI….

The updates from the CLI are automatically written to the config.json. There are some considerations when changing settings:

  • Dotted Notation: you have to put the section of json file ahead of the config setting. Shown below as TeamSettings.ExperimentalDefaultChannels
  • Arrays: CLI accepts multiple values for array settings. In the example as Public-Channel01 PublicChannel-02
"ExperimentalDefaultChannels": []

sudo ./mattermost config set TeamSettings.ExperimentalDefaultChannels Public-Channel01 PublicChannel-02

"ExperimentalDefaultChannels": [
    "Public-Channel01",
    "PublicChannel-02"
]

I’ll will add to this post over time so it is a resource to keep coming back to.

But for now this is the story so far…

/remind me

My memory is not my strongest quality somebody told me once, I forget who.

To help with this Mattermost has a /remind plugin developed by a member of the Mattermost community Scott Davis. This can be really useful as a prompt for your forgetful self or a friendly nudge to @co-worker. If you want to see your options then type:

/remind help

If you want to make sure meeting notes get sent out after your call…

When you receive a notification you get prompted by the Remindbot with a new message in your UNREADS

You have options to complete or delete, and can even reschedule the reminder

To install the plugin go to https://integrations.mattermost.com and search for remind or go to the GitHub page.

A simple productivity tip to help you in your day.