Category Archives: Security

How to monitor Docker containers with Nagios and NRPE

Monitoring whether or not a Docker container is alive on a remote host should be fairly easy, right?

The standard approach in this is to include a suitable NRPE script on the remote host, and call that remotely from your Nagios server via the NRPE TCP daemon on the remote host. This script is a good example of same, and we’ll refer to it in the rest of the article.

This generally works fine when you’re doing innocuous things like checking free disk space or if a certain process is running. Checking a Docker container is a little bit harder, because the command:

docker inspect

can only be run as root, whereas the NRPE service on the remote host runs as a non-privileged user (usually called nagios).

As such, when you test your NRPE call from the Nagios server, like so:

/usr/lib64/nagios/plugins/check_nrpe -H dockerhost.yourdomain.com -c check_docker_container1

Your will see a response like:

NRPE: Unable to read output

or

UNKNOWN - container1 does not exist.

You get this response because the nagios user cannot execute the docker control command.

Your could get around this by running NRPE on the remote host as the root user, but that really isn’t a good idea, and you should never do this.

A better play (if you are confident that your Nagios set up is secure) is to extend controlled privileged to the nagios user via sudo. You can create the following file in /etc/sudoers.d/docker to achieve this:

nagios    ALL=(ALL:ALL)  NOPASSWD: /usr/bin/docker inspect *
nagios    ALL=(ALL:ALL)  NOPASSWD: /usr/lib64/nagios/plugins/check-docker-container.sh *

This allows the nagios user to run both the wrapper script around the docker inspect command and the docker control command itself, without requiring a password. Note, only inspect permission is granted. Obviously, we don’t want to give nagios permission to actually manipulate containers.

In addition to this, we must make provision for NRPE to run the command using sudo when called via the NRPE TCP daemon. So, in nrpe.cfg, instead of:

command[check_docker_container1]=/usr/lib64/nagios/plugins/check-docker-container.sh container1

we have:

command[check_docker_container1]=sudo /usr/lib64/nagios/plugins/check-docker-container.sh container1

 

 

Never presume its safe to use your credit card online

In light of the recent information security breach at TalkTalk, I thought it would be a good opportunity to share my thoughts on information security and the use of credit cards to purchase goods and services online.

I am not by any means an information security expert, but I can credibly claim to have more knowledge of the subject that the average member of the public, and probably the average IT professional too.

My experience derives primarily from managing systems used to book flights and hotel rooms, in which customer credit cards are used as the method of payment. In my most recent role in this regard, over €40m of revenue per month was flowing through systems under my control.

To understand the underlying risk in passing sensitive data to a computer system you encounter online, the most important point to understand is that for every commercial organisation that ever existed, information security is a drag on profitability.

In itself, that isn’t surprising or unique. There are lots of business functions that are a drag on profitability. The difference with information security is that while its significance is generally understood by decision makers, the complexity of the risk involved is not, which means that when its drag factor is considered with every other drag factor, its tends to get bumped down the list more easily when decisions have to be made about priorities.

For instance, if a Marketing Director says to a CEO that a product launch needs to be delayed to develop a new advertising campaign, because a focus group indicated that the original marketing campaign was not appealing, the logic is immediately accessible to the CEO, and the decision is relatively simple.

If, an the other hand, an IT Director says to a CEO that a new product launch has to be delayed because the software underpinning the payment system for the product hasn’t been penetration tested for Cross Site Scripting vulnerabilities, the logic is less accessible, and the CEO will probably consider the situation of terms of probability, not logic.

“What are the chances of our software being targeted when there are millions of online systems for hackers to choose from? We’ve released products with this software before and everything was fine, so why not this time?”

This type of thinking is pervasive in corporations that rely on information systems and ask us to trust them with our data. When it comes to information security, more often than not, risk is considered in terms of what is probable, not what is possible. The critical flaw in this is that a decision makers estimation of probability is always influenced both by their experience and their wider commercial objectives. If they keep subconsciously diluting risk because it interferes with their commercial objectives, and nothing ever goes wrong, it becomes easier to dilute that risk further and further each time. More often than not they’ll get away with this. There are millions of online systems, and you need to be unlucky to be targeted, but you’re just as likely to be targeted as any one else.

A real life example of this is readily available. I recently had to make a trip to England, and needed to book 4 days parking at Dublin Airport, which would cost in the region of €30, which I knew I would have to pay for with my credit card.

Being relatively familiar with what goes on behind the scenes on such sites,  I tend to rank the security they offer higher than more obvious considerations like price. The gold standard for me is the availability of PayPal as a payment option. There was a time when asking users to pay using PayPal was seen as second rate, which resulted in many online retailers discounting it in favour of custom solutions that made their online presence seem apparently more sophisticated.

Thankfully, those days are gone. PayPal have a proven track record when it comes to credit card security, and my advise would be to always choose PayPal as your payment option rather than entering your card details anew into a integrated payment system.

In the absence of PayPal, I would always favour payment solutions that link into payment gateways like Realex and WorldPay. These solutions require you to enter your credit cards details, but these are either forwarded directly to the payment gateway provider, or entered on a page provided by the gateway provider. The key thing to understand is that the organisation you purchasing the product or service from has very limited responsibility in terms of managing your credit card data. This is a good thing, because it means they have to deal with the issue of information security being a drag on profitability less frequently than organisations that attempt to process credit card payments themselves.

When neither of these options are obviously available, I will always look for an online statement from the product or service provider about how they deal with credit card payments. The key element to look for in such a statement is either attestation to or certification of what is known as PCI DSS compliance.

PCI DSS compliance is a set of standards agreed by the credit card industry which organisations storing or communicating credit card data are supposed to adhere to. There is no law or industry requirement that they should, although many banks will refuse to deal with organisations if they don’t.

Implementing PCI DSS compliance in an organisation is an onerous and expensive task. Not only is the baseline standard difficult to achieve, particularly if the organisation has grown rapidly and the standard has to be implemented retrospectively, but it is also evolving all the time, requiring organisations to devote dedicated resources to ensuring that processes and procedures are up to the date. Its also a standard that involves much more than software. It touches every part of the organisation, from HR to Accounting to Marketing to IT.

Any organisation that has been through this knows the pain involved, and when you achieve compliance, its something you want to tell people about. As such, if I were dealing with an organisation that has full PCI DSS compliance in place, I’d expect them to make that clear on their website.

Lets take a look at the parking options available at Dublin Airport to see how they fared in relation to the payments options available to me.

The first one I tried, because it was the cheapest, was http://quickpark.ie/. I got my quote and clicked through the booking process to where my credit card details were required. There was no PayPal option, and no evidence of the site using a payment gateway, so I started searching for some evidence of PCI DSS compliance. This is what I found on their Frequently Asked Questions page:

How do I know my booking details are secure?

You can rest assured that your personal data is safe with us. Every booking is encrypted via SSL protocol. Along with encryption we take all the measures required to keep all your personal data safe.

This is an incredibly anaemic statement for a website asking you to enter your credit card details. Their reference to the “SSL protocol” means that your details are encrypted while they are in transit from your computer to theirs (which is a very basic standard), but no information is provided regarding what happens to your details once they arrive at the other end. The reference to “SSL” is also telling. The primary protocols used in the transfer of data across the web are SSL and TLS. Up until about a year ago, SSL was predominant, until a flaw was discovered in it, resulting in the well-managed websites switching to TLS. The fact that QuickPark’s web site continues to refer to “SSL” is not encouraging, never mind the total absence or any reference to the PCI DSS standard.

My next attempt was on the Dublin Airport Authority’s website, where you can book parking in the parking areas owned by the airport. This was at http://www.dublinairport.com/gns/to-from-the-airport/car-parking.aspx.

Again, there was no PayPal option, and no reference to a payment gateway provider, so I again went searching for a statement on information security. This is what I found on their Pre-Booking Frequently Asked Questions page:

How do I know my booking details are secure?

The details you provide are encrypted to prevent them being read over the internet. This is indicated by the GeoTrust icon on the car parking page. You can click on the icon for more detail.

This statement is similarly anaemic to the one provided by QuickPark, which surprised me, given that the Dublin Airport Authority is a long-standing semi-state body, compared to QuickPark, which is a relatively small private company.

Again there is reference to the transmission of data over the internet, but no reference to the management of data on the other side. The reference to the GeoTrust logo is meaningless, but was presumably included as it features the word “Trust”. Obtaining a a GeoTrust logo for your website, or a logo from any one of hundreds of similar providers, costs about $20 per annum. All it signifies is from whom you bought your digital certificate to encrypt your data transmission. It means nothing is terms of how your data is managed by the company once its gets to its destination.

At this point, I decided to change tack, and did a Google search for “Dublin airport parking pci dss”.

The first result I got was for the Clayton Hotel, which is just off the motorway near the exit for the airport, and which offers car parking to airport users.

On their parking Frequesting Asked Questions page (http://www.claytonhoteldublinairport.com/park-and-fly/parking-faqs/) they state:

How do I know my booking details are secure?

To ensure that you are trading in a secure environment, Clayton Hotel Dublin Airport has contracted the services of Advam. Advam is the leading provider of integrated global card services for private enterprise and government agencies in Australia and around the globe. Advam is a Tier 1 payment processor which adheres to the most stringent of industry accreditations including Level 1 PCI DSS compliance, EMV certification and ISO 9001 accreditation. When you enter your payment details online, you will notice that you will are using a secure site which uses 1024 bit tunnelling encryption to protect your information during transmission. Every transaction processed through Advam’s payment switch is protected by the latest in encryption technology and a combination of state of the art firewalls and intrusion detection systems guard every point of ingress and egress on the Advam network.

This was obviously cut and paste from another document provided by the company referenced in it, Advam, but that in itself isn’t a problem. From this statement I can see that not only is the transmission of the data secure, but that responsibility for the data has been handed of to a third party who specialise in credit card security and who have achieved PCI DSS compliance. This is the type of statement I would expect to see from an organisation asking for my credit card data, and this was the option I chose.

In considering this example, I need to go back to my earlier point about information security being a drag on profitability. As noted, its one drag in a mix of many different drags, but its a drag that tends to get pushed aside because it isn’t one that decision makers can easily relate to.

When this is not the case, or in other words when a decision maker has decided to sufficiently prioritise information security, its a painful process for the organisation involved, and part of the payback is making sure that everyone knows the effort you’ve gone to, particularly if your competitors haven’t.

From that point of view, finding anaemic statements like those referred to above turns on a warning light for me. The absence of more comprehensive information about information security doesn’t mean that these organisation are insecure, but it does mean that they aren’t particularly bothered about promoting information security as a feature of their service offering, which suggests they’ve haven’t invested particularly deeply in it.

At this juncture, it would be nice to be able to go back and view what TalkTalk said about their information security before they were hacked, but at the time of writing, the entire TalkTalk website is just one big blurb about how TalkTalk take information security more seriously than anything else. Presumably, it will be that way for some time.

That said, its unlikely that very many of TalkTalk’s customers ever bothered checking out their statements about information security.

If you don’t want to be in the same position they are in today, its probably a habit you should get in to.

 

 

Command line tool for checking status of instances in Amazon EC2

I manage between 10 and 15 different Amazon AWS accounts for different companies.

When I needed to find out information about a particular instance, it was a pain to have log into the web interface each time. Amazon do provide an API that allows you query data about instances, but to use that, you need to store an Access Key and Secret on your local computer, which isn’t very safe when you’re dealing with multiple account.

To overcome, this I patched together Tim Kay’s excellent aws tool with GPG and a little PHP, to create a tool which allows you query the status of all instances in a specific region in an Amazon EC2 account, using access credentials that are locally encrypted, so that storing them locally isn’t an issue.

Output from the tool is presented on a line by line basis, so you can use grep to filter the results.

Sample output:

ec2sitrep.sh aws.account1 us-east-1

"logs-use"  running  m1.medium  us-east-1a  i-b344b7cb  172.32.1.172  59.34.113.133
"adb2-d-use"  running  m1.small  us-east-1d  i-07d3e963  172.32.3.54  67.45.139.235
"pms-a-use"  running  m1.medium  us-east-1a  i-90852ced  172.32.1.27  67.45.108.146
"s2-sc2-d-use"  running  m1.medium  us-east-1d  i-3d40b442  172.32.3.26  67.45.175.244
"ks2-sc3-d-use"  running  m1.small  us-east-1d  i-ed2ed492  172.32.3.184  67.45.163.141
"ks1-sc3-c-use"  running  m1.small  us-east-1c  i-6efb9612  172.32.2.195  67.45.159.221
"adb1-c-use"  running  m1.small  us-east-1c  i-98cf44e4  172.32.2.221  67.45.139.196
"s1-sc1-c-use"  running  m1.medium  us-east-1c  i-956a76e8  172.32.2.96  67.45.36.97
"sms2-d-use"  running  m1.medium  us-east-1d  i-a86ef686  172.32.3.102  34.90.28.159
"uatpms-a-use"  running  m1.small  us-east-1a  i-b8cf5399  172.32.1.25  34.90.163.110
"uatks1-sc3-c-use"  running  t1.micro  us-east-1c  i-de336dfe  172.32.2.26  34.90.99.226
"uats1-sc1-c"  running  m1.medium  us-east-1c  i-35396715  172.32.2.217  34.90.183.23
"uatadb1-c-use"  running  t1.micro  us-east-1c  i-4d316f6d  172.32.2.29  34.90.109.171
"sms1-c-use"  running  m1.medium  us-east-1c  i-31b29611  172.32.2.163  34.90.100.25

(Note that public ips have been changed in this example)

You can obtain the tool from Bitbucket:

https://bitbucket.org/garrethmcdaid/amazon-ec2-sitrep/

How to monitor the Amazon Linux AMI Security Patch RSS feed with Nagios

People who use Amazon AWS will be familiar with the Amazon Linux AMI, which is a machine image provided by Amazon with a stripped down installation of Linux.

The AMI acts as a starting point for building up your own AMIs, and has its own set of repos maintained by Amazon for obtaining software package updates.

Amazon also maintains an RSS feed, which announces the availability of new security patches for the AMI.

One of the requirements of PCI DSS V2 compliance is as follows:

6.4.5 Document process for applying security patches and software updates

That means you have to have a written down process for being alerted to and applying software patches to servers in your PCI DSS scope.

You could of course commit to reading the RSS feed every day, but that’s human intervention, which is never reliable. You could also set up your Amazon servers to simply take a system wide patch update every day, but if you’d prefer to review the necessity and impact of patches before applying them, that isn’t going to work.

Hence, having your monitoring system tell you if a new patch has been released for a specific software component would be nice thing to have, and here it is, in the form of a Nagios plugin.

The plugin is written in PHP (I’m a ex-Web Developer) but is just as capable as when it comes to Nagios as PERL and Python (without the need for all those extra modules).

I’ve called it check_rss.php, as it can be used on any RSS feed. There is another check_rss Nagios plugin, but it won’t work in this instance, as it only checks the most recent port in the RSS stream, and doesn’t include any way to retire alerts.

You can obtain the Plugin from Bitbucket:

https://bitbucket.org/garrethmcdaid/nagios-rss-checker/src

The script takes the following arguments:

“RSS Feed URL”

“Quoted, comma Separated list of strings you want to match in the post title”

“Number of posts you want to scan”

“Number of days for which you want the alert to remain active”

eg

commands.cfg

define command {
    command_name check_rss
    command_line $USER1$/check_rss.php $ARG1$ $ARG2$ $ARG3$ $ARG4$
}

<sample>.cfg

<snip>
check_command   check_rss!http://aws.amazon.com/rss/amazon-linux-ami.rss!"openssl"!30!3
<snip>

You need to tell Nagios how long you want the alert to remain active, as you have no way of resolving the alert (ie you can’t remove it from the RSS feed)

This mechanism allows you to “silence” the alert after a number of days. This isn’t a feature of Nagios, rather of the script itself.

The monitor will alert if it finds *any* patches, and include *all* matching patches in its alert output.