Not a fan of articles that say "Here, type this command" and "Put this in that file" without explaining what's going on or even linking to more information.
The section that suggests putting
tmpfs /dev/shm tmpfs defaults,ro 0 0
into /etc/fstab might as well be quoting a line of ancient scripture. What does that do? Why does it solve the problem described? No answer. Likewise, the part about sshd begins "make sure these lines exist to secure SSH:" but does not explain what the 5 lines that follow do or why it is important.
I prefer guides that explain what they are telling you to do and why, providing a basis for real understanding. To that end, I'd recommend Linode's "Securing Your Server" guide[1] over this.
I found your article to be an interesting start-off point to read up more about certain tools and techniques on securing Ubuntu Linux machines.
So it's absolutely useful, good of you to put it there. Some criticisms on, however:
If the above was the goal, for an advanced audience, it could have been way way shorter. More like a checklist of tools and techniques, short paragraph explaining your choice and maybe a link to your particular config file if you think that's applicable. And that'd be fine, even if it means that I personally would have to Google most things to find out what they're for, it's a great starting point.
If, however, your goal was introducing securing an Ubuntu box to relative newbies, by which I mean people who are quite familiar on the Linux command line (like me) but lack specific knowledge about networking, iptables and the tools of the trade (again, like me), your article should have been way way longer than it is :) Or maybe not longer necessarily, but way more in-depth. Maybe a series, even (that would be awesome, but perhaps a lot of work, I'm not saying you should put in a lot of time that you maybe don't have, just that what you do write ought to be optimal for the effort put in ;) ).
Most of what seems to be missing in your article are answers to the questions "why (should I use this)?" and "what (is this for)?", both for the "advanced" audience, as for the "newbies".
Probably most important is that you must state upfront that this article is about securing a web server. Some of the choices you make are relevant only for web servers, but it wasn't until halfway that you mention "since this machine will be running as a web server ..." and before there were some points I was wondering "hm I'm not sure if I'd want to lock my (desktop) box so tight, but hey this guy seems to know what he's talking about" ... maybe I'm stupid and I should have noticed earlier. In all fairness, if you'd have just titled your article "Securing an Ubuntu web server", I probably wouldn't even have clicked the link (cause I don't have much use for that info right now) and you wouldn't have got all this feedback ;-)
You should have a short sentence explaining "Fail2ban is a program that ...", and another sentence going "iptables is a tool/configfile part of .../command/(no idea honestly[0]) that is used for ...". Not asking for an in-depth explanation, but a one-sentence "what it is".
Most of your checklist of Bastille settings are just cryptic enough that I cannot guess what they would exactly do and whether my settings should be the same as yours. If you target an "advanced" audience, you could suffice with "I'm using Bastille for XYZ because it's good at doing PQR, check its manual, here's my config, in particular pay attention to options ABC". For a more "newbie" audience you might want to say "Bastille is a hardening program, it consists of a set of tools and programs and configures them and your system to harden it. A few examples are PQR, but there's much more". "These are my settings" (do these go into a config file or is it a list of questions that the sudo bastille command asks? you can assume that most of your audience is not using your article as a step-by-step tutorial, so this is not obvious), and then explain succinctly what each of them does. Maybe just copy bits from the Bastille manpage. Or maybe just explain the ones that are different from the default options, and/or the ones that you suppose people might want to change, and say for the others just "make JKL extra secure, leave this set to 'on', <link to docs> for more info".
Hope that was useful for you! (And I don't mean you should do a complete rewrite of this article btw, you can also just take this general advice for the next one you write)
[0] I'm guessing "configfile" and part of some low level Linux networking thing? If I had been sitting in front of a Linux desktop I'd have been reading the manpage on it by now.
> If, however, your goal was introducing securing an Ubuntu box to relative newbies, by which I mean people who are quite familiar on the Linux command line (like me) but lack specific knowledge about networking, iptables and the tools of the trade (again, like me), your article should have been way way longer than it is :) Or maybe not longer necessarily, but way more in-depth. Maybe a series, even (that would be awesome, but perhaps a lot of work, I'm not saying you should put in a lot of time that you maybe don't have, just that what you do write ought to be optimal for the effort put in ;) ).
Careful with setting ro on shm. There are certain things (for instance LXC, which someone mentioned perhaps having a setup guide for), which require access to shm.
Actually, thinking about it more (and others have commented on it already)... changing shm to read-only is pretty much just a bad idea in general to suggest.
> A common exploit vector is going through shared memory (which can let you change the UID of running programs and other malicious actions).
What? Citation needed. /dev/shm is simply a mount point for a memory-backed filesystem, much like /tmp. There's absolutely nothing special you can do with it.
The POSIX shared memory (i.e. shm_open) implementation on Linux does put its files in /dev/shm. By making /dev/shm read-only you're effectively disabling POSIX shared memory. Maybe some applications have vulnerabilities because they use POSIX shared memory unsafely? In that case you probably want to identify and upgrade those applications instead of disabling something system-wide that may cause legitimate programs to break.
To be honest, I don't know what the author means; but I've seen web exploits using /dev/shm as alternative location to dump scripts (the other usual suspects are /tmp and /var/tmp). The web server process can write there, spawn a new process, etc.
I think it's better to protect the entry point (ie. in PHP, use php.ini open_basedir).
I could see making all writable filesystems noexec - _that_ would make sense. Making all of /dev/shm read-only, and thus breaking the POSIX shared memory API? Madness.
It is a common hiding place once the box has been broken into, not necessarily something that would be attacked right away. If you look on google you can see plenty of evidence of people's boxes being compromised through insecure shared memory and you can find plenty of examples of shared memory being exploited for privilege escalation and hiding malicious files.
In automated exploitation (e.g. scanning for common classes of web vulnerabilities) /dev/shm is better because it doesn't touch the disk filesystem. It's also not an obvious place people use on a regular basis (as opposed to /tmp) so (some) people are less likely to look for things in there.
/tmp sometimes uses tmpfs which is also memory based, but not always and you won't know until you're in /tmp.
/dev/shm is a better hiding place than /tmp or /var/tmp because if someone notices something weird on a server their instinct might to be power it off or reboot it. A power cycle will wipe /dev/shm since it is memory backed storage. Lots of places will remove the network, power off the machine, and disk image the affected server for later analysis.
Anyone interested in solid advise / best parctices should read the NSA RHEL hardening guide [1]. It is great and provides nice details. They also have a tips pamphlet for quick reading but limited details [2].
Perhaps, though I am more familiar with Ubuntu/Debian as I have used it extensively on the desktop. This means I am more confident in the implications of my actions and also have more consistent dev/deploy platforms.
It is worth switching, in my opinion, because of all the man-hours put into securing RHEL. The systems aren't THAT different and if you are pretty linux saavy already, I'd guess that you could have an understanding of the differences learned in ~6 weeks.
SELinux itself is a tool for Mandatory Access Control. As it is a tool, it needs to be used correctly. The SELinux policy which is deployed on most OS's by default now is indeed a band-aid. However, when developers and system admins understand how to create correct and secure SElinux policies, it does indeed approach bulletproof.
With SELinux root can build systems which are so secure that root himself cannot hack them.
Lots of questionable advice here (/dev/shm, fail2ban, PermitRootLogin no, "sysctl hardening") with little explanation and glaring omissions (SSH-port left unchanged, no check for listening services, no pointer to an iptables-wrapper).
None of the bullets seems immediately harmful security-wise, but many of them have the potential to break legitimate applications or lock you out of your host.
If you intend to follow this guide then make sure you fully understand each step before applying it.
Doesn't reducing log noise from automated attacks improve your ability to see significant events in the log? And why would it be hard to use a non-standard port?
It did, and probably still does to a lesser extent. It doesn't cost any real time or effort to scan most port ranges these days - especially with EC2 and other big cloud vendors and super fast multi-gigabit network connections.
the value of changing SSH ports versus using only public key authentication is somewhat debatable.
No, there never was a debate. The two changes address different concerns and you should always do both.
You turn off password authentication to prevent targeted brute force attacks, keyloggers and (more common) accidents such as typing your password into the shell.
You change the SSH-port to improve your odds in case of a vulnerability in the SSH-daemon itself (sweep-scans tend to probe only the default port).
I would say that is incorrect. A well versed attacker is going to have absolutely no issue finding your 'new' SSH port. Why do you think this is a security measure? As for shared keys... they also have their drawbacks. Someone can forget their shared key in the wrong place and suddenly you have an extremely easy entry-point into perhaps EVERY single server in an environment.
Use shared keys like you would salt, sparingly and with lots of care and knowledge of how to use it.
attacker is going to have absolutely no issue finding your 'new' SSH port.
As I wrote in my initial comment: It's not about targeted attacks, it's about evading netblock scans.
Use shared keys like you would salt, sparingly and with lots of care and knowledge of how to use it.
Bullshit. Use one password-protected keypair for each critical zone (work, personal, etc.). Keep it on your main machine and do not copy it to "places". Also don't spread FUD on public forums about the most secure ssh auth-option (short of multi-factor).
Excuse me, what? FUD? I wasn't spreading FUD, I was saying that people overuse shared keys and end up putting themselves in situations where they are open to easy exploitation if they don't use them right. People will generate a key and the put it on their dev machine, backup machines and similar all which are generally less secure than production devices. Someone figures out a way in and boom. How is this FUD? It happens in real-world usage when people don't take the time to consider how they should be properly using shared keys.
This is not a drawback of key-based authentication versus passwords because you have the exact same problem with passwords. If you want to explain proper key-management then explain it. But don't make it sound as if key-auth was some esoteric option that should be carefully considered versus password auth. Key auth should always be the default and password auth should always be disabled, exceptions to this rule are very rare.
Bastille linux hasn't been updated in close to 6 years. A year ago they announced that they were restarting development but there hasn't been a release since. Just saying I'd be a little nervous that it hasn't kept up with changes in Ubuntu.
I appreciate the effort but this advice is not especially good.
For example fail2ban is fine but it's almost always better to just use ssh key pairs.
His firewall configuration doesn't do anything basically, just gives access to the network services that are already running, and logging firewall errors is only good if they are actionable (they rarely are) otherwise they are just filling up disk space.
The sysctl entries, I haven't checked one by one by the look like the defaults in a modern distro.
You are right, keys are used in the OP, not sure then why you'd use fail2ban for ssh but OK.
Regarding the sysctl entries, I reviewed them down to execshield (after that there are tweaking entries) in a Amazon EC2 vanilla Ubuntu distro and all the entries are the same except log_martian (debatable, again, are you going to look at the logs, if then what are you going to do with them) and accept_redirects. Execshield is a RedHat thing, it's not used in Ubuntu.
Also, last I heard chroot is not secure for sandboxing, so why bother with it at all? I'd use cgroups or even lxc containers (those are pretty well supported in the latest ubuntu).
It is probably better than nothing (hard to get root if everything in the chroot is running as a user, no suid files, few devices). Probably need a kernel exploit then to get out. A container is better though.
Yeah, so what's the point? If you have a separate user, you can't access other users' files anyway, unless you have root. You can access the system files, but there should be nothing of much interest in there anyway.
You should have only the minimum necessary commands and libraries in your chroot jail which minimizes the attack surface. For instance, I use chroot with rssh to only allow sFTP/scp/rsync and not allowing chroot users to get to a shell prompt.
In my chroot /usr/bin, there are only two files (rssh, scp). In my chroot /lib there are only 16 files.
I guess to prevent an application (e.g a web server running as a non root user) from easily reaching outside and running arbitrary shell commands or reading files.
For example if you have something horrible like a PHP script that does: echo system($_POST['whatever']) you at least should try and prevent it from doing "ls" and "cat" to start exploring the filesystem.
You're correct. chroot works on a per-process basis by adding a relative fs root into the process (or process table? dunno) memory. If you're root, you can change it back to /.
I rely a lot on apparmor to secure an Ubuntu installation. It offers a very fine grained control on what processes are allowed to do, what files they can access. I find it more powerful and easier to setup than a chroot.
Installing mod_security can have nasty side effects. For example, I lost many hours finding cause of my AJAX file uploads failing. It turned out to be caused by this module.
Well, I'm not sure if I'd suggest looking at mod_security in a 'straightforward' manner particularly. Doing so will likely get you frustrated with the interaction you are likely to have with any default rule-set.
Mostly though, my suggestion is to deploy it, have your application break (you should be doing this all in a staging environment), figure out if unbreaking your application can be done with scripting changes, bypassing rules, etc.
You should do due diligence on any advice you read on the internet if you are using it for something important, not just copy & paste.
These are not "security bugs" in the sense of 0-day remote code execution vulnerabilities.
These are simply configuration options which may increase security for certain use cases.
Setting up specific chroots , firewall rules and restrictions could well not make sense for many use cases.
Also stuff like fail2ban can lock you out of your server and requiring key based auth can be more complicated to get end users to configure if you are setting up for something like a shared hosting service.
If you really need security consider using the grsecurity patchset¹ and pax². It's possible to patch the Ubuntu default kernel without problems. However understanding and configuring this takes time and patience. There is a great guide on Wikibooks through³
Some commenters here seem to suggest that the advice on that blog post is somewhat outdated or suboptimal. I'm curious, is there something like a commonly agreed on best practice for securing/hardening a modern Ubuntu production web server? Would maybe anyone care to provide a bullet point list of the most important steps?
1) Making sure SSH only accepts public keys. This removes the possibility of a hacker guessing a password.
2) Use fail2ban. This rate-limits SSH attempts.
3) Keep your packages up-to-date, especially Web applications.
4) Use a default-deny firewall configuration, and make sure your services bind to the localhost interface. Letting the outside world directly talk to your database, Redis and memcache lets someone see your application data without any security checks, and this should only be possible for the application itself.
Is there a ready-made trusted image that people can use as a secure base?
I'm not exactly good with dev ops and tweaking operating sytems. Even If I do attempt to fix it up and secure it, it will leave me uneasy because I don't fully understand the things I've done.
If there was something trusted and ready-made would be great.
Rather than manually configuring iptables, save yourself some time and use UFW (the Uncomplicated FireWall) https://help.ubuntu.com/10.04/serverguide/firewall.html which is far easier and just as secure (and reduces the risk of misconfiguration).
Can anyone share their experiences with the grsecurity/pax patches? I have only used it on firewalls that had no responsibilities other than moving packets. Have you had any issues using the patches on machines with more responsibilities (www/db/mail) and/or greater user interaction ie. desktop use??
I used it on several multi-purpose machines, no desktop through. I just patched the stock Ubuntu kernel¹. If you enable pax you have to watch out for applications that are incompatible with it. GRUB, Java, mod_python broke out of the box, but you can change the flags for pax on a per binary basis. It's probably useful to write some apt rules to apply the changes on every update of the program. Otherwise there where no problems. Using RBAC and configuring it on the other hand is a more difficult process.
Another little tidbit you can use in addition to fail2ban is to just change the SSH port. I generally use something in the 2000 range - 2022, 2222 etc. That way your poor VPS isn´t even handling all the people knocking on your SSH door.
I don't agree with this. A simple nmap is enough to find the SSH port. This will get you in trouble when you're on a network which blocks port. The ssh port is often open (at least if it's an establishment where programmers work) and a port like 2022 is not.
I am not an admin or anything. But won't the firewall rules lock down any access to : subversion, git, irc, or pretty much anything that doesn't use http or https.
I'm more interested in Ubuntu being more stable as a desktop environment than secure. If I was worried about security I'd use Debian stable or some other operating system more well known for its security like OpenBSD.
/dev/shm is used by the POSIX shared memory implementation (i.e. shm_open) and can also be used as a general temporary directory (like /tmp) that's memory-backed. Legitimate applications use it and legitimate applications will break if you follow this advice.
There's nothing inherently unsafe with /dev/shm - it's very similar to /tmp. It's possible some applications use POSIX shared memory unsafely, but that's a problem with those applications. Doing something drastic like making /dev/shm read-only, if it even serves a legitimate security purpose, should only be done as a last resort.
Read-only would be bad! You want no-execute on /tmp/, /var/tmp, /dev/shm and they should be their own partitions on a secure system.
The reasoning is that if someone does break into your system via apache/php or something else, they will almost always want to get a binary to elevate their access level (i.e. root the machine). They download a binary to a temporary location and execute the code (This can happen in apache/php directories too but most of the time these are locked down). Setting these to no-execute does not allow the attacker to execute code on your system. This is just a minor slowdown but does greatly help to prevent automated attacks.
The goal is to slow people down to improve your chances of being alerted. I think this is fiarly standard and is listed as a best practice via the NSA RHEL hardening guide [1].
"Setting these to no-execute does not allow the attacker to execute code on your system". It depends, with /lib/ld-linux.so.2 /tmp/script or bash /tmp/script.sh it may be possible to execute code on a noexec /tmp .
As you said, setting noexec still helps preventing automated attacks.
It really doesn't matter, its permissions are 0664 and the uid/guid are 0/0 (root.root).
It may protect you in case of a mistake, your editor should warn you that the file is read-only.
I don't like the post, it's too complicated for small benefits (if any). Besides there's no "generic secure setup", I think it depends on what you want to do with that server (ie. making /dev/shm read only may have impact on some software).
I tend to agree with you about the "generic secure setup". The problem with making read only some parts of the file system, as other persons said too, is when a legit application will want to access those locations, a better solution is to remove execute rights on the desired locations.
It seems like Unity and Ubuntu will continue to become more and more social. Social in simple words mean sending private data to Canonical and third party servers. It's going to get very hard to keep Ubuntu box secure.
The security article is relevant to any system based on Ubuntu core with server like processes. You don't actually have to be on ubuntu-server running on a dedicated server computer in your basement. That being said. I think there is no harm in discussing Ubuntu Desktop security.
The section that suggests putting
into /etc/fstab might as well be quoting a line of ancient scripture. What does that do? Why does it solve the problem described? No answer. Likewise, the part about sshd begins "make sure these lines exist to secure SSH:" but does not explain what the 5 lines that follow do or why it is important.I prefer guides that explain what they are telling you to do and why, providing a basis for real understanding. To that end, I'd recommend Linode's "Securing Your Server" guide[1] over this.
[1] https://library.linode.com/securing-your-server