livialima

joined 2 years ago
MODERATOR OF
linuxupskillchallenge
sorted by: new top controversial old
7
Day 14 - Users and Groups (programming.dev)
 

INTRO

Today you're going to set-up another user on your system. You're going to imagine that this is a help-desk person that you trust to do just a few simple tasks:

  • check that the system is running
  • check disk space with: df -h

...but you also want them to be able to reboot the system, because you believe that "turning it off and on again" resolves most problems :-)

You'll be covering a several new areas, so have fun!

ADDING A USER

Choose a name for your new user - we'll use "helen" in the examples, so to add this new user:

sudo adduser helen

(Names are case-sensitive in Linux, so "Helen" would be a completely different user)

The "adduser" command works very slightly differently in each distro - if it didn't ask you for a password for your new user, then set it manually now by:

sudo passwd helen

You will now have a new entry in the simple text database of users: /etc/passwd (check it out with: less), and a group of the same name in the file: /etc/group. A hash of the password for the user is in: /etc/shadow (you can read this too if you use "sudo" - check the permissions to see how they're set. For obvious reasons it's not readable to just everyone).

If you're used to other operating systems it may be hard to believe, but these simple text files are the whole Linux user database and you could even create your users and groups by directly editing these files - although this isn’t normally recommended.

Additionally, adduser will have created a home directory, /home/helen for example, with the correct permissions.

Login as your new user to confirm that everything works. Now while logged in as this user try to run reboot - then sudo reboot.

CLEVER SUDO TRICKS

Your new user is just an ordinary user and so can't use sudo to run commands with elevated privileges - until we set them up. We could simply add them to a group that's pre-defined to be able to use sudo to do anything as root - but we don't want to give "helen" quite that amount of power.

Use ls -l to look at the permissions for the file: /etc/sudoers This is where the magic is defined, and you'll see that it's tightly controlled, but you should be able to view it with: sudo less /etc/sudoers You want to add a new entry in there for your new user, and for this you need to run a special utility: visudo

To run this, you can temporarily "become root" by running:

sudo -i

Notice that your prompt has changed to a "#"

Now simply run visudo to begin editing /etc/sudoers - typically this will use nano.

All lines in /etc/sudoers beginning with "#" are optional comments. You'll want to add some lines like this:

# Allow user "helen" to run "sudo reboot"
# ...and don't prompt for a password
#
helen ALL = NOPASSWD:/sbin/reboot

You can add these line in wherever seems reasonable. The visudo command will automatically check your syntax, and won't allow you to save if there are mistakes - because a corrupt sudoers file could lock you out of your server!

Type exit to remove your magic hat and become your normal user again - and notice that your prompt reverts to: $

TESTING

Test by logging in as your test user and typing: sudo reboot Note that you can "become" helen by:

sudo su helen

If your ssh config allows login only with public keys, you'll need to setup /home/helen/.ssh/authorized_keys - including getting the owner and permissions correct. A little challenge of your understanding of this area!

EXTENSION

If you find this all pretty familiar, then you might like to check and update your knowledge on a couple of related areas:

RESOURCES

Some rights reserved. Check the license terms here

5
Day 13 - Who has permission? (programming.dev)
 

INTRO

Files on a Linux system always have associated "permissions" - controlling who has access and what sort of access. You'll have bumped into this in various ways already - as an example, yesterday while logged in as your "ordinary" user, you could not upload files directly into /var/www or create a new folder at /.

The Linux permission system is quite simple, but it does have some quirky and subtle aspects, so today is simply an introduction to some of the basic concepts.

This time you really do need to work your way through the material in the RESOURCES section!

OWNERSHIP

First let's look at "ownership". All files are tagged with both the name of the user and the group that owns them, so if we type "ls -l" and see a file listing like this:

-rw------- 	1 steve  staff  	4478979  6 Feb  2011 private.txt
-rw-rw-r-- 	1 steve  staff  	4478979  6 Feb  2011 press.txt
-rwxr-xr-x 	1 steve  staff  	4478979  6 Feb  2011 upload.bin

Then these files are owned by user "steve", and the group "staff".

PERMISSIONS

Looking at the '-rw-r--r--" at the start of a directory listing line, (ignore the first "-" for now), and see these as potentially three groups of "rwx": the permission granted to the user who owns the file, the "group", and "other people".

For the example list above:

  • private.txt - Steve has "rw" (ie Read and Write) permission, but neither the group "staff" nor "other people" have any permission at all
  • press.txt - Steve can Read and Write to this file too, but so can any member of the group "staff" - and anyone can read it
  • upload.bin - Steve can write to the file, all others can read it. Additionally all can "execute" the file - ie run this program

You can change the permissions on any file with the chmod utility. Create a simple text file in your home directory with vim (e.g. tuesday.txt) and check that you can list its contents by typing: cat tuesday.txt or less tuesday.txt.

Now look at its permissions by doing: ls -ltr tuesday.txt

-rw-rw-r-- 1 ubuntu ubuntu   12 Nov 19 14:48 tuesday.txt

So, the file is owned by the user "ubuntu", and group "ubuntu", who are the only ones that can write to the file - but any other user can read it.

Now let’s remove the permission of the user and "ubuntu" group to write their own file:

chmod u-w tuesday.txt

chmod g-w tuesday.txt

...and remove the permission for "others" to read the file:

chmod o-r tuesday.txt

Do a listing to check the result:

-r--r----- 1 ubuntu ubuntu   12 Nov 19 14:48 tuesday.txt

...and confirm by trying to edit the file with nano or vim. You'll find that you appear to be able to edit it - but can't save any changes. (In this case, as the owner, you have "permission to override permissions", so can can write with :w!). You can of course easily give yourself back the permission to write to the file by:

chmod u+w tuesday.txt

GROUPS

On most modern Linux systems there is a group created for each user, so user "ubuntu" is a member of the group "ubuntu". However, groups can be added as required, and users added to several groups.

To see what groups you're a member of, simply type: groups

On an Ubuntu system the first user created (in your case ubuntu), should be a member of the groups: ubuntu, sudo and adm - and if you list the /var/log folder you'll see your membership of the adm group is why you can use less to read and view the contents of /var/log/auth.log

The "root" user can add a user to an existing group with the command:

usermod -a -G group user

so your ubuntu user can do the same simply by prefixing the command with sudo. For example, you could add a new user fred like this:

adduser fred

Because this user is not the first user created, they don't have the power to run sudo - which your user has by being a member of the group sudo.

So, to check which groups fred is a member of, first "become fred" - like this:

sudo su fred

Then:

groups

Now type "exit" to return to your normal user, and you can add fred to this group with:

sudo usermod -a -G sudo fred

And of course, you should then check by "becoming fred" again and running the groups command.

POSTING YOUR PROGRESS

Just for fun, create a file: secret.txt in your home folder, take away all permissions from it for the user, group and others - and see what happens when you try to edit it with vim.

EXTENSION

Research:

  • umask and test to see how it's setup on your server
  • the classic octal mode of describing and setting file permissions. (e.g. chmod 664 myfile)

Look into Linux ACLs:

Also, SELinux and AppArmour:

RESOURCES

Some rights reserved. Check the license terms here

7
Day 12 - Transferring files (programming.dev)
 

INTRO

You've now had a working Internet server of your own for some time, and seen how you can create and edit small files there. You've created a web server where you've been able to edit a simple web page.

Today we'll be looking at how you can move files between your other systems and this server - tasks like:

  • Taking a copy of some files from your server onto your desktop machine
  • Copying up some text to your server to put on your webpage
  • Uploading some photos and logos for your webpage

YOUR TASKS TODAY

  • Upload a file to the server
  • Download a file from the server
  • Synchronize a backup

PROTOCOLS

There are a wide range of ways a Linux server can share files, including:

  • SMB: Microsoft's file sharing, useful on a local network of Windows machines
  • AFP: Apple’s file sharing, useful on a local network of Apple machines
  • WebDAV: Sharing over web (http) protocols
  • FTP: Traditional Internet sharing protocol
  • scp: Simple support for copying files
  • rsync: Fast, very efficient file copying
  • SFTP: file access and copying over the SSH protocol (Despite the name, the SFTP protocol at a technical level is completely unrelated to traditional FTP)

Each of these have their place, but for copying files back and forth from your local desktop to your server, SFTP has a number of key advantages:

  • No extra setup is required on your server
  • Top quality security
  • Allows browsing through the directory structure
  • You can create and delete folders

If you’re successfully logging in via ssh from your home, work or a cybercafe then you'll also be able to use SFTP from this same location because the same underlying protocol is being used.

By contrast, setting up your server for any of the other protocols will require extra work. Not only that, enabling extra protocols also increases the "attack surface" - and there's always a chance that you’ll mis-configure something in a way that allows an attacker in. It's also very likely that restrictive firewall policies at a workplace will interfere with or block these protocols. Finally, while old-style FTP is still very commonly used, it sends login credentials "in clear", so that your flatmates, cafe buddies or employer may be able to grab them off the network by "packet sniffing". Not a big issue with your "classroom" server - but it's an unacceptable risk if you're remotely administering production servers.

SFTP client software

What’s required to use SFTP is some client software. A command-line client (unsurprisingly called sftp) comes standard on every Apple OSX or Linux system. If you're using a Linux desktop, you also have a built-in GUI client via your file manager. This will allow you to easily attach to remote servers via SFTP. (For the Nautilus file manager for example, press ctrl + L to bring up the 'location window" and type: sftp://username@myserver-address).

Although Windows and Apple macOS have no built-in GUI client there are a wide range of third-party options available, both free and commercial. If you don't already have such a client installed, then choose one such as:

  • WinSCP or FileZilla - for Windows users
  • CyberDuck or FileZilla - for macOS users

Download locations are under the RESOURCES section.

Configuring and using your choice of these should be straightforward. The only real potential for confusion is that these clients generally support a wide range of protocols such as scp and FTP that we're not going to use. When you're asked for SERVER, give your server's IP address, PORT will be 22, and PROTOCOL will be SFTP or SSH.

INSTRUCTIONS

  • Configure your chosen SFTP client to login to your server as your username
  • Copy some files from your server down to your local desktop (try files from your "home" folder, and from /var/log)
  • Create an "images" folder under your "home" folder on the server, and upload some images to it from your desktop machine
  • Go up to the root directory. You should see /etc, /bin and other folders. Try to create an "images" folder here too - this should fail because you are logging in as an ordinary use, so you won't have permission to create new files or folders. In your own "home" directory you of course have full permission.

Once the files are uploaded you can login via ssh and use sudo to give yourself the necessary power to move files about.

RESOURCES

Some rights reserved. Check the license terms here

7
Day 11 - Finding things... (programming.dev)
 

INTRO

Today we’ll look at how you find files, and text inside these files, quickly and efficiently.

It can be very frustrating to know that a file or setting exists, but not be able to track it down! Master today’s commands and you’ll be much more confident as you administer your systems.

Today you’ll look at some useful tools:

  • locate
  • find
  • grep
  • which

YOUR TASKS TODAY

  • Find all files that have the word "Permission" in it

INSTRUCTIONS

locate

If you're looking for a file called access.log then the quickest approach is to use "locate" like this:

$ locate access.log
/var/log/apache2/access.log
/var/log/apache2/access.log.1
/var/log/apache2/access.log.2.gz

(If locate is not installed, do so with sudo apt install mlocate)

As you can see, by default it treats a search for "something" as a search for "*something*". It’s very fast because it searches an index, but if this index is out of date or missing it may not give you the answer you’re looking for. This is because the index is created by the updatedb command - typically run only nightly by cron. It may therefore be out of date for recently added files, so it can be worthwhile updating the index by manually running: sudo updatedb.

find

The find command searches down through a directory structure looking for files which match some criteria - which could be name, but also size, or when last updated etc. Try these examples:

find /var -name access.log
find /home -mtime -3

The first searches for files with the name "access.log", the second for any file under /home with a last-modified date in the last 3 days.

These will take longer than locate did because they search through the filesystem directly rather from an index. Also, because find uses the permissions of the logged-in user you’ll get “permission denied” messages for many directories if you search the whole system. Starting the command with sudo of course will run it as root - or you could filter the errors with grep like this: find /var -name access.log 2>&1 | grep -vi "Permission denied".

These examples are just the tip of a very large iceberg, check the articles in the RESOURCES section and work through as many examples as you can - time spent getting really comfortable with find is not wasted.

grep -R

Rather than asking "grep" to search for text within a specific file, you can give it a whole directory structure, and ask it to recursively search down through it, including following all symbolic links (which -r does not). This trick is particularly handy when you "just know" that an item appears "somewhere" - but are not sure where.

As an example, you know that “PermitRootLogin” is an ssh parameter in a config file somewhere under /etc, but can’t recall exactly where it is kept:

grep -R -i "PermitRootLogin" /etc/*

Because this only works on plain text files, it's most useful for the /etc and /var/log folders. (Notice the -i which makes the search “case insensitive”, finding the setting even if it’s been entered as “Permitrootlogin”

You may now have logs like /var/log/access.log.2.gz - these are older logs that have been compressed to save disk space - so you can't read them with less, or search them with grep. However, there are zless and zgrep, which do work, and on ordinary as well as compressed files.

which

It's sometimes useful to know where a command is being run from. If you type nano, and it starts, where is the nano binary coming from? The general rule is that the system will search through the locations setup in your "path". To see this type:

echo $PATH

To see where nano comes from, type:

which nano

Try this for grep, vi and service and reboot. You'll notice that they’re typically always in subfolders named bin, but that there are several different ones.

EXTENSION

The -exec feature of the find command is extremely powerful.

But "finding things" can go so much further than that! You can not only track down the content of a file, but also its usage with commands like lsof and fuser.

Test some examples of this from the RESOURCES links.

RESOURCES

TROUBLESHOOT AND MAKE A SAD SERVER HAPPY!

Practice what you've learned with some challenges at SadServers.com:

Some rights reserved. Check the license terms here

5
Day 10 - Getting the computer to do your work for you (programming.dev)
 

INTRO

Linux has a rich set of features for running scheduled tasks. One of the key attributes of a good sysadmin is getting the computer to do your work for you (sometimes misrepresented as laziness!) - and a well configured set of scheduled tasks is key to keeping your server running well.

YOUR TASKS TODAY

  • Schedule a job to apt update and apt upgrade everyday

CRON

Each user potentially has their own set of scheduled task which can be listed with the crontab command (list out your user crontab entry with crontab -l and then that for root with sudo crontab -l ).

However, there’s also a system-wide crontab defined in /etc/crontab - use less to look at this. Here's example, along with an explanation:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
17 *	* * *   root	cd / && run-parts --report /etc/cron.hourly
25 6	* * *   root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6	* * 7   root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6	1 * *   root	test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

Lines beginning with "#" are comments, so # m h dom mon dow user command defines the meanings of the columns.

Although the detail is a bit complex, it's pretty clear what this does. The first line says that at 17mins after every hour, on every day, the credential for "root" will be used to run any scripts in the /etc/cron.hourly folder - and similar logic kicks off daily, weekly and monthly scripts. This is a tidy way to organise things, and many Linux distributions use this approach. It does mean we have to look in those /etc/cron.* folders to see what’s actually scheduled.

On your system type: ls /etc/cron.daily - you'll see something like this:

$ ls /etc/cron.daily
apache2  apt  aptitude  bsdmainutils  locate  logrotate  man-db  mlocate  standard  sysklog

Each of these files is a script or a shortcut to a script to do some regular task, and they're run in alphabetic order by run-parts. So in this case apache2 will run first. Use less to view some of the scripts on your system - many will look very complex and are best left well alone, but others may be just a few lines of simple commands.

Look at the articles in the resources section - you should be aware of at and anacron but are not likely to use them in a server.

Google for "logrotate", and then look at the logs in your own server to see how they've been "rotated".

SYSTEMD TIMERS

All major Linux distributions now include "systemd". As well as starting and stopping services, this can also be used to run tasks at specific times via "timers". See which ones are already configured on your server with:

systemctl list-timers

Use the links in the RESOURCES section to read up about how these timers work.

RESOURCES

Some rights reserved. Check the license terms here

6
Day 9 - Diving into networking (programming.dev)
 

INTRO

The two services your server is now running are sshd for remote login, and apache2 for web access. These are both "open to the world" via the TCP/IP “ports” - 22 and 80.

As a sysadmin, you need to understand what ports you have open on your servers because each open port is also a potential focus of attacks. You need to be be able to put in place appropriate monitoring and controls.

YOUR TASKS TODAY

  • Secure your web server by using a firewall

INSTRUCTIONS

First we'll look at a couple of ways of determining what ports are open on your server:

  • ss - this, "socket status", is a standard utility - replacing the older netstat
  • nmap - this "port scanner" won't normally be installed by default

There are a wide range of options that can be used with ss, but first try: ss -ltpn

The output lines show which ports are open on which interfaces:

sudo ss -ltp
State   Recv-Q  Send-Q   Local Address:Port     Peer Address:Port  Process
LISTEN  0       4096     127.0.0.53%lo:53        0.0.0.0:*      users:(("systemd-resolve",pid=364,fd=13))
LISTEN  0       128            0.0.0.0:22           0.0.0.0:*      users:(("sshd",pid=625,fd=3))
LISTEN  0       128               [::]:22              [::]:*      users:(("sshd",pid=625,fd=4))
LISTEN  0       511                  *:80                *:*      users:(("apache2",pid=106630,fd=4),("apache2",pid=106629,fd=4),("apache2",pid=106627,fd=4))

The network notation can be a little confusing, but the lines above show ports 80 and 22 open "to the world" on all local IP addresses - and port 53 (DNS) open only on a special local address.

Now install nmap with apt install. This works rather differently, actively probing 1,000 or more ports to check whether they're open. It's most famously used to scan remote machines - please don't - but it's also very handy to check your own configuration, by scanning your server:

$ nmap localhost

Starting Nmap 5.21 ( http://nmap.org ) at 2013-03-17 02:18 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00042s latency).
Not shown: 998 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

Port 22 is providing the ssh service, which is how you're connected, so that will be open. If you have Apache running then port 80/http will also be open. Every open port is an increase in the "attack surface", so it's Best Practice to shut down services that you don't need.

Note that however that "localhost" (127.0.0.1), is the loopback network device. Services "bound" only to this will only be available on this local machine. To see what's actually exposed to others, first use the ip a command to find the IP address of your actual network card, and then nmap that.

Host firewall

The Linux kernel has built-in firewall functionality called "netfilter". We configure and query this via various utilities, the most low-level of which are the iptables command, and the newer nftables. These are powerful, but also complex - so we'll use a more friendly alternative - ufw - the "uncomplicated firewall".

First let's list what rules are in place by typing sudo iptables -L

You will see something like this:

Chain INPUT (policy ACCEPT)
target  prot opt source             destination

Chain FORWARD (policy ACCEPT)
target  prot opt source             destination

Chain OUTPUT (policy ACCEPT)
target  prot opt source             destination

So, essentially no firewalling - any traffic is accepted to anywhere.

Using ufw is very simple. It is available by default in all Ubuntu installations after 8.04 LTS, but if you need to install it:

sudo apt install ufw

Then, to allow SSH, but disallow HTTP we would type:

sudo ufw allow ssh
sudo ufw deny http

BEWARE! Don't forget to explicitly ALLOW ssh, or you’ll lose all contact with your server! If not allowed, the firewall assumes the port is DENIED by default.

And then enable this with:

sudo ufw enable

Typing sudo iptables -L now will list the detailed rules generated by this - one of these should now be:

“DROP       tcp  --  anywhere             anywhere             tcp dpt:http”

The effect of this is that although your server is still running Apache, it's no longer accessible from the "outside" - all incoming traffic to the destination port of http/80 being DROPed. Test for yourself! You will probably want to reverse this with:

sudo ufw allow http
sudo ufw enable

In practice, ensuring that you're not running unnecessary services is often enough protection, and a host-based firewall is unnecessary, but this very much depends on the type of server you are configuring. Regardless, hopefully this session has given you some insight into the concepts.

BTW: For this test/learning server you should allow http/80 access again now, because those access.log files will give you a real feel for what it's like to run a server in a hostile world.

Using non-standard ports

Occasionally it may be reasonable to re-configure a service so that it’s provided on a non-standard port - this is particularly common advice for ssh/22 - and would be done by altering the configuration in /etc/ssh/sshd_config.

Some call this “security by obscurity” - equivalent to moving the keyhole on your front door to an unusual place rather than improving the lock itself, or camouflaging your tank rather than improving its armour - but it does effectively eliminate attacks by opportunistic hackers, which is the main threat for most servers.

But, if you're going to do it, remember all the rules and security tools you already have in place. If you are using AWS, for example, and change the SSH port to 2222, you will need to open that port in the EC2 security group for your instance.

EXTENSION

Even after denying access, it might be useful to know who's been trying to gain entry. Check out these discussions of logging and more complex setups:

RESOURCES

TROUBLESHOOT AND MAKE A SAD SERVER HAPPY!

Practice what you've learned with some challenges at SadServers.com:

Some rights reserved. Check the license terms here

2
Day 8 - The infamous "grep" and other text processors (programming.dev)
 

INTRO

Your server is now running two services: the sshd (Secure Shell Daemon) service that you use to login; and the Apache2 web server. Both of these services are generating logs as you and others access your server - and these are text files which we can analyse using some simple tools.

Plain text files are a key part of "the Unix way" and there are many small "tools" to allow you to easily edit, sort, search and otherwise manipulate them. Today we’ll use grep, cat, more, less, cut, awk and tail to slice and dice your logs.

The grep command is famous for being extremely powerful and handy, but also because its "nerdy" name is typical of Unix/Linux conventions.

YOUR TASKS TODAY

  • Dump out the complete contents of a file with cat like this: cat /var/log/apache2/access.log
  • Use less to open the same file, like this: less /var/log/apache2/access.log - and move up and down through the file with your arrow keys, then use “q” to quit.
  • Again using less, look at a file, but practice confidently moving around using gg, GG and /, n and N (to go to the top of the file, bottom of the file, to search for something and to hop to the next "hit" or back to the previous one)
  • View recent logins and sudo usage by viewing /var/log/auth.log with less
  • Look at just the tail end of the file with tail /var/log/apache2/access.log (yes, there's also a head command!)
  • Follow a log in real-time with: tail -f /var/log/apache2/access.log (while accessing your server’s web page in a browser)
  • You can take the output of one command and "pipe" it in as the input to another by using the | (pipe) symbol
  • So, dump out a file with cat, but pipe that output to grep with a search term - like this: cat /var/log/auth.log | grep "authenticating"
  • Simplify this to: grep "authenticating" /var/log/auth.log
  • Piping allows you to narrow your search, e.g. grep "authenticating" /var/log/auth.log | grep "root"
  • Use the cut command to select out most interesting portions of each line by specifying "-d" (delimiter) and "-f" (field) - like: grep "authenticating" /var/log/auth.log| grep "root"| cut -f 10- -d" " (field 10 onwards, where the delimiter between field is the " " character). This approach can be very useful in extracting useful information from log data.
  • Use the -v option to invert the selection and find attempts to login with other users: grep "authenticating" /var/log/auth.log| grep -v "root"| cut -f 10- -d" "

The output of any command can be "redirected" to a file with the ">" operator. The command: ls -ltr > listing.txt wouldn't list the directory contents to your screen, but instead redirect into the file "listing.txt" (creating that file if it didn't exist, or overwriting the contents if it did).

WHERE'S MY /VAR/LOG/AUTH.LOG?

If you didn't find the file /var/log/auth.log you're probably using a minimal version of Ubuntu (it can be your own local VM or a version in one of the VPS). That minimal image is, well... minimal. It only has the systemd journal available and it didn't come with the old syslog system by default.

But don't worry! To get that back, sudo apt install rsyslog and the file will be created. Just give it a few minutes to populate before working on the lesson.

It also be missing a few of the other programs we use in the challenge, but you can always install them.

POSTING YOUR PROGRESS

Re-run the command to list all the IP's that have unsuccessfully tried to login to your server as root - but this time, use the the ">" operator to redirect it to the file: ~/attackers.txt. You might like to share and compare with others doing the course how heavily you're "under attack"!

EXTENSION

  • See if you can extend your filtering of auth.log to select just the IP addresses, then pipe this to sort, and then further to uniq to get a list of all those IP addresses that have been "auditing" your server security for you.
  • Investigate the awk and sed commands. When you're having difficulty figuring out how to do something with grep and cut, then you may need to step up to using these. Googling for "linux sed tricks" or "awk one liners" will get you many examples.
  • Aim to learn at least one simple useful trick with both awk and sed

RESOURCES

TROUBLESHOOT AND MAKE A SAD SERVER HAPPY!

Practice what you've learned with some challenges at SadServers.com:

Some rights reserved. Check the license terms here

8
Day 7 - The server and its services (programming.dev)
 

INTRO

Today you'll install a common server application - the Apache2 web server - also known as httpd - the "Hyper Text Transport Protocol Daemon"!

If you’re a website professional then you might do things slightly differently, but our focus with this is not on Apache itself, or the website content, but to get a better understanding of:

  • application installation
  • configuration files
  • services
  • logs

YOUR TASKS TODAY

  • Install and run apache, transforming your server into a web server

INSTRUCTIONS

  • Refresh your list of available packages (apps) by: sudo apt update - this takes a moment or two, but ensures that you'll be getting the latest versions.
  • Install Apache from the repository with a simple: sudo apt install apache2
  • Confirm that it’s running by browsing to http://[external IP of your server] - where you should see a confirmation page.
  • Apache is installed as a "service" - a program that starts automatically when the server starts and keeps running whether anyone is logged in or not. Try stopping it with the command: sudo systemctl stop apache2 - check that the webpage goes dead - then re-start it with sudo systemctl start apache2 - and check its status with: systemctl status apache2.
  • As with the vast majority of Linux software, configuration is controlled by files under the /etc directory - check the configuration files under /etc/apache2 especially /etc/apache2/apache2.conf - you can use less to simply view them, or the vim editor to view and edit as you wish.
  • In /etc/apache2/apache2.conf there's the line with the text: "IncludeOptional conf-enabled/*.conf". This tells Apache that the *.conf files in the subdirectory conf-enabled should be merged in with those from /etc/apache2/apache2.conf at load. This approach of lots of small specific config files is common.
  • If you're familiar with configuring web servers, then go crazy, setup some virtual hosts, or add in some mods etc.
  • The location of the default webpage is defined by the DocumentRoot parameter in the file /etc/apache2/sites-enabled/000-default.conf.
  • Use less or vim to view the code of the default page - normally at /var/www/html/index.html. This uses fairly complex modern web design - so you might like to browse to http://165.227.92.20/sample where you'll see a much simpler page. Use View Source in your browser to see the code of this, copy it, and then, in your ssh session sudo vim /var/www/html/index.html to first delete the existing content, then paste in this simple example - and then edit to your own taste. View the result with your workstation browser by again going to http://[external IP of your server]
  • As with most Linux services, Apache keeps its logs under the /var/log directory - look at the logs in /var/log/apache2 - in the access.log file you should be able to see your session from when you browsed to the test page. Notice that there's an overwhelming amount of detail - this is typical, but in a later lesson you'll learn how to filter out just what you want. Notice the error.log file too - hopefully this one will be empty!

Note for AWS/Azure/GCP users

Don't forget to add port 80 to your instance security group to allow inbound traffic to your server.

POSTING YOUR PROGRESS

Practice your text-editing skills, and allow your "classmates" to judge your progress by editing /var/www/html/index.html with vim and posting the URL to access it to the forum. (It doesn’t have to be pretty!)

SECURITY

  • As the sysadmin of this server, responsible for its security, you need to be very aware that you've now increased the "attack surface" of your server. In addition to ssh on port 22, you are now also exposing the apache2 code on port 80. Over time the logs may reveal access from a wide range of visiting search engines, and attackers - and that’s perfectly normal.
  • If you run the commands: sudo apt update, then sudo apt upgrade, and accept the suggested upgrades, then you'll have all the latest security updates, and be secure enough for a test environment - but you should re-run this regularly.

EXTENSION

Read up on:

RESOURCES

TROUBLESHOOT AND MAKE A SAD SERVER HAPPY!

Practice what you've learned with some challenges at SadServers.com:

Some rights reserved. Check the license terms here

8
Day 6 - Editing with "vim" (programming.dev)
 

INTRO

Simple text files are at the heart of Linux, so editing these is a key sysadmin skill. There are a range of simple text editors aimed at beginners. Some more common examples you'll see are nano and pico. These look as if they were written for DOS back in the 1980's - but are pretty easy to "just figure out".

The Real Sysadmintm however, uses vi - this is the editor that's always installed by default - and today you'll get started using it.

Bill Joy wrote Vi back in the mid 1970's - and even the "modern" Vim that we'll concentrate on is over 20 years old, but despite their age, these remain the standard editors on command-line server boxes. Additionally, they have a loyal following among programmers, and even some writers. Vim is actually a contraction of Vi IMproved and is a direct descendant of Vi.

Very often when you type vi, what the system actually starts is vim. To see if this is true of your system type, run:

vi --version

You should see output similar to the following if the vi command is actually symlinked to vim:

user@testbox:~$ vi --version
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Oct 01 2021 01:51:08)
Included patches: 1-2434
Extra patches: 8.2.3402, 8.2.3403, 8.2.3409, 8.2.3428
Modified by team+vim@tracker.debian.org
Compiled by team+vim@tracker.debian.org
...

YOUR TASKS TODAY

  • Run vimtutor
  • Edit a file with vim

WHAT IF I DON'T HAVE VIM INSTALLED?

The rest of this lesson assumes that you have vim installed on your system, which it often is by default. But in some cases it isn't and if you try to run the vim commands below you may get an error like the following:

user@testbox:~$ vim
-bash: vim: command not found

OPTION 1 - ALIAS VIM

One option is to simply substitute vi for any of the vim commands in the instructions below. Vim is reverse compatible with Vi and all of the below exercises should work the same for Vi as well as for Vim. To make things easier on ourselves we can just alias the vim command so that vi runs instead:

echo "alias vim='vi'" >> ~/.bashrc
source ~/.bashrc

OPTION 2 - INSTALL VIM

The other option, and the option that many sysadmins would probably take is to install Vim if it isn't installed already.

To install Vim on Ubuntu using the system package manager, run:

sudo apt install vim

Note: Since Ubuntu Server LTS is the recommended Linux distribution to use for the Linux Upskill Challenge, installing Vim for all of the other various Linux "distros" is outside of the scope of this lesson. The command above "should" work for most Debian-family Linux OS's however, so if you're running Mint, Debian, Pop!_OS, or one of the many other flavors of Ubuntu, give it a try. For Linux distros outside of the Debian-family a few simple web-searches will probably help you find how to install Vim using other Linux's package managers.

THE TWO THINGS YOU NEED TO KNOW

  • There are two "modes" - with very different behaviours
  • Little or nothing onscreen lets you know which mode you're currently in!

The two modes are "normal mode" and "insert mode", and as a beginner, simply remember:

"Press Esc twice or more to return to normal mode"

The "normal mode" is used to input commands, and "insert mode" for writing text - similar to a regular text editor's default behaviour.

INSTRUCTIONS

So, first grab a text file to edit. A copy of /etc/services will do nicely:

cd
pwd
cp -v /etc/services testfile
vim testfile

At this point we have the file on screen, and we are in "normal mode". Unlike nano, however, there’s no onscreen menu and it's not at all obvious how anything works!

Start by pressing Esc once or twice to ensure that we are in normal mode (remember this trick from above), then type :q! and press Enter. This quits without saving any changes - a vital first skill when you don't yet know what you're doing! Now let's go in again and play around, seeing how powerful and dangerous vim is - then again, quit without saving:

vim testfile

Use the keys h j k and l to move around (this is the traditional vi method) then try using the arrow keys - if these work, then feel free to use them - but remember those hjkl keys because one day you may be on a system with just the traditional vi and the arrow keys won't work.

Now play around moving through the file. Then exit with Esc Esc :q! as discussed earlier.

Now that you've mastered that, let's get more advanced.

vim testfile

This time, move down a few lines into the file and press 3 then 3 again, then d and d again - and suddenly 33 lines of the file are deleted!

Why? Well, you are in normal mode and 33dd is a command that says "delete 33 lines". Now, you're still in normal mode, so press u - and you've magically undone the last change you made. Neat huh?

Now you know the three basic tricks for a newbie to vim:

  • Esc Esc always gets you back to "normal mode"
  • From normal mode :q! will always quit without saving anything you've done, and
  • From normal mode u will undo the last action

So, here's some useful, productive things to do:

  • Finding things: From normal mode, type G to get to the bottom of the file, then gg to get to the top. Let's search for references to "sun", type /sun to find the first instance, hit enter, then press n repeatedly to step through all the next occurrences. Now go to the top of the file (gg remember) and try searching for "Apple" or "Microsoft".
  • Cutting and pasting: Go back up to the top of the file (with gg) and look at the first few lines of comments (the ones with "#" as the first character). Play around with cutting some of these out, and pasting them back. To do this simply position the cursor on a line, then (for example), type 11dd to delete 11 lines, then immediately paste them back in by pressing P - and then move down the file a bit and paste the same 11 lines in there again with P
  • Inserting text: Move anywhere in the file and press i to get into "insert mode" (it may show at the bottom of the screen) and start typing - and Esc Esc to get back into normal mode when you're done.
  • Writing your changes to disk: From normal mode type :w to "write" but stay in vim, or :wq to “write and quit”.

This is as much as you ever need to learn about vim - but there's an enormous amount more you could learn if you had the time. Your next step should be to run vimtutor and go through the "official" Vim tutorial. It typically takes around 30 minutes the first time through. To solidify your Vim skills make a habit of running through the vimtutor every day for 1-2 weeks and you should have a solid foundation with the basics.

Note: If you aliased vim to vi for the excercises above, now might be a good time to install vim since this is what provides the vimtutor command. Once you have Vim installed, you can run :help vimtutor from inside of Vim to view the help as well as a few other tips/tricks.

However, if you're serious about becoming a sysadmin, it's important that you commit to using vim (or vi) for all of your editing from now on.

One last thing, you may see reference to is the Vi vs. Emacs debate. This is a long running rivalry for programmers, not system administrators - vi/vim is what you need to learn.

WHY CAN'T I JUST STICK WITH NANO?

  • In many situations as a professional, you'll be working on other people's systems, and they're often very paranoid about stability. You may not have the authority to just "sudo apt install " - even if technically you could.

  • However, vi is always installed on any Unix or Linux box from tiny IoT devices to supercomputer clusters. It is actually required by the Single Unix Specification and POSIX.

  • And frankly it's a shibboleth for Linux pros. As a newbie in an interview it's fine to say you're "only a beginner with vi/vim" - but very risky to say you hate it and can never remember how to exit.

So, it makes sense if you're aiming to do Linux professionally, but if you're just working on your own systems then by all means choose nano or pico etc.

EXTENSION

If you're already familiar with vi / vim then use today's hour to research and test some customisation via your ~/.vimrc file. The link below is specifically for sysadmins:

RESOURCES

Some rights reserved. Check the license terms here

5
Day 5 - More or less... (programming.dev)
 

INTRO

Today we'll end with a bang - with a quick introduction to five different topics. Mastery isn't required today - you'll be getting plenty of practice with all these in the sessions to come!

Don’t be misled by how simplistic some of these commands may seem - they all have hidden depths and many sysadmins will be using several of these every day.

YOUR TASKS TODAY

  • Use tab completion
  • Search in the command history
  • Read a dot file using more and less
  • Change / customize your prompt

Use the links in the Resources section to complete these tasks:

  • Get familiar with using more and less for viewing files, including being able to get to the top or bottom of a file in less, and searching for some text

  • Test how “tab completion” works - this is a handy feature that helps you enter commands correctly. It helps find both the command and also file name parameters (so typing les then hitting “Tab” will complete the command less, but also typing less /etc/serv and pressing “Tab” will complete to less /etc/services. Try typing less /etc/s then pressing “Tab”, and again, to see how the feature handles ambiguity.

  • Now that you've typed in quite a few commands, try pressing the “Up arrow” to scroll back through them. What you should notice is that not only can you see your most recent commands - but even those from the last time you logged in. Now try the history command - this lists out the whole of your cached command history - often 100 or more entries. There are number of clever things that can be done with this. The simplest is to repeat a command - pick one line to repeat (say number 20) and repeat it by typing !20 and pressing “Enter”. Later when you'll be typing long, complex, commands this can be very handy. You can also press Ctrl + r, then start typing any part of the command that you are looking for. You'll see an autocomplete of a past command at your prompt. If you keep typing, you'll get more specific options appear. You can either run it by pressing return, or editing it first by pressing arrows or other movement keys. You can also keep pressing Ctrl + r to see other instances of the same command you used with different options.

  • Look for “hidden” files in your home directory. In Linux the convention is simply that any file starting with a "." character is hidden. So, type cd to return to your "home directory" then ls -l to show what files are there. Now type ls -la or ls -ltra (the "a" is for "all") to show all the files - including those starting with a dot. By far the most common use of "dot files" is to keep personal settings in a home directory. So use your new skills with less to look at the contents of .bashrc , .bash_history and others.

  • Finally, use the nano editor to create a file in your home directory and type up a summary of how the last five days have worked for you.

EXTENSION

We're using bash as our terminal shell for now (it is standard in many distros) but it is not the only one out there. If you want to test out zsh, fish or oh-my-zsh, you will see that there are a few differences and the features are usually the main differentiator. Try that, poke around.

After that, you can go up a notch and try to have several shell sessions open at the same time in the same terminal window with a terminal multiplexer. Try screen - that's a little simpler and maybe too terse in the beginning - or tmux, that have many features and colors. There are so much material out there on "how to customize your tmux", have fun.

RESOURCES

Some rights reserved. Check the license terms here

3
Day 4 - Installing software, exploring the file structure (programming.dev)
 

INTRO

As a sysadmin, one of your key tasks is to install new software as required. You’ll also need to be very familiar with the layout of the standard directories in a Linux system.

You’ll be getting practice in both of these areas in today’s session.

YOUR TASKS TODAY

  • Install a new application from the online repositories
  • Become familiar with some of the standard directories
  • Look at the format and content of some configuration files.

If you've used a smartphone "app store " or "market", then you'll immediately understand the normal installation of Linux software from the standard repositories. As long as we know what the name or description of a package (=app) is, then we can search for it:

 apt search "midnight commander"

This will show a range of matching "packages", and we can then install them with apt install command. So to install package mc (Midnight Commander) on Ubuntu:

 sudo apt install mc

(Unless you're already logged in as the root user you need to use sudo before the installation commands - because an ordinary user is not permitted to install software that could impact a whole server).

Now that you have mc installed, start it by simply typing mc and pressing Enter.

This isn't a "classic" Unix application, but once you get over the retro interface you should find navigation fairly easy, so go looking for these directories:

/root /home /sbin /etc /var/log

...and use the links in the Resources section below to begin to understand how these are used. You can also read the official manual on this hierarchy by typing man hier.

Most key configuration files are kept under /etc and subdirectories of that. These files, and the logs under /var/log are almost invariably simple text files. In the coming days you'll be spending a lot of time with these - but for now simply use F3 to look into their contents.

Some interesting files to look at are: /etc/passwd, /etc/ssh/sshd_config and /var/log/auth.log

Use F3 again to exit from viewing a file.

F10 will exit mc, although you may need to use your mouse to select it.

(On an Apple Mac in Terminal, you may need to use ESC+3 to get F3 and ESC+0 for F10)

Now use apt search to search for and install some more packages: Try searching for “hangman”. You will probably find that an old text-based version is included in a package called bsdgames. Install and play a couple of rounds...

Posting your progress

  • Post your progress, comments and questions to the forum.

EXTENSION

  • Use mc to view /etc/apt/sources.list where the actual locations of the repositories are specified. Often these will be “mirror” sites that are closer to your server than the main Ubuntu servers.
  • Read Repositories - CommandLine for more of the gory details.

RESOURCES

Some rights reserved. Check the license terms here

6
Day 3 - Power trip! (programming.dev)
 

INTRO

You may have been logging in as an ordinary user at your server, yet you're probably aware that root is the power user on a Linux system. This administrative or "superuser" account, is all powerful - and a typo in a command could potentially cripple your server. As a sysadmin you're typically working on systems that are both important and remote, so avoiding such mistakes is A Very Good Idea.

In ancient times, sysadmins used to login as root in production systems, but it’s now common Best Practice to discourage or disallow login directly by root and instead to give specified trusted users the permission to run root-only commands via the sudo command.

YOUR TASKS TODAY

  • Change the password of your sudo user
  • Change the hostname
  • Change the timezone

Check out the demo

LOCAL CHANGES VS GLOBAL CHANGES

Global: programs/environments that any user can use, used across the system. A global change affects all users. Local or By user: programs/environments that a particular user runs, not available to other users. A local change affects only one user.

WHO ARE YOU AND WHAT CAN YOU DO?

There are 3 types of users in a Linux system:

  • root - the powerful superuser that can execute any command at any level in the system. They can do all global changes as well as local changes for any user.
  • sudoers - regular users that are allowed to use sudo, i.e., they can execute commands in one or more levels in the system, can do some or all global changes. It's common to have at least one sudoer that has the same powers as root, but the amount of priviledges other sudoers have can vary.
  • regular users - users that can use the system but can only do local changes, i.e., can only deal with their own files/directories and environment variables.

We will get into more detail about users and their permissions on Day 13 and Day 14.

STOP USING ROOT

If you created a VM with one of the big VPS providers, root is already "disabled" and your default user (ubuntu, azureuser, etc) already has sudo powers.

However, if you really, really want to use root, there are ways to do it in AWS, Azure and GCP. But do it at your own risk!

However, if you created a VM locally or with other VPS providers, it is very likely that you have your root user readily available.

Stop using root. If you followed the guides, you should have created a regular user and adding it to a sudoers group, like this:

adduser snori74

usermod -a -G sudo snori74

Adding a regular user to a group with sudo priviledges is the easiest way to do it, as the sudo group is pretty standard in Ubuntu. But this can also be accomplished by modifying the /etc/sudoers using the command visudo.

Login with this new user from now on.

CHANGE PASSWORD

If you're using a password to login (rather than public key), then now is a good time to ensure that this is very strong and unique - i.e. at least 10 alphanumeric characters - because your server is fully exposed to bots that will be continuously attempting to break in. Use the passwd command to change your password. To do this, think of a new, secure password, then simply type passwd, press “Enter” and give your current password when prompted, then the new one you've chosen, confirm it - and then WRITE IT DOWN somewhere. In a production system of course, public keys and/or two factor authentication would be more appropriate.

A NOTE ON "HARDENING"

Your server is protected by the fact that its security updates are up to date, and that you've set Long Strong Unique passwords - or are using public keys. While exposed to the world, and very likely under continuous attack, it should be perfectly secure. Next week we'll look at how we can view those attacks, but for now it's simply important to state that while it's OK to read up on "SSH hardening", things such as changing the default port and fail2ban are unnecessary and unhelpful when we're trying to learn - and you are perfectly safe without them.

THE POWER OF SUDO

  • Use the links in the "Resources" section below to understand how sudo works
  • Use ls -l to check the permissions of /etc/shadow - notice that only root has any access. Can you use cat, less or nano to view it?
  • This file is where the hashed passwords are kept. It is a prime target for intruders - who aim to grab it and use offline password crackers to discover the passwords.
  • Now try with sudo, e.g. sudo cat /etc/shadow
  • Test running the reboot command, and then via sudo (i.e. sudo reboot)

Once you've reconnected back:

  • Use the uptime command to confirm that your server did actually fully restart
  • Test fully “becoming root” by the command sudo -i. This can be handy if you have a series of commands to do "as root". Note the change to your prompt.
  • Type exit or logout to get back to your own normal “support” login.
  • Use less to view the file /var/log/auth.log, where any use of sudo is logged
  • You could "filter" this by typing: grep "sudo" /var/log/auth.log

Normally invoking the sudo command will ask you to re-confirm your identity with your password. However, this can be changed in the sudoers configuration file so it does NOT prompt for a password.

ADMINISTRATIVE TASKS

We will go into detail of the many things you can do to your server, but here are some examples of simple administrative tasks that require sudo.

If you wish to, you can now rename your server. Traditionally you would do this by editing two files, /etc/hostname and /etc/hosts and then rebooting - but the more modern, and recommended, way is to use the hostnamectl command; like this:

sudo hostnamectl set-hostname mylittlecloudbox

No reboot is required but if you want to see the new name in the prompt, just open a new session with bash (or logoff and login again, same effect).

For a cloud server, you might find that the hostname changes after a reboot. To prevent this, edit /etc/cloud/cloud.cfg and change the "preserve_hostname" line to read:

preserve_hostname: true

You might also consider changing the timezone your server uses. By default this is likely to be UTC (i.e. GMT) - which is pretty appropriate for a worldwide fleet of servers. You could also set it to the zone the server is in, or where you and your headquarters are. For a company this is a decision not to be taken lightly, but for now you can simply change as you please!

First check the current setting with:

timedatectl

Then get a a list of available timezones:

timedatectl list-timezones

And finally select one, like this:

sudo timedatectl set-timezone Australia/Sydney

Confirm:

timedatectl

The major practical effects of this are (1) the timing of scheduled tasks, and (2) the timestamping of the logs files kept under /var/log. If you make a change, there will naturally be a "jump" in the dates and time recorded.

WRAP

As a Linux sysadmin you may be working on client or custom systems where you have little control, and many of these will default to doing everything as root. You need to be able to safely work on such systems - where your only protection is to double check before pressing Enter.

On the other hand, for any systems where you have full control, setting up a "normal" account for yourself (and any co-admins) with permission to run sudo is recommended. While this is standard with Ubuntu, it's also easy to configure with other popular server distros such as Debian, CentOS and RHEL.

EXTENSION

What's difference between "sudo -i" and "sudo -s"?

Both sudo -i and sudo -s are commands that allow a user to obtain root privileges on a Unix-based system. However, they have some differences in how they function.

  • sudo -i stands for "sudo interactive" and it launches a new login shell for the root user. This means that it creates a new environment for the root user with the root user's home directory and shell configuration files. This makes it similar to logging in directly as the root user, and any commands executed from this shell will have the privileges of the root user.
  • sudo -s stands for "sudo shell" and it launches a new shell for the root user, but it does not create a new login shell. This means that it does not change the environment or shell configuration files of the current user. Any commands executed from this shell will have the privileges of the root user, but the environment will still be that of the current user.

In summary, sudo -i is more powerful and creates a new shell with the full environment of the root user, while sudo -s is less powerful and only launches a new shell with the root user's privileges but with the same environment as the current user.

RESOURCES

Some rights reserved. Check the license terms here

view more: ‹ prev next ›