I'm working on a guide focused on securing Linux servers and I'd like to ask you what your essential hardening techniques and tips are? Your feedback would be greatly appreciated

Not sure if there is any activity on Lemmy. Let's find out.

ITTavern Changelog Week 31

General

Added a SEARCH function:

• only for the titles, fulltext search follows
• added it to the menu

Changed the design sligthly:

• headers are having a light grey background

Notes Update

Update ITTavern.com:

• reworked the beginning and removed some things that might not needed

Blog Updates

Update Getting started with iperf3 - Network Troubleshooting:

• fixed an error: -P instead of -p for parallel streams

Update ICMP echo requests on Linux and Windows - Reference Guide:

• added more tags to make it easier to find

Update SSH - How to use public key authentication on Linux:

• added a new and prefered way to stop the ssh-agent with eval "$(ssh-agent)"

Feedback is welcome!

General

• I deleted my Mastodon account and removed the links from the blog
• remove 'Projects' from the menu and move content to 'Notes'

Notes Update

Update ITTavern.com:

• added a Cyberchef quick access list for various tasks

Blog Updates

Update URL explained - The Fundamentals :

• formatting + spelling mistakes
• domains must not start with a dash (-)
• subdomains CAN contain an underscore (_), but shouldn't

Update Getting started with nmap:

• added the option to check the results every x seconds/minutes with --stats-every 1m / 10s

Update Getting started with tmux:

• add a way to kill the whole session with :kill-session

Project/ Service Updates

Switching secondary domain from itt.sh to brrl.net. The reason for the change is the .sh TLD. Not a big fan and I recommend to block it.

Thank you for the feedback! - The goal is to keep all posts up-to-date and add more content over time.

A deep-dive into the world of URLs. I'll explain the syntax, the functions, some information about domains, and the difference between URL, URI, URN and URC.

Feedback is welcome

cross-posted from: https://infosec.pub/post/306795

I am interested in your ways to identify a bottleneck within a network.

In my case, I've got 2 locations, one in UK, one in Germany. Hardware is Fortigates for FW/routing and switches are Cisco/HPE. Locations are connected through an Ipsec VPN over the internet and all internet connections have at least a bandwidth of 100 Mbps.

The problem occurs as soon as one client in UK tries to download data via SSH from a server in Germany. The max download speed is 10 Mbps and for the duration of the download the whole location in UK has problems accessing resources through the VPN in Germany (Citrix, Exchange, Sharepoint, etc).

I've changed some information for privacy reasons but I'd be interested in your first steps on how to tackle such a problem. Do you have some kind of runbook that you follow? What are common errors that your encounter? (independently from my case too, just in general)

EDIT: Current list

• packet capture on client and server to check for packet loss, latency, etc. - if packets dropped, check intermediate devices
• check utilization of intermediate devices (CPU, RAM, etc)
• check throughput with different tools (ipfer3, nc, etc) and protocols (TCP, UDP, etc) and compare
• check if traffic shaper/ QoS are in place
• check ports intermediate devices for port speed mismatch
• MTU/MSS mismatch
• is the internet connection affected too, or just traffic through the VPN
• Ipsec configuration
• turn off security function of FW temporary and check if it is still reproducible
• traceroute from A to B, any latency spikes?
• check RTT, RWND, MSS/MTU, TTL via pcap, on the transferring client itself and reference client, without and while an active data transfer

Prob not related but noteworthy:

• check I/O of server and client

I'll keep this list updated and appreciate further tips.

Update I had to postpone the session and will do the stress test on Monday or Tuesday evening. I'll update you as soon as I have the results.

Update2 So, I'll try to keep it short.

First iperf3 over TCP run (UK < DE) with same FW rules let me reproduce the problem. Max speed 10 Mbps, and DE < UK even slower, down to 1-2 Mbps. Pattern of the test implies an unreliable connection (short up to 30 Mbts, then 0, and so on). Traceroute shows same hops in both directions, no latency spikes, all good.

BUT ICMP and iperf3 over UDP runs show a packet loss of min 10% and up to 30% in both directions! Multiple speed tests to endpoints over the internet (UK>Internet) showed a download of 80 Mbts andupload of like 30 Mbts, which indicates a problem with the IPSec tunnel.

Some smaller things we've tried without any positive effect:

• routing changes
• disabling all security features for affected rule set
• removed traffic shaper
• Port speed/duplex negotiations are looking good
• and some other things that I already forgot

Things we prepared:

• We have opened some tickets at our ISPs to let them check it on their site > waiting for response
• Set up smokeping to ping all provider/public/gw/ipsec endpoinrts/host IPs and see where packets could be dropped (server located in DE)
• Planned a new session with an Fortigate expert to look in-depth into the IPSec configuration.

Need to do:

• look through all packet captures (takes some time)
• MSS/MTU missmatches / DF flags
• further iperf3 tests with smaller/larger packet
• double check ipsec configuration
• QoS on Switches

I wish I had more time. I'll keep you updated

Update3 Most likely the last big update.

So, the actual infrastructure is a little bit more complex than I've described in this post, so nobody could have suggested tips for this case.

We think that we have found the problem, but we couldn't implement the fix yet since it requires some downtime, and I was on a business trip. We've got multiple locations in the UK that are connected to a third party (MLPS) where their internet breakout points are too. We've now got multiple IPSec tunnels that terminate on the same FW in Germany. The problem is that the third-party FW uses the same IP AND port for all IPSec tunnels too, which most likely causes all the issues. In short: only use one tunnel or change the GW on the German side.

Don't ask me why, please! - It is a cluster fuck, and the goal is to fix it in the future. One site had a large flat /16 network not long ago.

I might share a final update when we get the fix implemented.

I am interested in your ways to identify a bottleneck within a network.

In my case, I've got 2 locations, one in UK, one in Germany. Hardware is Fortigates for FW/routing and switches are Cisco/HPE. Locations are connected through an Ipsec VPN over the internet and all internet connections have at least a bandwidth of 100 Mbps.

The problem occurs as soon as one client in UK tries to download data via SSH from a server in Germany. The max download speed is 10 Mbps and for the duration of the download the whole location in UK has problems accessing resources through the VPN in Germany (Citrix, Exchange, Sharepoint, etc).

I've changed some information for privacy reasons but I'd be interested in your first steps on how to tackle such a problem. Do you have some kind of runbook that you follow? What are common errors that your encounter? (independently from my case too, just in general)

EDIT: Current list

• packet capture on client and server to check for packet loss, latency, etc. - if packets dropped, check intermediate devices
• check utilization of intermediate devices (CPU, RAM, etc)
• check throughput with different tools (ipfer3, nc, etc) and protocols (TCP, UDP, etc) and compare
• check if traffic shaper/ QoS are in place
• check ports intermediate devices for port speed mismatch
• MTU/MSS mismatch
• is the internet connection affected too, or just traffic through the VPN
• Ipsec configuration
• turn off security function of FW temporary and check if it is still reproducible
• traceroute from A to B, any latency spikes?
• check RTT, RWND, MSS/MTU, TTL via pcap, on the transferring client itself and reference client, without and while an active data transfer

Prob not related but noteworthy:

• check I/O of server and client

I'll keep this list updated and appreciate further tips.

Update I had to postpone the session and will do the stress test on Monday or Tuesday evening. I'll update you as soon as I have the results.

Update2 So, I'll try to keep it short.

First iperf3 over TCP run (UK < DE) with same FW rules let me reproduce the problem. Max speed 10 Mbps, and DE < UK even slower, down to 1-2 Mbps. Pattern of the test implies an unreliable connection (short up to 30 Mbts, then 0, and so on). Traceroute shows same hops in both directions, no latency spikes, all good.

BUT ICMP and iperf3 over UDP runs show a packet loss of min 10% and up to 30% in both directions! Multiple speed tests to endpoints over the internet (UK>Internet) showed a download of 80 Mbts andupload of like 30 Mbts, which indicates a problem with the IPSec tunnel.

Some smaller things we've tried without any positive effect:

• routing changes
• disabling all security features for affected rule set
• removed traffic shaper
• Port speed/duplex negotiations are looking good
• and some other things that I already forgot

Things we prepared:

• We have opened some tickets at our ISPs to let them check it on their site > waiting for response
• Set up smokeping to ping all provider/public/gw/ipsec endpoinrts/host IPs and see where packets could be dropped (server located in DE)
• Planned a new session with an Fortigate expert to look in-depth into the IPSec configuration.

Need to do:

• look through all packet captures (takes some time)
• MSS/MTU missmatches / DF flags
• further iperf3 tests with smaller/larger packet
• double check ipsec configuration
• QoS on Switches

I wish I had more time. I'll keep you updated

Update3 Most likely the last big update.

So, the actual infrastructure is a little bit more complex than I've described in this post, so nobody could have suggested tips for this case.

We think that we have found the problem, but we couldn't implement the fix yet since it requires some downtime, and I was on a business trip. We've got multiple locations in the UK that are connected to a third party (MLPS) where their internet breakout points are too. We've now got multiple IPSec tunnels that terminate on the same FW in Germany. The problem is that the third-party FW uses the same IP AND port for all IPSec tunnels too, which most likely causes all the issues. In short: only use one tunnel or change the GW on the German side.

Don't ask me why, please! - It is a cluster fuck, and the goal is to fix it in the future. One site had a large flat /16 network not long ago.

I might share a final update when we get the fix implemented.

General

Since I keep all blog posts up-to-date and try to add more content over time, I thought it would be a great idea to share the changelog weekly over here at Lemmy.

I'll probably publish it on Sunday every week, but I wanted to test the format beforehand.

Notes update

Update ITTavern.com:

• none

Blog updates

Update Getting started with nmap:

• added the -d flag to increase verbosity
• added HTML anchors
• added link to nmap scripts guide

Update Getting started with nmap scripts:

• added a section on how I use nmap scripts for

Update SSH Troubleshooting Guide:

• added method to spin up a second SSHD instance for troubleshooting / logging for single clients

Update Getting started with GNU screen - Beginners Guide:

• added command to create new/ reattach last session (screen -d -RR)

Update Backup Guide - how to secure crucial data:

• added HTML anchors