amazon/amazon-ecs-network-sidecar repository overview

⁠Amazon ECS Network Sidecar (V2)

⁠What does it do?

The purpose of this container is to be deployed in the same network namespace as another docker container and to provide a user with easy access to various network related troubleshooting tools. In accordance with Docker design principles, application containers most often are lightweight by design and do not often have easy access to valuable troubleshooting tools that would have to be installed in the container after gaining access to it in the first place.

This tool can easily be deployed on any Linux host that has a docker daemon running on it and be attached to the network namespace of an existing container (or even the host's network namespace), to perform tasks such as capturing TCP packets, running a traceroute and more, all without the need of accessing the container we're troubleshooting directly or installing additional software.

To explain how this works, network namespaces provide isolation of the system resources associated with networking. Docker uses network and other type of namespaces (pid, mount, user..etc) to create an isolated environment for each container. Everything from interfaces, routes, and IPs is completely isolated within the network namespace of the container and by deploying this tool into the same network namespace as another container any network information that is being collected by the tools provided by 'amazon-ecs-network-sidecar' will apply to the container we are attaching to.

⁠How to deploy it?

⁠With host access:

When SSH access to the underlying host is possible, you can simply launch the container directly into the network namespace as a running container. First find the CONTAINER ID of the running container you'd like to troubleshoot with docker ps.

Example:

CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS        PORTS     NAMES
c282e7783f42   nginx     "/docker-entrypoint.…"   2 seconds ago   Up 1 second   80/tcp    nostalgic_kare

Now start the sidecar container with the following command syntax (fill in your own container ID):

docker run -it --rm --net container:c282e7783f42 public.ecr.aws/aws-se/amazon-ecs-network-sidecar /usr/bin/zsh

Likewise you can also start the Amazon ECS Network Sidecar container in the same network namespace as the host to use any of the installed tooling to troubleshoot host related network issues:

docker run -it --rm --net public.ecr.aws/aws-se/amazon-ecs-network-sidecar /usr/bin/zsh

⁠Deploy in ECS Fargate

As running docker run commands in this situation is not possible you will need to include the Amazon ECS Network Sidecar container in the same task definition as the container you are trying to troubleshoot, similar to the following example (container definition):

{
  "name" : "sidecar",
  "image" : "public.ecr.aws/aws-se/amazon-ecs-network-sidecar",
  "essential": false,
  "environment": [
    {
      "name" : "SIDECAR_RUN_MODE",
      "value" : "SLEEPER"
    }
  ],
  "linuxParameters": {
    "initProcessEnabled": true
  }
}

Then, using Amazon ECS Exec⁠ you will be able to connect to the running container. If your cluster does not support Amazon ECS Exec, additional run modes are supported both for backwards compatibility with V1 of this image.

Example:

aws ecs execute-command --cluster cluster-name --task task-id \
    --container sidecar --interactive --command /usr/bin/zsh

⁠Available Tools/Packages

• tcpdump - A popular packet analyzer, used to capture network packets for an attached network interface.
• traceroute - Diagnostic tool for displaying possible routes (paths) and measuring transit delays of packets across an Internet Protocol (IP) network.
• telnet - An application protocol used to provide a bidirectional interactive text-oriented communication facility using a virtual terminal connection.
• nmap - A port scanning tool that can be used to verify if a listening port is being blocked by a target host.
• bind-utils - Provides various DNS related commands, such as dig & nslookup, useful in determining DNS resolution.
• iperf3 - Iperf is a tool for network performance measurement and tuning.
• net-tools - A collection of base networking utilities for Linux, such as ifconfig and netstat.
• iproute2 - Another collection of base networking utilties for Linux using the ip command.
• dhcping - A utility for checking DHCP-servers using unicast packages.
• ethtool - A utility for displaying and modifying some parameters of network interface controllers and their device drivers.
• mtr - MTR provides the functionality of both the ping and traceroute commands and is useful to catch intermittent issues.
• iftop - System monitor tool that produces a frequently updated list of network connections and bandwidth usage.
• ctop - Ctop provides a concise and condensed overview of real-time metrics for multiple containers.
• curl - A command-line tool for transferring data using various network protocols.

⁠Configuration Environment Variables

• SIDECAR_RUN_MODE - The run mode the container will run in. Options are: SSH_KEY and SLEEPER. If passing SSH_KEY make sure to also pass the SIDECAR_KEY_STR environment variable.
• SIDECAR_KEY_STR - The public key line(s) to be injected into the authorized_keys file before starting SSH. It is strongly recommended that you pass this using a secret.

⁠Migration Notes

This package has been designed to be a "like for like" replacement in all aspsects possible compared to the Amazon ECS Network Sidecar V1 container. Note that there are a few differences:

• The package runs on Amazon Linux 2022
• unicornscan is no longer supported. This is because we were unable to location an Amazon Linux 2022
• The default shell has been changed to /usr/bin/zsh and uses the purer prompt by default.
• While the default mode is to run the container with OpenSSH password authentication this is strongly discourage and will be fully deprecated in a future release. Switch to key based authentication or ECS Exec going forward.
• While the arm64v8 tag is included and will track latest this is no longer required as the image is built for both ARM 64 and X86 64 environments.

⁠Example Use Cases

⁠tcpdump

A common use-case for tcpdump would be to capture packets for the network interface used by your container. The Amazon ECS Network Sidecar tool can be used regardless of the network mode being used by your container and will function in Fargate as well. The captured information will be for the network namespace the tool was launched into and will perform the packet capture for the interface (eth0) for that namespace, which will include the application container that the sidecar is attached to. To perform a full packet capture (often requested by support), the following command may be used:

tcpdump -i eth0 -s 65535 -w capture.pcap

The resulting .pcap file can be provided to Support for analysis, or you can review the captured packets using the popular open source Wireshark tool. The AWS CLI tool is included to allow to you export any packet captures to S3. It is therefore recommend that you ensure that you provide your Task Role the required permissions to expect to S3.

⁠traceroute

Traceroute can be used to identify possible routes taken between your container and a destination location (IP or domain based) and provides latency information for round trip times of each 'hop' along the path, to identify possible causes behind network latency. Example command and output:

❯ ~ root@ip-172-31-25-31 traceroute -n aws.amazon.com
traceroute to aws.amazon.com (13.35.139.75), 30 hops max, 60 byte packets
 1  54.252.0.203  8.367 ms * 54.252.0.193  1.605 ms
 2  100.65.16.0  17.396 ms 100.65.17.32  17.934 ms 100.65.20.224  12.919 ms
 3  100.66.8.110  17.246 ms * 100.66.8.44  16.655 ms
 4  100.66.10.174  17.655 ms 100.66.11.110  17.448 ms 100.66.10.0  11.691 ms
 5  241.0.11.71  0.182 ms 241.0.11.76  0.200 ms 241.0.11.75  0.192 ms
 6  240.1.188.17  0.188 ms 240.1.188.24  0.224 ms 240.1.188.22  0.245 ms
 7  242.4.119.137  3.045 ms 242.4.118.17  1.543 ms 242.4.119.145  1.280 ms
 8  52.95.38.38  1.489 ms 52.95.38.34  1.698 ms 52.95.38.104  2.227 ms
 9  52.95.38.127  2.576 ms 52.95.38.187  3.079 ms 52.95.36.126  23.167 ms
10  52.95.36.147  1.254 ms 52.95.36.11  1.424 ms 52.95.36.147  1.288 ms
11  100.64.50.253  12.775 ms  23.156 ms  22.794 ms
12  100.64.50.43  22.556 ms 100.64.50.59  19.862 ms 100.64.50.31  16.544 ms
13  100.64.50.254  1.487 ms  1.538 ms  1.835 ms
14  100.93.4.74  9.234 ms 100.93.4.10  2.150 ms 100.93.4.74  8.524 ms
15  100.93.4.7  5.987 ms 100.93.4.67  7.100 ms 100.93.4.69  7.249 ms
16  13.35.139.75  1.211 ms  2.202 ms  1.221 ms

⁠nmap

nmap will allow you to check the status of open ports by sending a query and checking the targets' response. This tools primary function for troubleshooting is to check if a specific port for a target destination and determine if traffic to this port is allowed or blocked/filtered. Example syntax and output:

❯ ~ root@ip-172-31-25-31 nmap -p 443 google.com
Starting Nmap 7.80 ( https://nmap.org ) at 2022-12-12 09:26 UTC
Nmap scan report for google.com (172.217.167.78)
Host is up (0.0013s latency).
Other addresses for google.com (not scanned): 2404:6800:4006:80a::200e
rDNS record for 172.217.167.78: syd15s06-in-f14.1e100.net

PORT    STATE SERVICE
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

⁠bind-utils

This package provides several useful tools that you may wish to familiarize yourself with, but one of the most common uses for troubleshooting includes using the dig tool to determine how your container resolves various DNS queries. Example syntax and output:

❯ ~ root@ip-172-31-25-31 dig amazon.com +short
52.94.236.248
54.239.28.85
205.251.242.103

❯ ~ root@ip-172-31-25-31 dig amazon.com MX +short
5 amazon-smtp.amazon.com.

❯ ~ root@ip-172-31-25-31 dig amazon.com NS +short
ns3.p31.dynect.net.
ns1.p31.dynect.net.
ns2.p31.dynect.net.
ns4.p31.dynect.net.
pdns6.ultradns.co.uk.
pdns1.ultradns.net.

⁠ctop

Ctop is an open source tool that can attach to the docker socket to monitor multiple containers. Due to the fact that this tool requires the docker socket to be bind mounted to properly function and receive container information from the Docker daemon, this tool, unlike the other tools provided by the Amazon ECS Network Sidecar can not be leveraged for Fargate tasks and will only function for containers launched on EC2 or a host which can be accessed via SSH. To use this tool, please use the example syntax:

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock public.ecr.aws/aws-se/amazon-ecs-network-sidecar ctop