IPCDump – Tool For Tracing Interprocess Communication (IPC) On Linux
ipcdump is a tool for tracing interprocess communication (IPC) on Linux. It covers most of the common IPC mechanisms — pipes, fifos, signals, unix sockets, loopback-based networking, and pseudoterminals. It’s a useful tool for debugging multi-process applications, and it’s also a simple way to understand how the different moving parts in your system communicate with one another. ipcdump can trace both the metadata and the contents of this communication, and it’s particularly well-suited to tracing IPC between short-lived processes, which can be difficult using traditional debugging tools, like strace or gdb. It also has some basic filtering capabilities to help you sift through large quantities of event s. Most of the information ipcdump collects comes from BPF hooks placed on kprobes and tracepoints at key functions in the kernel, although it also fills in some bookkeeping from the /proc filesystem. To this end ipcdump makes heavy use of gobpf, which provides golang binding for the bcc framework.
Requirements & Usage
- golang >= 1.15.6
Tested operating systems and kernels
|Ubuntu 18.04 LTS||Ubuntu 20.04 LTS|
*Requires building bcc from source
- Install golang
snap install go --classic
or directly from golang website
- Install BCC using iovisor’s instructions depending on the operation system you chose (usually the newer versions will require building from source)
git clone https://github.com/guardicore/IPCDump
Usage of ./ipcdump:
max number of bytes to dump per event, or 0 for complete event (may be large). meaningful only if -x is specified.
filter by destination comm (can be specified more than once)
-L do not output lost event information
filter by comm (either source or destination, can be specified more than once)
filter by source comm (can be specified more than once)
exit after <count> events
filter by destination pid (can be specified more than once)
<text|json> output format (default is text) (default "text")
filter by pid (either source or destination, can be specified more than once)
filter by source pid (can be specified more than once)
filter by type (can be specif ied more than once).
possible values: a|all k|signal u|unix ud|unix-dgram us|unix-stream t|pty lo|loopback lt|loopback-tcp lu|loopback-udp p|pipe
-x dump IPC bytes where relevant (rather than just event details).
Run as root:
# dump all ipc on the system
# dump signals sent between any two processes
./ipcdump -t kill
# dump loopback TCP connection metadata to or from pid 1337
./ipcdump -t loopback-tcp -p 1337
# dump unix socket IPC metadata and contents from Xorg
./ipcdump -t unix -x -S Xorg
# dump json-formatted pipe i/o metadata and first 64 bytes of contents
./ipcdump -t pipe -x -B 64 -f json
- Support for pipes and FIFOs
- Loopback IPC
- Signals (regular and realtime)
- Unix streams and datagrams
- Pseudoterminal-based IPC
- Event filtering based on process PID or name
- Human-friendly or JSON-formatted output
ipcdump is built of a series of collectors, each of which is in charge of a particular type of IPC event. For example,
In practice, all of the collectors are built using bpf hooks attached to kprobes and tracepoints. Their implementations are entirely separate, though — there’s no particular reason to assume our information will always come from bpf. That said, the different collectors do have to share a single bpf module, because there’s some common code that they need to share. To this end, we share a single BpfBuilder (which is essentially a wrapper around concatenating strings of bcc code) and each collector registers its own code with that builder. The full bcc script is then loaded with gobpf, and each module places the hooks it needs.
There are currently two kinds of bookkeeping that are shared between IPC collectors:
SocketIdentifier (internal/collection/sock_id.go)— maps between kernel
struct sock*and the processes that use them.
CommIdentifier (internal/collection/comm_id.go)— maps between pid numbers and the corresponding process name (
/proc/<pid>/comm). The bookkeeping done in each of these is particularly important for short-lived processes; while this information can be filled out later in usermode by parsing
/proc, often the relevant process will have disappeared by the time the event hits the handler. That said, we do sometimes fill in information from
/proc. This happens mostly for processes that existed before ipcdump was run; we won’t catch events like process naming in this case.
CommIdentifiersort of try and abstract this duality between bcc code and
/procparsing behind a single API, although it’s not super-clean. By the way, in super-new versions of Linux (5.8), bpf iterators can entirely replace this bookkeeping, although for backwards compatibility we should probably stick to the hooks-and-procfs paradigm for now.
Event output is done through the common
EmitIpcEvent() function, which takes a standard event format (source process, dest process, metadata key-value pairs, and contents) and outputs it in a unified format. To save event bandwidth, collectors typically don’t output IPC contents if the
-x flag isn’t specified. This is done with some fancy preprocessing magic in
Please do! Check out TODO for the really important stuff. Most of the early work on ipcdump will probably involve making adjustments for different kernel versions and symbols.