strace Syscall Tracer

What it is

strace is a diagnostic, debugging and instructional userspace utility for Linux that traces and logs system calls made by a process and signals received by a process. It’s your go-to for understanding how a program interacts with the operating system kernel.

Installation

Linux:

sudo apt update && sudo apt install strace
# or
sudo yum install strace
# or
sudo dnf install strace

macOS: strace is not natively available on macOS. The closest equivalent is dtruss, which uses DTrace.

sudo dtruss <command>

Windows: strace is not available on Windows. Similar functionality might be achieved with tools like Process Monitor from Sysinternals.

Core Concepts

• System Calls (Syscalls): These are the fundamental interface between a user program and the Linux kernel. When a program needs to perform an operation like reading a file, writing to the network, or allocating memory, it makes a system call. strace intercepts and displays these calls.
• Signals: Signals are a form of inter-process communication used to notify a process of an event. strace can show when a process receives or sends signals.
• File Descriptors: These are small non-negative integers that the kernel uses to identify open files, sockets, pipes, and other I/O resources. strace shows which file descriptors are used in system calls.

Commands / Usage

Tracing a New Process

Start a new command and trace its system calls.

strace ls -l /tmp

Trace ls -l /tmp and show all its system calls.

Attaching to a Running Process

Trace an already running process by its Process ID (PID).

sudo strace -p 12345

Attach to the process with PID 12345 and trace its system calls.

Filtering System Calls

Trace only specific system calls.

strace -e trace=open,read,write ls /tmp

Trace only open, read, and write system calls for ls /tmp.

Trace all file-related system calls.

strace -e trace=file ls /tmp

This is a shortcut for tracing common file operations like open, read, write, close, stat, etc.

Trace all network-related system calls.

strace -e trace=network ls /tmp

This is a shortcut for tracing network operations like socket, connect, sendto, recvfrom, etc.

Exclude specific system calls.

strace -e 'trace=!futex' <command>

Trace all system calls except futex.

Displaying Signal Information

Show signals received by the process.

strace -p 12345 -e signal=all

Show all signals received by the process with PID 12345.

Show only specific signals.

strace -e signal=SIGINT,SIGTERM <command>

Trace only SIGINT and SIGTERM signals for a new command.

Following Child Processes

Trace system calls made by child processes created via fork(), vfork(), or clone().

strace -f <command>

Trace ls and all its child processes.

Output Formatting

Print timestamps for each system call.

strace -t <command>

Trace ls and show the time of day for each syscall.

Print relative timestamps for each system call.

strace -r <command>

Trace ls and show the time elapsed since the previous syscall.

Print absolute timestamps for each system call.

strace -tt <command>

Trace ls and show the time of day with microseconds.

Print time elapsed since the beginning of the trace.

strace -ttt <command>

Trace ls and show seconds and microseconds since the start of strace.

Print timestamps and time elapsed since the start.

strace -r -ttt <command>

Trace ls and show relative and absolute time differences.

Simplify output by decoding strings.

strace -s 1024 <command>

Trace ls and display up to 1024 characters of strings.

Show system call return values.

strace -v <command>

Trace ls and show verbose output, including values that are normally hidden. (Note: -v behavior can vary; often used for more detailed output of structures).

Saving Output

Save the trace output to a file.

strace -o trace.log <command>

Trace ls and save all output to trace.log.

Other Useful Flags

Print process start time.

strace -c <command>

Count system calls, time spent in each, and errors for ls. This provides a summary rather than a live trace.

Follow network connections.

strace -yy <command>

Trace ls and print paths associated with file descriptors. Useful for network sockets.

Print file descriptor paths.

strace -yy -e trace=open,openat <command>

Trace open and openat calls and show the paths for the file descriptors created.

Interpret string arguments as paths.

strace -P /etc/passwd <command>

Trace command but only show system calls that access /etc/passwd.

Common Patterns

Debugging a "hang"

When a program seems stuck, attach strace to see what system call it’s currently blocked on.

sudo strace -p <PID_OF_HANGING_PROCESS>

Investigating "permission denied" errors

Trace open and openat calls to see which files the program is trying to access and why it’s failing.

strace -e trace=open,openat <command> 2>&1 | grep '= -1 ENOENT'

This shows open or openat calls that resulted in "No such file or directory".

strace -e trace=open,openat <command> 2>&1 | grep '= -1 EACCES'

This shows open or openat calls that resulted in "Permission denied".

Seeing what files a program opens

strace -e trace=open,openat <command>

A quick way to see all files a program attempts to open.

Debugging network issues

Trace network-related syscalls to understand connection attempts, data sends/receives.

strace -e trace=network <command>

Analyzing configuration file access

See which configuration files a program reads.

strace -e trace=open,openat,read <command> 2>&1 | grep '/etc/'

This filters the trace to show open, openat, and read calls, then greps for lines containing /etc/, indicating configuration file access.

Tracing a shell script

Trace all commands executed by a shell script.

strace -f -o script_trace.log bash your_script.sh

The -f flag is crucial here to follow all child processes spawned by the script.

Combining with grep for specific events

Find all write calls to stdout (file descriptor 1).

strace -e trace=write <command> 2>&1 | grep 'write(1, '

Gotchas

• Performance Impact: strace significantly slows down the traced process because it intercepts every system call. Use it judiciously, especially on performance-sensitive applications.
• Output Volume: strace can produce a massive amount of output. Use filtering (-e), output redirection (-o), and tools like grep to manage it.
• Permissions: To trace a process owned by another user, or a system process, you typically need root privileges (sudo strace).
• Strace vs. Ltrace: strace traces system calls, while ltrace traces library calls. They are complementary tools. If you’re seeing issues within a library function that doesn’t seem to involve a system call, ltrace might be more appropriate.
• Decoding Complex Structures: Some system call arguments are complex data structures (e.g., struct sockaddr). strace attempts to decode them, but the output can sometimes be cryptic. The -v flag might offer more detail, but often requires manual interpretation or comparison with system headers.
• Signal Handling: By default, strace might interfere with how signals are delivered to the traced process. Use -e signal= to explicitly control signal tracing if needed.
• Pipes and FIFOs: When tracing processes that communicate heavily via pipes or FIFOs, the output can become interleaved and harder to follow, especially without -f.