STrace Inspector

version: 2.0.0

STrace Inspector is a python library for transforming large amounts of traces of system calls generated by the strace utility into a Directly-Follows-Graph, which can be used to analyse computer programs in terms of requests make to the operating system.

Background: The efficiency of a computer program is often inhibited by contention for system resources, such as the contention for accessing the memory or the storage device. In programs executed by users, requests to access system resources happen through the operating system. The user program communicates with the operating system by issuing system calls. STrace Inspector is a tool that inspects the traces of system calls from one or more arbitrary user program, and facilitates the identification of contention for system resources.

Input: Traces generated by strace

strace is a linux utility that generates traces of system calls of arbitrary commands. You can use strace by prefixing it to the command you want to trace.

Example:

Here is an example of the trace output for the ls command with timestamps included:

strace -tt ls

The first five lines of the output are shown:

15:06:38.137080 execve("/usr/bin/ls", ["ls"], 0x7ffd09937918 /* 43 vars */) = 0
15:06:38.137381 brk(NULL)               = 0x561795398000
15:06:38.137517 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe4066c680) = -1 EINVAL (Invalid argument)
15:06:38.137651 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fbfeb554000
15:06:38.137742 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
15:06:38.137840 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
...

More details on generating the traces of system calls using strace can be found here. The logs of trace files from one or more commands are processed and used as input for STrace Inspector.

What does STrace Inspector do?

STrace Inspector first classifies each trace record into a value, referred to as activity, through a customizable mapping function. The traces are then synthesized into a Directly-Follows-Graph (DFG). The nodes of the DFG represent the system call activities. An edge from one activity to another - say from activity A to activity B - indicates that activity A immediately precedes activity B according to the trace records. This DFG representation is used to identify the contentions for system resources in a program in terms of requests made to the operating system. It is also possible to compare multiple programs or different configurations of the same program, and identify the differences in the system call activities. At the moment, the following inspections are possible:

1. Inspection of I/O activities

In order to focus on the I/O activities, only the I/O related system calls are parsed. Each trace record is mapped to an activity value that indicates the name of the system call and path of the directory that was accessed. The directory path is truncated to at most the top two levels. The nodes are appended with I/O statistics related to the concerned activity. The number on the edges indicate the number of times the corresponding directly-follows relation was observed in the trace records.

Example:

In the following, we show an example of the DFG that depicts the patterns of file accesses from the traces of the commands ls and ls -l. The red nodes and edges indicate the activities and relations that occur exclusively in the ls -l command. The green nodes and edges indicate the activities and relations that occur exclusively in the ls command. The remaining nodes and edges occur in the traces of both the commands.

The DFG initially truncates the file paths to top-two directory levels. Each node of the DFG can be expanded into a separate DFG, which focuses on the next directory levels from that node. For example, the DFG at the bottom focuses only I/O activities whose file path start with the substring /usr/lib.

Citation of the Methodology:

The details on the methodology behind the DFG synthesis and the inspection of I/O traces can be found in this article: Inspection of I/O Operations from System Call Traces using Directly-Follows-Graph. If you are using or improving upon this methodology, please cite this article.

bibtex:

@misc{sankaran2024inspectioniooperationstraces,
      title={Inspection of I/O Operations from System Call Traces using Directly-Follows-Graph}, 
      author={Aravind Sankaran and Ilya Zhukov and Wolfgang Frings and Paolo Bientinesi},
      year={2024},
      eprint={2408.07378},
      archivePrefix={arXiv},
      primaryClass={cs.PF},
      url={https://arxiv.org/abs/2408.07378}, 
}

Installation

This package depends on Python3.6+ and graphviz.

After installing the dependencies, clone this directory and run the following command:

make install

Usage

The workflow is short: set ST_LOG_DIR, run the target program through one of the installed tracer commands, then inspect the generated .st files with the rest of STrace Inspector.

1. Set the log directory

All tracer commands write their output to the directory pointed to by ST_LOG_DIR:

export ST_LOG_DIR=traces

2. Choose the tracer API

After make install, the tracer commands are already available:

• stio: trace I/O-related calls including open, openat, read, write, and lseek
• stiorw: trace only read/write-oriented calls; this is the usual entry point
• stiorwl: like stiorw, but also includes lseek

All three commands share the same interface:

<tracer> [-t TAG] [-i ITER] <command> [args...]

The optional fields are:

• -t: run tag, default x
• -i: iteration id, default 0

3. Trace the target program

For most workflows, use stiorw:

stiorw -t ls-test -i 0 ls -l

This runs ls -l under strace and writes a trace file into ST_LOG_DIR. The output filename is of the form:

stiorw.<tag>.<iter>.<hostname>.<pid>.st

That .st file is the input to the downstream event-log, DFG, and dashboard steps.

4. Continue with the Python CLI

The Python commands installed from pyproject.toml complete the workflow:

• st-prepare-event-log: convert one or more folders of .st files into an HDF5 event log
• st-dfg-stats: build a statistics-colored DFG from one event log
• st-dfg-partitioned: build a partitioned DFG from two event logs
• stx-dashboard: open the web dashboard on a profiles directory

Typical flow:

st-prepare-event-log traces --outfile data/ls_l.h5
st-dfg-stats data/ls_l.h5 --outdir data/profiles --name ls_l --system local --classifier src/strace_inspector/mappings/call_and_top_level_path.py
stx-dashboard data/profiles

Use st-dfg-partitioned instead of st-dfg-stats when the goal is to compare two runs.

5. Run the bundled JUBE experiments

The repository also ships ready-made JUBE experiments under jube/, for example:

• jube/rw_ls/jureca/e1.xml
• jube/ior/jureca/posix_mpiio.xml
• jube/ior/jureca/ssf_fpp.xml

Run them with JUBE:

jube run jube/rw_ls/jureca/e1.xml

or

jube run jube/ior/jureca/posix_mpiio.xml

JUBE creates an artifact directory for the run. Inside that artifact tree you will find the generated profiles directory. Point the dashboard there:

stx-dashboard <artifact-dir>/profiles

This is the quickest way to reproduce the packaged experiments and inspect the resulting profiles.

Software Citation

If you are using the implementation of our methodology through this library, please cite the software.