					jw, 16.5.2006
					jw, Fri Jun  2 17:05:19 CEST 2006
					jw, Wed Feb  7 08:22:52 CET 2007

IOANA -- a process I/O analyzer
===============================

Abstract
--------
This presents a tool to monitor behaviour of processes in a system.
The focus is on tracking of data-flow, so that dependencies between
inbound and outbound data streams (or files) can be described.
This can be used to analyze inter-process communication.

An optional regression module compares a current analysis with that of 
previous sessions. Repeated patterns can be extracted to predict the 
behaviour of the system or to signal unexpected changes in the data-flow.


Existing tools
--------------
strace(1), ltrace(1), truss(1 at&t), ltrace(1), ptrace(2), dtrace(1 solaris),
tcpdump(1), lsof(1), SYS::Ptrace(3). inotify(3), audit(7)

Ioana does not replace or reimplement features of the existing monitoring
tools. It uses one or more of the above tools as backends. 
The output of these tools is captured and digested into ioana reports.

Ioana needs to know more than some of thes could do.
E.g. inotify generates events without pid. When a file is written, it is written 
by one single process. Ioana needs to know which files were previously opened
by this particular process.
Audit and inotify do not capture interprocess communication.
If cpp pipes its output into cc1, ioana needs to see this.

ptrace-based tools (strace, ltrace, truss,dtrace, SYS::Ptrace) can generate all sorts 
of events, but do not necessarily see an initial state. Open files in a process, 
first actions after fork. Such (static) information can be provided by lsof. 

Inotify and audit process events asynchronuously. In a busy system, there is no
guarantee that all events are seen. Ptrace-based tools serialize themselves with the
processes they monitor, slowing them down if needed. Thus ptrace-based tools guarantee
that no events are lost.



Suggested usage
---------------
a) List which header files may contribute to a certain binary object by
monitoring the compiler toolchain when the binary object is built. This uses
the Asumption that the compiler toolchain works on a need to know basis.
If it reads files without using them, they can easily appear in the listing 
as false positives. False negatives are unlikely, nevertheless.

b) Debug the cups printing pipeline. Traditionally cups is a one way 
pipeliine with very little feedback from its backends.
ioana can help analysing where the pipeline breaks in case of an error.

For a) and b) the concept of a dependency differs slightly.
E.g. in a) the binary does not depend on the compiler frontend.
The compiler frontend /usr/bin/cc constructs command lines for its children
cc1, as, ld and calls these. But the compiler frontend never generates any
output by itself. When discussing derivative works, objects need to be present
in the same adress space. We asume that after an exec() system call, no part
of the former text segment is present.

In b) we count all wrapper processes as part of the dependency chain.
A process that constructs a non-trivial command line for its child may
influence the result of the filter pipeline. 


Backend design
--------------
At start of ioana, its horizon is defined by
the process (or processes) that it attaches to. A fildescriptor leading to a
plain file has the filename (as detected by lsof) at its horizon. A
filedescriptor leading to an unnamed pipe (as created before fork()) has the
pipe-id (as detected by lsof) as its horizon. Named pipes or sockets are
considered as filenames in this context.

Strace can be run as a backend. It can be run by ioana as one instance per traced process, or it can be run once, tracing multiple processes. 
One strace per process has the advantage of a simpler output parser. It avoids
"<... continued ...>" system calls. This may also be more robust against
abnormal child termination. The drawback is, that when a process forks, 
we cannot attach the child until after its pid appeared in the (possibly 
buffered) output stream. This is a problem, as the child may run untraced for a
while and thus creates loss of dependancy information.

System call serialisation is important. Imagine this scenario: 
Process A writes a file, then Process B reads the file while Process C
truncates the file to size 0 and writes to it.
If B reads before C is done, then it has a dependency on process A, if it reads
after C is done, it depends on C.
With multiple straces attached to one process each, we could only try to
determine the order by comparing timestamps, which is unreliable on a multi-CPU
system. With a single strace monitoring all, we have a better chance for the kernel to 
correctly serialize syscalls for us. At least, no attached process can perform syscalls 
while strace is processing a syscall.

We run strace -f -F mode, so that it automatically attaches as soon as possible to the new
child. This has the sideeffect, that the backend parser for strace must handle suspended
and resumend system calls. Imagine this scenario:

333 open("foo", O_RDONLY	<unfinished...>
444 rename("bar", "foo")    = 0
333 <open resumed> , )      = 3

This is the kernel's way to not let us know, whether fd=3 points to "foo" or "bar".
Ioana will assume "foo", as it assembles and parses the resumed systemcall after the rename.
While processing the open syscall, ioana always tries to stat the file. If it finds the inode of
"bar" where it expects "foo", it will exit with an error message.

