ptee 0.4.1

Description:

ptee 0.4.1

Introduction
ptee (for “Progress Tee”) is a console utility that builds upon the basic
functionality provided by the standard Unix tee command. It accepts lines
of text from a running command (such as an invocation of make) and displays
them to the console such that consecutive less-important lines are overwritten
in-place, providing feedback regarding the progress of the overall operation
without allowing the more-important lines (such as compiler warnings and errors)
to scroll away and be overlooked. In addition, as with standard tee, a copy
of the text from stdin may optionally be written verbatim to one or more output
files.
These less-important lines are called “context” lines, as they provide context
leading up to the important lines; the more-important lines are called “regular”
lines. Each new context overwrites previous context lines in-place on the
console, forming a “status” line that stays put without scrolling. When a
regular line appears, the text composing the status line is kept (i.e., scrolled
up) to provide context for the regular text.
For example, suppose an invocation of make generates the following output:
$ make
gcc -c -Wall -W -o file1.o file1.c
gcc -c -Wall -W -o file2.o file2.c
file2.c:1:12: warning: ‘x’ defined but not used [-Wunused-variable]
gcc -c -Wall -W -o file3.o file3.c
gcc -c -Wall -W -o file4.o file4.c
gcc -c -Wall -W -o file5.o file5.c
gcc -o app file1.o file2.o file3.o file4.o file5.o
The compiler invocation lines (gcc -c ...) become uninteresting as soon as
the next line shows up, unless there are warnings or errors associated with that
invocation. With ptee, you can supply a regular expression to match these
“context” lines to allow them to overwrite each other on the console. Lines not
matching the regular expression will be displayed on a line of their own (along
with the previous context line, if any). In the above example, the output would
ultimately look like this:
$ make 2>&1 | ptee --regex '^gcc'
gcc -c -Wall -W -o file2.o file2.c
file2.c:1:12: warning: ‘x’ defined but not used [-Wunused-variable]
The 2>&1 in the above invocation redirects stderr to the same location as
stdout, so that both stdout and stderr are piped into ptee. This is needed
because gcc’s diagnostic messages go to stderr by default.
During the run of make, each line of status will be written on top of the
previous line of status, providing continuous feedback while keeping the
interesting lines from scrolling too far off the screen.


Context Levels
Context lines may have an associated level, indicating their position in a
hierarchy. Levels are integers, starting at zero. When a context line of
level N is detected, the status line will be built of the most recent lines of
context from levels zero through N, concatenated into a single status line.
This can be useful for retaining bigger-picture context information while
more detailed context information is coming in.
For example, consider a build system invoked with a script named buildall,
which generates the following output:
$ ./buildall
x86:
Building component1:
[compile] file1.o
[compile] file2.o
file2.c:1:12: warning: ‘x’ defined but not used [-Wunused-variable]
[link] component1
Building component2:
[compile] file3.o
[link] component2
x86_64:
Building component3:
[compile] file4.o
[link] component3
In this build output, some components are being built, first for the x86
platform, then for the x86_64 platform. This output has three levels of
context hierarchy:

Level 0: the platform (x86: or x86_64);
Level 1: the component (e.g., Building component1);
Level 2: the build step (e.g., [compile] source1, [link] component3).

Consider the following shell script to invoke buildall:
#!/bin/sh

./buildall 2>&1 | ptee build.out \
--level-regex 0 '^(x86|x86_64):' \
--level-regex 1 '^Building ' \
--level-regex 2 '^\[.*\]'
The filename build.out is passed to ptee such that a verbatim copy of
the build output will be recorded in the file build.out for possible future
analysis. When running ./ba, the uninteresting context lines are stripped
away, leaving only the regular lines (the warning message, in this case) and the
context lines at each level leading up to each regular line:
$ ./ba
x86:
Building component1:
[compile] file2.c
file2.c:1:12: warning: ‘x’ defined but not used [-Wunused-variable]
More text actually goes to the console for status and feedback, but it is
overwritten by writing a carriage return (\r) instead of a newline (\n).
Below is the actual output, post-processed to show the carriage returns and the
subsequent overwriting taking place:
x86:\r
x86: Building component1:\r
x86: Building component1: [compile] file1.o\r
x86: Building component1: [compile] file2.o\r
\r
x86:
Building component1:
[compile] file2.o
file2.c:1:12: warning: ‘x’ defined but not used [-Wunused-variable]
x86: Building component1: [link] component1\r
x86: Building component2: \r
x86: Building component2: [compile] file3.o\r
x86: Building component2: [link] component2\r
x86_64: \r
x86_64: Building component3:\r
x86_64: Building component3: [compile] file4.o\r
x86_64: Building component3: [link] component3\r
Notice that the status line that appears briefly during compilation of file1.c
contains all three levels of context line, and that the first two levels of
context are the same when subsequently compiling file2.c, so that
bigger-picture context persists longer in the status line:
x86: Building component1: [compile] file1.o\r
x86: Building component1: [compile] file2.o\r


Heading lines
In addition to context lines, ptee supports the notion of “heading” line.
These lines do not contribute to the status line; instead, they are printed
as-is on the console. Unlike regular lines, however, no context lines are
printed before a heading line. This can be useful for long lines that would be
awkward if prepended to the status line. Consider a second example with the
following modified output:
$ ./buildall2
------------------------------ x86 ------------------------------
Building component1:
[compile] file1.o
[compile] file2.o
file2.c:1:12: warning: ‘x’ defined but not used [-Wunused-variable]
[link] component1
Building component2:
[compile] file3.o
[link] component2
------------------------------ x86_64 ---------------------------
Building component3:
[compile] file4.o
[link] component3
The banner lines starting with ------ are too long to conveniently prepend
to the status line. Instead, the ba2 script treats them as headings:
#!/bin/sh

./buildall2 2>&1 | ptee build.out \
--heading-regex '^-----' \
--level-regex 1 '^Building ' \
--level-regex 2 '^\[.*\]'
Leading to this output:
$ ./ba2
------------------------------ x86 ------------------------------
Building component1:
[compile] file2.o
file2.c:1:12: warning: ‘x’ defined but not used [-Wunused-variable]
------------------------------ x86_64 ---------------------------


Skipping lines
Sometimes input contains lines that should be skipped entirely, rather than
being treated as status lines. An example might include spurious compiler
warnings that can’t easily be suppressed. The switch
--skip-regex COUNT SKIP_REGEX provides a way to skip one or more lines that
match a given pattern. For example, given the following input:
[compile] file1.o
system-header.h:999:18: warning: this is a spurious message
in argument 2 of function `badly_written(x, y)`
--------------------------------------------^
[compile] file2.o
To skip the three lines of spurious warning, use this invocation:
ptee --skip-regex 3 system-header.h:999:18:
This effectively transforms the input to:
[compile] file1.o
[compile] file2.o


Stripping overwritten lines
When writing to the console, status lines are continuously written and
overwritten to provide feedback on overall progress. When the operation
completes, only the important lines of text remain. But if this console output
were redirected to a file or piped into another program, the illusion of the
status lines being overwritten would fall apart, because all of the status lines
would be still be present in the output. Therefore, when not writing to the
console, ptee strips out any status lines that would be overwritten. This
default behavior can be overridden via the --strip option (to force the
status to be removed even when writing to a console) and the --no-strip
option (to retain the status lines even when not writing to a console). As an
example, the post-processed output shown above was generated something like
this:
./buildall 2>&1 | ptee [switches] --no-strip | perl -0777 -pe 's/\r/\\r\n/g'


Partial lines
In general, ptee processes whole lines of text. But sometimes the input stream
may pause after a partial line, such as when a program displays a prompt to the
user and pauses for a response. To allow the user to see such partial lines,
ptee by default will wait an amount of time controlled by the
–partial-line-timeout switch; if the input stream stalls for longer than this
amount of time, the partial input will be displayed without further processing,
and all future input up to the next newline will be immediately displayed.
Setting the timeout value to zero disables the timeout feature.


Text encoding option
By default, text is assumed to be in UTF-8 format on stdin and stdout. This
may be overridden using the --encoding option, e.g., for a hypothetical
program that generates latin1 text:
generate-latin1-text | ptee --encoding latin1 --regex '<regular expression>'
See ptee --help for more information.