\input texinfo
@setfilename keine.info
@settitle Keine

@titlepage
@title Keine
@subtitle Lightweight time tracking system
@end titlepage

@contents

@ifnottex
@node Top
@top Keine
@end ifnottex

Consume and digest your history.

@menu
* Overview::         Why bother?
* Usage::            How do I track time?
* Patterns::         How do I track lots of time?
* Implementation::   What's happening behind my back?
@end menu

@c ----------------------------------------------------------------------
@c {{{
@node Overview
@chapter Overview

Keine is a lightweight time tracking system.  "Time tracking" meaning
she is something like a stopwatch.  "Lightweight" meaning she is not
that much more complicated than a stopwatch.  Keine is a very simple
solution to a not so simple problem.

This document is meant to be a manual to help others use this tool,
but the manual part is rather sparse since Keine is such a simple tool
(@xref{Usage}).  Most of this manual is more like a diary in
behavioral engineering.

@menu
* Motivation::    The problem
* Solution::      What Keine did and what she couldn't do
* Results::       How the world ends up to be a better place
@end menu

@c {{{
@node Motivation
@section Motivation

Short story: I want to find where all my time went.

Long story: I know where all my time went, I just want to know how bad
it is.

I don't think I am too bad as an engineer, but even I miss milestones
once in a while.  It isn't because I have been playing games at work
(which is not true) or because I work slower than I claim (which I
don't believe).  My theory is that my time are lost to unplanned
things outside my control -- debugging unexpected problems, being
interrupted to answer questions, and so on and so forth.  I want
something to tell me where all the time went for each of the events.
@c }}}

@c {{{
@node Solution
@section Solution

The first step to solving most problems would be to gather data, which
calls for a tool to track time.  It need not be too fancy, something
that automatically timestamps each event and sum the deltas between
them would be sufficient.

Some combination of @code{tcsh}'s command history and a small Perl
script might have solved it, but they still a bit short of what I
wanted.  Thus I set out to write my own tool, with the following
requirements:

@itemize @bullet
@item Track time across multiple tasks -- Keine must be able to sum
times from multiple discontinuous tasks and report how long each task
has taken.
@item Minimal overhead -- Keine must be unobtrusive to the user, it
should be possible to record most events with a few keystrokes.
@item Persistent logs -- Keine must be able to track tasks across
restarts, and be able to deal with timing data gathered over the
period of weeks.
@end itemize

Non-goals (@xref{Patterns}):

@itemize @bullet
@item Track concurrent tasks -- Keine is not expected to track things
like "read mail while waiting for my code to compile", although it
happens regularly.  I expect most people to be "mostly" focused on one
task at any one time, so this is not really worth the effort.
@item Track events automatically -- Keine is not expected to interact
deeply with the system to automatically watch over the user, for
example follow the X-windows focus and record what the user sees.
Features like that would have been useful, but troublesome to
implement portably.  Too clever for my own good.
@item Fine-grained time tracking -- Keine is not expected to track
events that contain a lot of context switches.  For example, finding
who talks the most and interrupts the most during meetings, although
this is a separate problem that I might want to solve later.  Tracking
at that granularity requires an interface with even lower overhead,
not quite something that I want to spend effort on now.
@item Become a tool for micromanagement -- I wrote Keine for my own
use so I think this should be obvious, but just in case if it's not:
Keine is not a tool for managers, unless you manage yourself.  If you
are one of those management types, go buy one of those fancy time card
punching devices instead.
@end itemize
@c }}}

@c {{{
@node Results
@section Results

I have implemented Keine to the requirements specified above, gathered
enough data, presented the results to my manager, and planned my next
quarter's schedule accordingly.  More things are happening on schedule
now, much to my joy and I believe to my manager as well.

What really happened?  I can't really give details on what I do at
work, but it's roughly:

@itemize @bullet
@item 1/3 of my time were actually spent on doing scheduled tasks:
writing code, writing design docs, deploying binaries, etc.
@item The next 1/3 of my time were spent doing something else that
were worth doing, but weren't on the schedule: last minute features,
debugging unexpected problems, reading mail, etc.
@item The remaining 1/3 were lost to annoyances and other unproductive
time: answering interrupts, eating, etc. (I happen to work for a
Company where all 3 meals are free, although most my peers laughed at
the notion of marking the time spent eating as "unproductive").
@end itemize

The solution: plan for the unplanned, and move some of the middle 1/3
time to the scheduled 1/3 time to derive a more realistic schedule.
There will always be some time lost that are just beyond your control,
you just have to plan for it.

On a side note, I found the most productive day last quarter was
actually when I @emph{went on vacation} to do work.  No time lost to
context switches!  Moral of the story -- no amount of time management
is going to better than just having more free time in the first place.
You will need to be workaholic enough to have excessive vacation
accrued, then even more workaholic to use vacation for work.  Results
not typical.
@c }}}
@c }}}

@c ----------------------------------------------------------------------
@c {{{
@node Usage
@chapter Usage

There are only a few things you can do, and few ways to do them.  I
don't plan to add any more commands than these, and if there is
anything I can remove, I would have removed them already.

@menu
* Running::       Run Keine run
* Events::        Put text in
* Report::        Get numbers back
* Shortcuts::     Spare a few keystrokes
* Reload::        Bring back your editor
* Exit::          No more of this logging stuff
@end menu

@c {{{
@node Running
@section Running

Keine operates in one of two modes, either interactively:

@example
./keine @var{log.txt}
@end example

Or in batch mode:

@example
./keine < @var{log.txt}
@end example

In interactive mode, Keine runs in a loop that accepts event entries
and commands, maintaining persistent state in the log file specified
in command line arguments.  In batch mode, Keine loads the log file,
prints out a summary report (@xref{Report}), and exit immediately.
Most of this chapter describes the interactive mode.

After Keine starts, she prints out a message that looks something like
this:

@example
Loaded 28 events from sample_log.txt
Enter ? for help
>
@end example

From there on, you can perform actions described in the rest of this
chapter.  Entering @samp{?} will give a short summary of all possible
commands.
@c }}}

@c {{{
@node Events
@section Events

Entering most text messages at the @samp{>} prompt causes events to be
recorded, for example:

@example
> @kbd{deploy hoge}
2007-10-01 20:12:34 deploy hoge
@end example

This means "I am going to start doing this task now".  Keine echoes
both the timestamp and event to confirm what you have just entered.

Besides this pattern, you would often want to record "I have been
doing this other task since this time."  In that case, you can prefix
the line with a timestamp:

@example
> @kbd{2007-10-31 14:00 halloween party}
2007-10-31 14:00:00 halloween party
> @kbd{15:30 debug alerts}
2007-10-31 15:30:00 debug alerts
@end example
@c }}}

@c {{{
@node Shortcuts
@section Shortcuts

Since we often perform only a small set of tasks throughout the week,
Keine supports a few shortcuts for recording recurring tasks:

@itemize @bullet
@item @kbd{/@var{substring}} - duplicate previous event that contains
@var{substring}.  Log history will be scanned from most recent event
first.
@item @kbd{.@var{prefix}} - duplicate previous event that matches
@var{prefix}.
@item @kbd{@var{key}=@var{value}} - define @var{key} abbreviation as
an abbreviation for @var{value}, and record @var{value} in event log
immediately.  @var{key} must be all one word, @var{value} is
everything from @kbd{=} to end of line.
@item @kbd{#@var{key}=@var{value}} - define @var{key} as an
abbreviation for @var{value}, without recording @var{value} in event
log.
@end itemize

These shortcuts can also be prefixed with a timestamp, follows the
same semantics described in the previous section.

@example
> @kbd{i = interrupted by hage}
2007-11-01 10:00:00 interrupted by hage
> @kbd{10:30 i}
2007-11-01 10:30:00 interrupted by hage
@end example

These concepts came from @samp{!?}, @samp{!} and @samp{alias} commands
found in most shells.  There are no plans make Keine any more
shell-like than these functions.
@c }}}

@c {{{
@node Report
@section Report

Entering the empty string (pressing enter at the prompt) will cause
Keine to print out a report summarizing all event times.  The report
looks like this:

@example
Total time:    39779 (11h 2m)
    9637 (2h 40m): implement hoge  [24.2%]
    6728 (1h 52m): update documentation for  [16.9%]
       3983 (1h 6m): hogehoge  [10.0%]
       2745 (0h 45m): hoge  [6.9%]

    5764 (1h 36m): meeting with  [14.5%]
       3723 (1h 2m): piyo  [9.4%]
       2041 (0h 34m): piyopiyo  [5.1%]

    4282 (1h 11m): interrupted by hage  [10.8%]
    3676 (1h 1m): lunch  [9.2%]
    3209 (0h 53m): design fuga  [8.1%]
    2422 (0h 40m): dinner  [6.1%]
    1645 (0h 27m): review  [4.1%]
       1524 (0h 25m): hogera  [3.8%]
        121 (0h 2m): hogepiyo

    1643 (0h 27m): breakfast  [4.1%]
     773 (0h 12m): mail  [1.9%]
@end example

The events are grouped by common prefixes, then sorted by event
duration.  For example, the report above says that I have spent 1 hour
and 52 minutes updating documentation, 45 minutes of which are for
@var{hoge}, which took up 6.9% of all time tracked by Keine.
@c }}}

@c {{{
@node Reload
@section Reload

Entering @kbd{!} will cause Keine to reload the log file.  This is
useful for synchronization after after making edits in an external
text editor.  @xref{Editing text}.
@c }}}

@c {{{
@node Exit
@section Exit

To exit Keine and stop logging, send EOF (@key{Ctrl+D} on Linux,
@key{Ctrl+Z} followed by enter on Windows).

@key{Ctrl+C} or SIGINT might also work, but you may or may not need to
hit enter after the @key{Ctrl+C} depend on which version of Keine you
are using.
@c }}}
@c }}}

@c ----------------------------------------------------------------------
@c {{{
@node Patterns
@chapter Patterns

Using Keine could change your life.  Really.

More like, to make use of Keine, you may need to change some of your
working habits, hopefully for the better.

@menu
* Recording events::       Making a habit
* Entering time::          Rising and falling edges
* Entering descriptions::  Big-endian history
* Editing text::           Write only memory
@end menu

@c {{{
@node Recording events
@section Recording events

First hurdle to using Keine would be just remembering to run the
program in the first place.  Keine is not obtrusive by design, and
for the first few days you might even forget that she is sitting there
waiting to for events to be entered.  This might be a difficult habit
to acquire depending on your current working style -- in particular,
how many different things you do every day.

If you find that using Keine is too much overhead, you are probably
switching between tasks too often, or perhaps trying to multitask too
much.  In that case, recognize that you are doing mostly one thing at
any one time, and record only the main event, and that should reduce
the overhead significantly.

Another pattern that tends to happen often is not recording an event
because they are perceived to have low costs ("it will take me just
a few seconds to answer this mail!  Just a few minutes!  A few...")
No task is really zero cost in time, if it's going to take you more
than a few seconds to think about whether it's going to take a long
time or not, you should just record the event anyways, and make it a
habit.
@c }}}

@c {{{
@node Entering time
@section Entering time

Using Keine requires the ability to predict the future.  Quite a
difficult task for most people, even if it's only a few minutes into
the future.  Alright, so it's not quite that bad, but you really need
to get into the habit of thinking about "this is what I am going to do
next" as opposed to "this is what I just did."

For a while, I considered the alternative design of recording the
@emph{end times} of each event, as opposed to @emph{start times}.
It's either one or the other, and start time is more intuitive in many
ways (including making manual edits to logs), so there was a bias
toward using start times since the first version.  But recording end
times does have one advantage -- most people do much better at
remembering what they just did, compared to planning what they are
going to do.  Recording end times would therefore make the event log
reflect reality more accurately.

Philosophically, though, I am against this backward looking business.
Just by forcing yourself to think about what you are going to do next
tend to makes it more likely that you will actually do them.  Personally,
in working with this peculiarity of Keine, I found myself more focused
on fewer tasks, and having fewer context switches than before I
started using Keine.  This might be yet another habit that takes a bit
of time to be acquired, but it's one that's worth the effort.

As a workaround, Keine does accept an optional timestamp in event
entry, specifically for the purpose of tracking things you just did.
But if you find yourself using that feature a lot, it may be time to
try a change in working style.
@c }}}

@c {{{
@node Entering descriptions
@section Entering descriptions

Keine's report generation algorithm is hierarchical, meaning you a bit
of additional clustering for free if the events were entered a certain
way.  For example, if your daily activities are mostly code or
documentation related, and you want to summarize how much total time
you have spent on coding versus documentation, those words should be
placed at the beginning of each event description.

@example
> @kbd{implementing hoge}
> @kbd{documenting hoge}
> @kbd{implementing piyo}
> @kbd{documenting piyo}
@end example

This would produce reports that are summarized by activities:
@example
   ####: implementing
      ####: hoge
      ####: piyo
   ####: documenting
      ####: hoge
      ####: piyo
@end example

If you want to summarize tasks by milestones, but don't care too much
on what's done for each milestone, then the activities should be
placed near the end:

@example
> @kbd{hoge code}
> @kbd{hoge docs}
> @kbd{piyo code}
> @kbd{piyo docs}
@end example

This produce reports that are grouped by milestones:

@example
   ####: hoge
      ####: code
      ####: docs
   ####: piyo
      ####: code
      ####: docs
@end example

Depending on which type of report is more useful to you, you should
try to phrase the descriptions accordingly.  Either way, try to
develop a convention and stick to it.  If you had entered totally
random events, the summary report would be much harder and less useful
to read.

Whenever possible, try to use one of the shortcuts to enter repeated
events (@xref{Shortcuts}).  Besides saving a few keystrokes, they
would make the event descriptions more consistent.

Maintaining a consistent convention over a long period of time is
going to take some effort.  Often it's useful to have a separate Perl
script to unify the different log event variations, and have Keine
generate report from the cleaned up version rather than the raw
version (this is why Keine supports reading log files from pipes).
Often I need to write this filter script anyways to anonymize the
events before publishing the summary.
@c }}}

@c {{{
@node Editing text
@section Editing text

Keine has a reload function because there are no functions to edit the
log events in memory (if you haven't noticed yet).  This is
intentional, as soon as I add one editing function or support one
editing model, there would be no end to it.  Keine is not a text
editor.

As a workaround (actually, "intended behavior"), if you need to
correct some previously entered entry, edit the log file in a
@emph{real} text editor, then use the reload function to make the
edits visible to edit.

Of course, it would be much more straightforward if everyone simply
enter all events correctly on the first try.  It's not that hard, but
may take some practice.
@c }}}
@c }}}

@c ----------------------------------------------------------------------
@c {{{
@node Implementation
@chapter Implementation

@menu
* Log format::
* Internals::
* History::
* Keine::
@end menu

@c {{{
@node Log format
@section Log format

Keine keeps all state in a plain text file, which consist of two types
of lines: event definitions and abbreviation definitions (lines that
do not match either format are silently ignored).

@example
2007-02-01 12:34:56 some event
# key = value
@end example

Log file format has been designed to make manual edits easy.  Actually
that's the only design consideration.  No efforts has been made to
make the log files parser friendly, reduce disk footprint, or protect
data integrity.  For all these other bits, consider rotating log files
once every few months, and do your own backups and cleanups on old
logs.  Rotating logs once in a while also tend to make them more
manageable for manual edits.

@menu
* Log events::
* Log timestamps::
* Log abbreviations::
@end menu

@node Log events
@subsection Events

Events are plain text strings.  They are mostly kept as is without
much additional processing, other than stripping some redundant
whitespaces.

Lack of additional processing also means lack of special support for
internationalization.  Assuming your terminal supports something other
than ASCII, Keine will accept those bytes as is, but what happens
after that isn't always well defined.  Shift-JIS has been tested to
work, UTF-8 will probably work.  Regardless of the encoding, if the
event tokens are not separated by half-width whitespaces, hierarchical
summary reports won't work as expected (@xref{Report}).

Keine treats most events equally, except @samp{@{@{@{} and
@samp{@}@}@}}, which marks the beginning and end of a session,
respectively.  @samp{@{@{@{} is mostly a no-op, but useful for finding
the corresponding @samp{@}@}@}} on most editors.  @samp{@}@}@}}
terminates the previous event without starting a new event.  These
were selected because they are language neutral, but also because they
are the default fold markers supported by VIM, so manual edits with
VIM may be slightly easier than some other editors.

@node Log timestamps
@subsection Timestamps

Timestamps are human readable time strings, in @samp{yyyy-mm-dd
HH:MM:SS} format.  One thing worth noting is that they are always in
local time to simplify manual edits.  This means if you are in a place
affected by daylight saving time, and use Keine during those two or
three special hours of the year, Keine might not handle those times
correctly, so don't do that.

Timestamps must be valid, in the sense that they should be
monotonically increasing.  If they are not, Keine would print out line
numbers of the offending lines when they are loaded, so that they can
be corrected or removed.  If they can't even be parsed (that is, they
do not match @samp{yyyy-mm-dd HH:MM:SS} format), those lines are
silently ignored.

@node Log abbreviations
@subsection Abbreviations

Abbreviation definitions consist of (key, value) pairs, on lines
prefixed by @samp{#}.  Keys must be one word (no spaces), but values
can be anything.
@c }}}

@c {{{
@node Internals
@section Internals

Keine is basically a list and a hashtable, the list contains all the
events and the hashtable contains all the abbreviations.  The basic
input loop looks like this:

@itemize @bullet
@item Read a line of text from stdin or file.
@item Decode timestamp, or assign a new timestamp.
@item Define or expand abbreviations as appropriate.
@item Push expanded (timestamp, event) pair to list.
@item Append (timestamp, event) pair to log file, and flush
immediately.
@end itemize

Because log files are written immediately (without buffering), a crash
would not cause any loss in event data.  This also means the writes
are not quite efficient if user wants to enter many long events at
once, but the expected usage pattern is one event every few minutes,
so this shouldn't be a problem.

When summary is requested, Keine does the following:

@itemize @bullet
@item Scan over all events and collect (event, duration) pairs.
@item Group events by prefix, and output prefixes with the longest
duration first.
@item Recursively traverse each prefix group and print out sorted
duration times, until there is only one event in group.
@end itemize

This means the time needed to generate the report will be roughly
proportional to the number of words in event log.  It's fairly easy to
craft an event log that will take a long time to summarize, but those
logs are not expected to occur often in regular use.  I do recommend
(again) to rotate logs once in a while, to reduce report processing
time for these operations.
@c }}}

@c {{{
@node History
@section History

Keine is implemented first in Perl, then again in Perl, and finally in
OCaml.

@menu
* Initial implementation::
* Current implementation::
* Final re-implementation::
* The fine manual::
@end menu

@node Initial implementation
@subsection Initial implementation

A "lightweight stopwatch" really isn't that complicated, basically two
parts:

@itemize @bullet
@item Record events and timestamps
@item Summarize times for each event
@end itemize

For the event entry part, I realized that there are a lot of overlaps
with how shells work: "!" shortcuts to match prefixes, "!?" shortcuts
to match substrings, aliases, etc.  But I don't really want to
reimplement any shell.  And actually I don't really like the "!" and
"?" too much, since they require pressing the shift key (personal
quest of mine -- avoid shift keys as much as I can to reduce stress on
my fingers, although not to the point where I would stop typing
uppercase characters).

Keeping the minimal and easy to implement features from shells, I had
prefix and substring search, and also aliases all implemented in the
first version of Keine.  It was enough to enter most events
efficiently without having to cut and paste a lot.  Most other things
are not there, including something as basic as "undo last entry".
Keine is not a shell, nor is Keine a text editor.

For the time summary part, it was simply summing all the deltas, then
presenting them in sorted order.  For the initial version, there was
also the concept of "interrupts" -- time lost due to annoyances.  This
was to confirm my early theory of where times went by summing the
annoyances separately.  In practice, I realized that the same set of
events always get marked as interrupts (and conversely, the other set
of events never get marked as interrupts).  This lead me to design the
next version, which had a more clever summary routine, but otherwise
treated all events uniformly.

In terms of actual code, this was all done in not too many lines of
Perl (much smaller than this manual, at least).  Part of the
complexity came from treating the interrupts differently, all more
reasons to drop it in the next version.

@node Current implementation
@subsection Current implementation

The interrupt concept was a bit hackish in the first version, so I
rewrote the summary to be more generalized and hierarchical.  This
means by entering events using a certain convention, times lost to
interrupts and other annoyances will be tracked automatically
(@xref{Entering descriptions}).

Another major change was to accept optional timestamps before each
event, so that I can back populate history more easily (@xref{Entering
time}).  This is a less than satisfactory solution to a larger
problem, but it's better than what I used to do before (editing the
log in an external text editor and then reloading the log file).

The underlying code is actually quite different.  Cleaner, I would
say.  Some of the abbreviation semantics broken in the previous
version have been fixed here, and more comprehensive tests were added.
I also made Keine use @code{Term::ReadLine} for input, so I get a bit
of console text editing for free.

Then I realized... it's still all Perl, I can't get an executable for
Windows people.

@node Final re-implementation
@subsection Final re-implementation

For our Perl-less friends, I rewrote Keine in OCaml so that I can get
an executable.  It didn't have to be OCaml, I did it mostly for kicks.
It's tested using the exact same regression test as the Perl version,
so the behavior should be mostly identical.  Known differences:

@itemize @bullet
@item OCaml version is much faster, but you probably won't notice this
unless you are running benchmarks.
@item Definition of whitespace and non-whitespace are different, Perl
version will treat both tabs and spaces as a "whitespace", OCaml
version only distinguishes space versus non-space, and treat tabs like
non-whitespace characters.  This was a side effect due to differences
regular expression conveniences.
@item There is no @code{readline} support.  The built-in
prefix/substring search functions is a good enough substitute for the
most part.  On Windows (NT and above, probably), the console does have
minimal history editing capabilities depending on how it's configured,
so you may get lucky.
@item SIGINT (ctrl+c) is caught, and causes Keine to exit cleanly.
SIGINT is also trapped in the Perl version, but there it doesn't work
since @code{readline} will install its own signal handler on top of
ours (you would have to send SIGINT then press enter on the prompt in
the Perl version, which means you can't force Keine to do a clean exit
remotely).
@item Depending on which versions of Perl and OCaml that was used, you
may get CR-LF or LF at end of lines.
@end itemize

This being OCaml, I can claim the code works entirely as intended
since it compiles, ha!

Really, all implementations of Keine have been tested for quite a
while, and I haven't had to change the code ever since I started
writing this manual.  There are still some potential optimizations and
cleanups left to do, but as far as I care, the code is good enough to
be frozen.

That is to say, I don't really want to spend more time on this time
tracking software.  Keine was developed entirely outside of Company
time, since I thought using Company time to track Company time has a
bad ring to it.  That said, it's amazing that what used to be a short
script written over one weekend ends up to be this executable two
months later (but does more or less the same thing), so I am obviously
not very efficient at using my time outside of work.  Perhaps I need
to gather data on how my weekends are spent as well...

@node The fine manual
@subsection The fine manual

Like most other projects these days, the last thing I do is to
complete the documentation.  It took especially long for Keine, mostly
to cover the minimalistic design and implementation.  There are good
reasons (and excuses) for minimalistic programs, but I don't think
documentation should ever be minimalist.

Of course, it shouldn't be one large blob of unorganized text either,
it should be just enough so that when I tell people RTFM, they can
find what they want quickly.
@c }}}

@c {{{
@node Keine
@section Keine

Keine was named after @uref{http://www.google.co.jp/search?q=%E4%B8%8A%E7%99%BD%E6%B2%A2%E6%85%A7%E9%9F%B3,Kamishirasawa Keine} from @uref{http://www.google.co.jp/search?q=%E6%9D%B1%E6%96%B9%E6%B0%B8%E5%A4%9C%E6%8A%84,Touhou Eiyashou},
for her ability to consume and create history.

Extra was easier than Lunatic, heh.
@c }}}
@c }}}

@bye