Keine
*****

Consume and digest your history.

Overview
********

Keine is a lightweight time tracking system.  "Time tracking" meaning she is something like a stopwatch.  "Lightweight" meaning she is not that much more complicated than a stopwatch.  Keine is a very simple solution to a not so simple problem.

This document is meant to be a manual to help others use this tool, but the manual part is rather sparse since Keine is such a simple tool (see Usage).  Most of this manual is more like a diary in behavioral engineering.

Motivation
==========

Short story: I want to find where all my time went.

Long story: I know where all my time went, I just want to know how bad it is.

I don't think I am too bad as an engineer, but even I miss milestones once in a while.  It isn't because I have been playing games at work (which is not true) or because I work slower than I claim (which I don't believe).  My theory is that my time are lost to unplanned things outside my control -- debugging unexpected problems, being interrupted to answer questions, and so on and so forth.  I want something to tell me where all the time went for each of the events.

Solution
========

The first step to solving most problems would be to gather data, which calls for a tool to track time.  It need not be too fancy, something that automatically timestamps each event and sum the deltas between them would be sufficient.

Some combination of tcsh's command history and a small Perl script might have solved it, but they still a bit short of what I wanted.  Thus I set out to write my own tool, with the following requirements:

  * Track time across multiple tasks -- Keine must be able to sum times from multiple discontinuous tasks and report how long each task has taken.

  * Minimal overhead -- Keine must be unobtrusive to the user, it should be possible to record most events with a few keystrokes.

  * Persistent logs -- Keine must be able to track tasks across restarts, and be able to deal with timing data gathered over the period of weeks.

Non-goals (see Patterns):

  * Track concurrent tasks -- Keine is not expected to track things like "read mail while waiting for my code to compile", although it happens regularly.  I expect most people to be "mostly" focused on one task at any one time, so this is not really worth the effort.

  * Track events automatically -- Keine is not expected to interact deeply with the system to automatically watch over the user, for example follow the X-windows focus and record what the user sees. Features like that would have been useful, but troublesome to implement portably.  Too clever for my own good.

  * Fine-grained time tracking -- Keine is not expected to track events that contain a lot of context switches.  For example, finding who talks the most and interrupts the most during meetings, although this is a separate problem that I might want to solve later.  Tracking at that granularity requires an interface with even lower overhead, not quite something that I want to spend effort on now.

  * Become a tool for micromanagement -- I wrote Keine for my own use so I think this should be obvious, but just in case if it's not: Keine is not a tool for managers, unless you manage yourself.  If you are one of those management types, go buy one of those fancy time card punching devices instead.

Results
=======

I have implemented Keine to the requirements specified above, gathered enough data, presented the results to my manager, and planned my next quarter's schedule accordingly.  More things are happening on schedule now, much to my joy and I believe to my manager as well.

What really happened?  I can't really give details on what I do at work, but it's roughly:

  * 1/3 of my time were actually spent on doing scheduled tasks: writing code, writing design docs, deploying binaries, etc.

  * The next 1/3 of my time were spent doing something else that were worth doing, but weren't on the schedule: last minute features, debugging unexpected problems, reading mail, etc.

  * The remaining 1/3 were lost to annoyances and other unproductive time: answering interrupts, eating, etc. (I happen to work for a Company where all 3 meals are free, although most my peers laughed at the notion of marking the time spent eating as "unproductive").

The solution: plan for the unplanned, and move some of the middle 1/3 time to the scheduled 1/3 time to derive a more realistic schedule. There will always be some time lost that are just beyond your control, you just have to plan for it.

On a side note, I found the most productive day last quarter was actually when I went on vacation to do work.  No time lost to context switches!  Moral of the story -- no amount of time management is going to better than just having more free time in the first place. You will need to be workaholic enough to have excessive vacation accrued, then even more workaholic to use vacation for work.  Results not typical.

Usage
*****

There are only a few things you can do, and few ways to do them.  I don't plan to add any more commands than these, and if there is anything I can remove, I would have removed them already.

Running
=======

Keine operates in one of two modes, either interactively:

     ./keine LOG.TXT

Or in batch mode:

     ./keine < LOG.TXT

In interactive mode, Keine runs in a loop that accepts event entries and commands, maintaining persistent state in the log file specified in command line arguments.  In batch mode, Keine loads the log file, prints out a summary report (see Report), and exit immediately. Most of this chapter describes the interactive mode.

After Keine starts, she prints out a message that looks something like this:

     Loaded 28 events from sample_log.txt
     Enter ? for help
     >

From there on, you can perform actions described in the rest of this chapter.  Entering ? will give a short summary of all possible commands.

Events
======

Entering most text messages at the > prompt causes events to be recorded, for example:

     > deploy hoge
     2007-10-01 20:12:34 deploy hoge

This means "I am going to start doing this task now".  Keine echoes both the timestamp and event to confirm what you have just entered.

Besides this pattern, you would often want to record "I have been doing this other task since this time."  In that case, you can prefix the line with a timestamp:

     > 2007-10-31 14:00 halloween party
     2007-10-31 14:00:00 halloween party
     > 15:30 debug alerts
     2007-10-31 15:30:00 debug alerts

Shortcuts
=========

Since we often perform only a small set of tasks throughout the week, Keine supports a few shortcuts for recording recurring tasks:

  * /SUBSTRING - duplicate previous event that contains SUBSTRING.  Log history will be scanned from most recent event first.

  * .PREFIX - duplicate previous event that matches PREFIX.

  * KEY=VALUE - define KEY abbreviation as an abbreviation for VALUE, and record VALUE in event log immediately.  KEY must be all one word, VALUE is everything from = to end of line.

  * #KEY=VALUE - define KEY as an abbreviation for VALUE, without recording VALUE in event log.

These shortcuts can also be prefixed with a timestamp, follows the same semantics described in the previous section.

     > i = interrupted by hage
     2007-11-01 10:00:00 interrupted by hage
     > 10:30 i
     2007-11-01 10:30:00 interrupted by hage

These concepts came from !?, ! and alias commands found in most shells.  There are no plans make Keine any more shell-like than these functions.

Report
======

Entering the empty string (pressing enter at the prompt) will cause Keine to print out a report summarizing all event times.  The report looks like this:

     Total time:    39779 (11h 2m)
         9637 (2h 40m): implement hoge  [24.2%]
         6728 (1h 52m): update documentation for  [16.9%]
            3983 (1h 6m): hogehoge  [10.0%]
            2745 (0h 45m): hoge  [6.9%]

         5764 (1h 36m): meeting with  [14.5%]
            3723 (1h 2m): piyo  [9.4%]
            2041 (0h 34m): piyopiyo  [5.1%]

         4282 (1h 11m): interrupted by hage  [10.8%]
         3676 (1h 1m): lunch  [9.2%]
         3209 (0h 53m): design fuga  [8.1%]
         2422 (0h 40m): dinner  [6.1%]
         1645 (0h 27m): review  [4.1%]
            1524 (0h 25m): hogera  [3.8%]
             121 (0h 2m): hogepiyo

         1643 (0h 27m): breakfast  [4.1%]
          773 (0h 12m): mail  [1.9%]

The events are grouped by common prefixes, then sorted by event duration.  For example, the report above says that I have spent 1 hour and 52 minutes updating documentation, 45 minutes of which are for HOGE, which took up 6.9% of all time tracked by Keine.

Reload
======

Entering ! will cause Keine to reload the log file.  This is useful for synchronization after after making edits in an external text editor.  See Editing text.

Exit
====

To exit Keine and stop logging, send EOF (Ctrl+D on Linux, Ctrl+Z followed by enter on Windows).

Ctrl+C or SIGINT might also work, but you may or may not need to hit enter after the Ctrl+C depend on which version of Keine you are using.

Patterns
********

Using Keine could change your life.  Really.

More like, to make use of Keine, you may need to change some of your working habits, hopefully for the better.

Recording events
================

First hurdle to using Keine would be just remembering to run the program in the first place.  Keine is not obtrusive by design, and for the first few days you might even forget that she is sitting there waiting to for events to be entered.  This might be a difficult habit to acquire depending on your current working style -- in particular, how many different things you do every day.

If you find that using Keine is too much overhead, you are probably switching between tasks too often, or perhaps trying to multitask too much.  In that case, recognize that you are doing mostly one thing at any one time, and record only the main event, and that should reduce the overhead significantly.

Another pattern that tends to happen often is not recording an event because they are perceived to have low costs ("it will take me just a few seconds to answer this mail!  Just a few minutes!  A few...") No task is really zero cost in time, if it's going to take you more than a few seconds to think about whether it's going to take a long time or not, you should just record the event anyways, and make it a habit.

Entering time
=============

Using Keine requires the ability to predict the future.  Quite a difficult task for most people, even if it's only a few minutes into the future.  Alright, so it's not quite that bad, but you really need to get into the habit of thinking about "this is what I am going to do next" as opposed to "this is what I just did."

For a while, I considered the alternative design of recording the end times of each event, as opposed to start times. It's either one or the other, and start time is more intuitive in many ways (including making manual edits to logs), so there was a bias toward using start times since the first version.  But recording end times does have one advantage -- most people do much better at remembering what they just did, compared to planning what they are going to do.  Recording end times would therefore make the event log reflect reality more accurately.

Philosophically, though, I am against this backward looking business. Just by forcing yourself to think about what you are going to do next tend to makes it more likely that you will actually do them.  Personally, in working with this peculiarity of Keine, I found myself more focused on fewer tasks, and having fewer context switches than before I started using Keine.  This might be yet another habit that takes a bit of time to be acquired, but it's one that's worth the effort.

As a workaround, Keine does accept an optional timestamp in event entry, specifically for the purpose of tracking things you just did. But if you find yourself using that feature a lot, it may be time to try a change in working style.

Entering descriptions
=====================

Keine's report generation algorithm is hierarchical, meaning you a bit of additional clustering for free if the events were entered a certain way.  For example, if your daily activities are mostly code or documentation related, and you want to summarize how much total time you have spent on coding versus documentation, those words should be placed at the beginning of each event description.

     > implementing hoge
     > documenting hoge
     > implementing piyo
     > documenting piyo

This would produce reports that are summarized by activities:

        ####: implementing
           ####: hoge
           ####: piyo
        ####: documenting
           ####: hoge
           ####: piyo

If you want to summarize tasks by milestones, but don't care too much on what's done for each milestone, then the activities should be placed near the end:

     > hoge code
     > hoge docs
     > piyo code
     > piyo docs

This produce reports that are grouped by milestones:

        ####: hoge
           ####: code
           ####: docs
        ####: piyo
           ####: code
           ####: docs

Depending on which type of report is more useful to you, you should try to phrase the descriptions accordingly.  Either way, try to develop a convention and stick to it.  If you had entered totally random events, the summary report would be much harder and less useful to read.

Whenever possible, try to use one of the shortcuts to enter repeated events (see Shortcuts).  Besides saving a few keystrokes, they would make the event descriptions more consistent.

Maintaining a consistent convention over a long period of time is going to take some effort.  Often it's useful to have a separate Perl script to unify the different log event variations, and have Keine generate report from the cleaned up version rather than the raw version (this is why Keine supports reading log files from pipes). Often I need to write this filter script anyways to anonymize the events before publishing the summary.

Editing text
============

Keine has a reload function because there are no functions to edit the log events in memory (if you haven't noticed yet).  This is intentional, as soon as I add one editing function or support one editing model, there would be no end to it.  Keine is not a text editor.

As a workaround (actually, "intended behavior"), if you need to correct some previously entered entry, edit the log file in a real text editor, then use the reload function to make the edits visible to edit.

Of course, it would be much more straightforward if everyone simply enter all events correctly on the first try.  It's not that hard, but may take some practice.

Implementation
**************

Log format
==========

Keine keeps all state in a plain text file, which consist of two types of lines: event definitions and abbreviation definitions (lines that do not match either format are silently ignored).

     2007-02-01 12:34:56 some event
     # key = value

Log file format has been designed to make manual edits easy.  Actually that's the only design consideration.  No efforts has been made to make the log files parser friendly, reduce disk footprint, or protect data integrity.  For all these other bits, consider rotating log files once every few months, and do your own backups and cleanups on old logs.  Rotating logs once in a while also tend to make them more manageable for manual edits.

Events
------

Events are plain text strings.  They are mostly kept as is without much additional processing, other than stripping some redundant whitespaces.

Lack of additional processing also means lack of special support for internationalization.  Assuming your terminal supports something other than ASCII, Keine will accept those bytes as is, but what happens after that isn't always well defined.  Shift-JIS has been tested to work, UTF-8 will probably work.  Regardless of the encoding, if the event tokens are not separated by half-width whitespaces, hierarchical summary reports won't work as expected (see Report).

Keine treats most events equally, except {{{ and }}}, which marks the beginning and end of a session, respectively.  {{{ is mostly a no-op, but useful for finding the corresponding }}} on most editors.  }}} terminates the previous event without starting a new event.  These were selected because they are language neutral, but also because they are the default fold markers supported by VIM, so manual edits with VIM may be slightly easier than some other editors.

Timestamps
----------

Timestamps are human readable time strings, in yyyy-mm-dd HH:MM:SS format.  One thing worth noting is that they are always in local time to simplify manual edits.  This means if you are in a place affected by daylight saving time, and use Keine during those two or three special hours of the year, Keine might not handle those times correctly, so don't do that.

Timestamps must be valid, in the sense that they should be monotonically increasing.  If they are not, Keine would print out line numbers of the offending lines when they are loaded, so that they can be corrected or removed.  If they can't even be parsed (that is, they do not match yyyy-mm-dd HH:MM:SS format), those lines are silently ignored.

Abbreviations
-------------

Abbreviation definitions consist of (key, value) pairs, on lines prefixed by #.  Keys must be one word (no spaces), but values can be anything.

Internals
=========

Keine is basically a list and a hashtable, the list contains all the events and the hashtable contains all the abbreviations.  The basic input loop looks like this:

  * Read a line of text from stdin or file.

  * Decode timestamp, or assign a new timestamp.

  * Define or expand abbreviations as appropriate.

  * Push expanded (timestamp, event) pair to list.

  * Append (timestamp, event) pair to log file, and flush immediately.

Because log files are written immediately (without buffering), a crash would not cause any loss in event data.  This also means the writes are not quite efficient if user wants to enter many long events at once, but the expected usage pattern is one event every few minutes, so this shouldn't be a problem.

When summary is requested, Keine does the following:

  * Scan over all events and collect (event, duration) pairs.

  * Group events by prefix, and output prefixes with the longest duration first.

  * Recursively traverse each prefix group and print out sorted duration times, until there is only one event in group.

This means the time needed to generate the report will be roughly proportional to the number of words in event log.  It's fairly easy to craft an event log that will take a long time to summarize, but those logs are not expected to occur often in regular use.  I do recommend (again) to rotate logs once in a while, to reduce report processing time for these operations.

History
=======

Keine is implemented first in Perl, then again in Perl, and finally in OCaml.

Initial implementation
----------------------

A "lightweight stopwatch" really isn't that complicated, basically two parts:

  * Record events and timestamps

  * Summarize times for each event

For the event entry part, I realized that there are a lot of overlaps with how shells work: "!" shortcuts to match prefixes, "!?" shortcuts to match substrings, aliases, etc.  But I don't really want to reimplement any shell.  And actually I don't really like the "!" and "?" too much, since they require pressing the shift key (personal quest of mine -- avoid shift keys as much as I can to reduce stress on my fingers, although not to the point where I would stop typing uppercase characters).

Keeping the minimal and easy to implement features from shells, I had prefix and substring search, and also aliases all implemented in the first version of Keine.  It was enough to enter most events efficiently without having to cut and paste a lot.  Most other things are not there, including something as basic as "undo last entry". Keine is not a shell, nor is Keine a text editor.

For the time summary part, it was simply summing all the deltas, then presenting them in sorted order.  For the initial version, there was also the concept of "interrupts" -- time lost due to annoyances.  This was to confirm my early theory of where times went by summing the annoyances separately.  In practice, I realized that the same set of events always get marked as interrupts (and conversely, the other set of events never get marked as interrupts).  This lead me to design the next version, which had a more clever summary routine, but otherwise treated all events uniformly.

In terms of actual code, this was all done in not too many lines of Perl (much smaller than this manual, at least).  Part of the complexity came from treating the interrupts differently, all more reasons to drop it in the next version.

Current implementation
----------------------

The interrupt concept was a bit hackish in the first version, so I rewrote the summary to be more generalized and hierarchical.  This means by entering events using a certain convention, times lost to interrupts and other annoyances will be tracked automatically (see Entering descriptions).

Another major change was to accept optional timestamps before each event, so that I can back populate history more easily (see Entering time).  This is a less than satisfactory solution to a larger problem, but it's better than what I used to do before (editing the log in an external text editor and then reloading the log file).

The underlying code is actually quite different.  Cleaner, I would say.  Some of the abbreviation semantics broken in the previous version have been fixed here, and more comprehensive tests were added. I also made Keine use Term::ReadLine for input, so I get a bit of console text editing for free.

Then I realized... it's still all Perl, I can't get an executable for Windows people.

Final re-implementation
-----------------------

For our Perl-less friends, I rewrote Keine in OCaml so that I can get an executable.  It didn't have to be OCaml, I did it mostly for kicks. It's tested using the exact same regression test as the Perl version, so the behavior should be mostly identical.  Known differences:

  * OCaml version is much faster, but you probably won't notice this unless you are running benchmarks.

  * Definition of whitespace and non-whitespace are different, Perl version will treat both tabs and spaces as a "whitespace", OCaml version only distinguishes space versus non-space, and treat tabs like non-whitespace characters.  This was a side effect due to differences regular expression conveniences.

  * There is no readline support.  The built-in prefix/substring search functions is a good enough substitute for the most part.  On Windows (NT and above, probably), the console does have minimal history editing capabilities depending on how it's configured, so you may get lucky.

  * SIGINT (ctrl+c) is caught, and causes Keine to exit cleanly. SIGINT is also trapped in the Perl version, but there it doesn't work since readline will install its own signal handler on top of ours (you would have to send SIGINT then press enter on the prompt in the Perl version, which means you can't force Keine to do a clean exit remotely).

  * Depending on which versions of Perl and OCaml that was used, you may get CR-LF or LF at end of lines.

This being OCaml, I can claim the code works entirely as intended since it compiles, ha!

Really, all implementations of Keine have been tested for quite a while, and I haven't had to change the code ever since I started writing this manual.  There are still some potential optimizations and cleanups left to do, but as far as I care, the code is good enough to be frozen.

That is to say, I don't really want to spend more time on this time tracking software.  Keine was developed entirely outside of Company time, since I thought using Company time to track Company time has a bad ring to it.  That said, it's amazing that what used to be a short script written over one weekend ends up to be this executable two months later (but does more or less the same thing), so I am obviously not very efficient at using my time outside of work.  Perhaps I need to gather data on how my weekends are spent as well...

The fine manual
---------------

Like most other projects these days, the last thing I do is to complete the documentation.  It took especially long for Keine, mostly to cover the minimalistic design and implementation.  There are good reasons (and excuses) for minimalistic programs, but I don't think documentation should ever be minimalist.

Of course, it shouldn't be one large blob of unorganized text either, it should be just enough so that when I tell people RTFM, they can find what they want quickly.

Keine
=====

Keine was named after Kamishirasawa Keine (http://www.google.co.jp/search?q=%E4%B8%8A%E7%99%BD%E6%B2%A2%E6%85%A7%E9%9F%B3) from Touhou Eiyashou (http://www.google.co.jp/search?q=%E6%9D%B1%E6%96%B9%E6%B0%B8%E5%A4%9C%E6%8A%84), for her ability to consume and create history.

Extra was easier than Lunatic, heh.