Harvester Design

[Michael Allan]

Hey C,

Based on our talk today, here's an updated diagram for the talk
harvesting architecture.

                                                    (remote)
                  Administrator       Reader         Reader
                       |                |              |
                       | harvest        | read         | read
                       |                |              |
             read      V      store     V     read     V
   Archive <------ Harvester -------> Cache <------ CacheWAP
                       |
                       | listen
                       |
                       V
                     Kicker
                       ^
                       |
                       | trigger
                       |
                    Detector


   Component   Function
   ---------   -----------------------------------------
   Archive     Remote archive of discussion posts
   Cache       Store post locally for quick retrieval
   CacheWAP    Web API for remote reader of cache
   Detector    Detect new post in remote archive
   Harvester   Fetch post from remote archive
   Kicker      Relay detection of new post to listeners
   Reader      Old difference feed, new talk track, etc.


> I don't understand what "local UI" is supposed to mean....

I changed it to Reader.  The only local reader at present is the
CacheWAP, which in turn supports remote readers.  But the design
allows for other local readers such as local UIs, analyzers or
multi-media aggregators.

> ... I see "Cache" as storage and (web) API for information
> retrieval. The "Cache" will only hold DiffMessages, not the complete
> communications stream.

By Cache I mean the local source of the communication stream for the
readers.  I called it a "cache" in order to emphasize that it contains
no original data, only a summary of the remote archive.  But we can
rename it if you like.  Next steps, if you agree:

  1. Redact the diagram above and put it in the Javadocs with links to
     the namesake classes and other docs (below).

  2. Draft the Javadoc API for the various components beginning with
     those most pointed at by arrows:

       a) Cache
       b) Kicker
       c) CacheWAP (Javadoc documenting the web API)

  3. Document the configuration of the Pipermail harvester.  The
     various harvesters should have similar forms of configuration,
     but this cannot be required.  There are two major parts to the
     configuration:

       a) User configuration in pollwiki, such as archive location
       b) Administrative configuration on server

  4. Draft the command interface for the Pipermail harvester.  Again
     the various harvesters should have similar command interfaces,
     but this cannot be required.

  5. Code it.


Does that make sense?  It's a lot of work!  Let me know if you need a
hand with any of the pieces.

-- 
Michael Allan

Toronto, +1 416-699-9528
http://zelea.com/


conseo said:
> Hi M,
> 
> > Can we roughly summarize the design in ASCII?  We need an overview of
> > how it works at the very top level, maybe something like this:
> > 
> > 
> >                                    local UI      remote UI *
> > 
> >                                        | read        | read
> > 
> >                                        V             V
> >            kick              store          read
> >    Kicker ------> Harvester -------> Cache <----- CacheWAP
> > 
> > 
> >    Component   Function
> >    ---------   --------------------------------------
> >    Kicker      Detect new post in remote archive
> >    Harvester   Fetch post from remote archive
> >    Cache       Store post locally
> >    CacheWAP    Web API for remote reader of cache
> >    UI          Show post to user, e.g. in TalkTrack *
> > 
> > 
> > Is this correct?  If not, please make necessary changes.  Afterwards
> > I'll ask questions about the details.
> > 
> > 
> >   * The talk track (a remote UI) is labeled 2 in this mockup:
> >     http://zelea.com/project/votorola/s/gwt/stage/_/mock/home-1.xht
> 
> I don't understand what "local UI" is supposed to mean. I see "Cache" as 
> storage and (web) API for information retrieval. The "Cache" will only hold 
> DiffMessages, not the complete communications stream. 
> 
> Besides that, the ASCII design matches my picture.
> 
> conseo