Harvester Design

[Michael Allan]

Hi C,

This is about the talk harvester as I've been calling it lately.  Is
that a good name?

Can we roughly summarize the design in ASCII?  We need an overview of
how it works at the very top level, maybe something like this:


                                   local UI      remote UI *
                                       |             |
                                       | read        | read
                                       V             V
           kick              store          read
   Kicker ------> Harvester -------> Cache <----- CacheWAP


   Component   Function
   ---------   --------------------------------------
   Kicker      Detect new post in remote archive
   Harvester   Fetch post from remote archive
   Cache       Store post locally
   CacheWAP    Web API for remote reader of cache
   UI          Show post to user, e.g. in TalkTrack *


Is this correct?  If not, please make necessary changes.  Afterwards
I'll ask questions about the details.


  * The talk track (a remote UI) is labeled 2 in this mockup:
    http://zelea.com/project/votorola/s/gwt/stage/_/mock/home-1.xht

-- 
Michael Allan

Toronto, +1 416-699-9528
http://zelea.com/


conseo said:
> Hi Mike,
> 
> I have roughed out a design for the Harvester. (1) Multithreaded execution is 
> handled by an Executor. Currently I propose to use a fixed Thread for every 
> Medium. This means all Jobs submitted by a single Medium will be scheduled in 
> the same thread. (2) Implementations of Job are nested in each Medium, 
> depending on how information has to be retrieved. Jobs can be interrupted by 
> the Harvester, if a higher priority Job arrives. This is supposed to be done 
> at certain key states by the Job itself internally (it returns from run()), to 
> allow a quick interruption where the state of the Job allows it. The Job 
> resubmits (a) Job(s) for the rest of the task if possible or reschedules 
> itself. (3) The Kickers trigger events, which must be interpreted by the 
> Medium appropriately (depending on state). The bottom grouped 
> Job/Scheduler/Medium/Kicker architecture is generic for other services 
> (besides difference related communication). To support a new Medium, you are 
> only supposed to inherit Forum (or implement a different Medium if necessary). 
> (4)
> 
> conseo
> 
> (1) http://whiletaker.homeip.net/votorola/harvester/1/harvester1.png
> Created with ArgoUML http://argouml.tigris.org/ source:
> http://whiletaker.homeip.net/votorola/harvester/1/harvester1.zargo
> 
> (2) Maybe the Executor Pattern does not fit here then. I have followed the 
> Command-Pattern which is realized in the Runnable/Jobs structure. 
> This also means that Medium/Forum classes do not have to be insanely 
> threadsafe (?)
> 
> (3) At the moment I suppose the state to be a timeline backwards with gaps. As 
> gaps are being closed new Jobs only close these gaps. I am not sure if time-
> dimensional browsing will do for all forums though. 
> 
> (4) Medium objects are configured with a ConstructionContext as usual (not 
> shown).