Harvester Roadmap

[conseo]

Hi M,

> 
> So to cover a new forum, we just give it a page in the pollwiki.  Like
> these ones: http://zelea.com/w/Concept:Forum
> Then its messages will automatically appear in the harvest?
> http://votorola.polyc0l0r.net/javadoc/votorola/s/wap/HarvestWAP.html
> Is that right?
Yes, it does that at runtime and it keeps the cache up-to-date for those 
forums already (update every 3 hours).

> 
> Whatever the order, we might not want to deploy a talk track (2) on
> stage unless it can sync with the forums.  Unlike the old prototype
> scenes, the stage is really intended for useable beta code.  Of
> course, the talk track could sit in a dev repo/branch until a suitable
> forum detector (1) is coded.  Or the detector could be coded first.
> We could deploy it immediately.  The users would not see any benefit,
> that's true, but it would at least improve the existing feed client as
> a prototype/demo.
What do you mean with "sync"? Atm. we have <3 hours update interval which 
makes a track 1) already quite usable imo and 2) has already everything we 
need to write a talk track. But if we come over issues in the track design we 
can fix them before we target the detector handling or further back-end work.

> 
> The detector problem is framed by the use case in this thread:
> http://mail.zelea.com/list/votorola/2012-January/001291.html
> http://mail.zelea.com/list/votorola/2012-January/001292.html
> http://mail.zelea.com/list/votorola/2012-February/001293.html
> 
> I can build detectors into (i) the bridge scene, and (ii) the staging
> of the forum archive.  The first would be the most useful.  It would
> also be easier for you to support.  You just have to ensure that
> redundant kicks are handled efficiently because I'd be calling your
> Kicker whenever a diff is viewed on the bridge.
> 
> I have one doubt about that.  I vaguely recollect discussing an
> email-based subscription detector for Mailman. (?)  Why did we discuss
> that?  Do you recall?  Hopefully it's not needed, because a bridge
> detector is much easier.

There are two problems with it: 1) The kick event does not have to occur, so 
messages might get lost if the author of the message does not trigger the 
difference event after sending to the forum. 2) We don't know from where the 
event comes (because it is likely that it is clicked in the mail (or other 
native client) and we have outruled referer-id for that reason (because we 
cannot burst on all forums all the time). I am afraid, but I think we need a 
Maildir/Mailman detector to get our <10s goal reliably.
Whether we simply write a detector for Maildir which can handle all kind of 
forum updates (often you can get notified for new messages by mail, which 
might work fairly well for many forums) or we separate it for each forum type 
is yet open.


> 
> > http://votorola.polyc0l0r.net/javadoc/votorola/a/diff/harvest/Configurator
> > .html Or is ConfigDetector better? as it also emits Kicks and detects
> > something... I think is still distinct from MaildirDetector or
> > IRCDetector.
> 
> I guess it's some kind of baseline detector, not of messages, but of
> whole forums.  I didn't think we needed that, because new forums can
> be revealed by other detectors, such as the bridge detector.  (The
> first use case in the thread above dealt with something like that.)
I don't think that will work reliable as pointed out before and above.

conseo