Kick notification source

[conseo]

Hi,

me and Mike have failed at our first attempt to design a Kick source for 
notifications about new messages. Up until now we have tried to strip down 
difference fetches to forums and that to harvest Kicks, but in fact difference 
requests and new messages are not tied to each other and it would result in a 
permanent crawling (polling) anyway.

So we have to come up with a new plan. My attempt is to follow the users 
behaviour closely and automate all possible steps on the way. 

Client <=========> Forum
            1        |
                   2 |
                  Archive

Only these two lines carry events of new messages, forums only expose a 
messaging event to subscribed clients and usually don't provide a public API 
for non-registered clients/services. If we want to emulate the client as a 
Detector, we need to deal with auto-subscription, which in fact makes us a bot 
for the third party forum provider. We cannot fix this problem by design and 
this will be a permanent struggle to automate, because it is checked against 
real human subscribers.
Instead we could let the registration on a forum be done by users, it only 
happens once per forum. To do that we can expose a single mail inbox of the 
MailDetector and let users register it. They have to activate message 
notifications in the account, but we don't need the account data. Usually a 
notification mail contains the message body including the difference url and a 
url to the forum (login or archive url). This is sufficient to map the 
notification to a forum in the DiffKick. Double subscriptions/instances only 
cause double events, and the message never hits the DB twice anyway. They can 
be filtered by the MTA by usual spam measures to block abuse. Harvesting will 
synchronize state and not crawl the same page twice, as we decided yesterday 
on Skype.

Potential problems:
1) It does not work for live chat or live mediums. We need separate Detectors 
for these, like an IRC bot.
2 ) It might not work with all notifications, e.g. if the content is stripped 
or if there is no URL to the forum (unlikely). 
3) Notifications can take too long for a smooth reaction. This is out of our 
hands and depends on the mail setup of the forum provider. Sometimes 
notifications come hours after the message has arrived.

coneso