The idea is to have source agents (`LocalFile` and `S3Agent` for now), which just emit a "file_pointer" and consuming
agents (`ReadFile` and `CsvAgent`) which get access to an `IO` object to actually access the file contents. This way we
do not have to pass megabytes of data through the database and the consuming agents can work with relatively unlimited
file sizes.
squashing 36 commits into one giant commit
Create twitter_favorite agent
Agent that returns favorite tweet of given user
Added history, number and clean up
'history' is buffer to hold historical tweet.id this can be used for memory optimization, 'number' is number of latest tweets that Agent retrieve.
Update to use Array, Clean code, renamed
I put Array into 'memory' to make it simpler, that clean up some not needed code. Hope it will be included to make it more wide used.
Fix'd typos
Fix'd typos #2
Fix'd ref on null object
Fixed `undefined method `include' for nil:NilClass`
Added Pocket variable
Added POCKET_OAUTH_KEY for Pocket support
Adding first scraps
pocket_concerns.rb scrap
something for pocket
Added Pocket gem
Revert "Added Pocket gem"
This reverts commit 701ecd5393b05a8926dba9e45834592dd09774ef.
Revert "pocket_concerns.rb scrap"
This reverts commit cd1b3a751dcb3ca6f36746cfddb23b3e889fb986.
Revert "Adding first scraps"
This reverts commit 4e63b1302364218d46c062f915e1023e63886f43.
Revert "Added Pocket variable"
This reverts commit bb390680bb7bfd6cb424a283d7103dbd15966538.
renamed file with agent to make file names consistent
added in factory girl and disabled master fixture replacement
some work on spec file
spec one fail one success
rm factory girl
cleaner specs
twitter favorites spec half done
deleted extra agent
comment out one of the test
twitter favorite specs
a more proper date
changed back to starting at
fixed number to history typo
altered memory in agent
shifting memory down by one instead of halfing memory last seen
removed typo
replaced all mockobject with agent
fixed indentation issue
setting up event payload information and switched to tectonic
asserted payload looks what it is suppose to
reduced 5 events to 1 event
updated tweets json and spec file
checking out my db schema
rm date from options
rm last test
removed saved from check method
put back last spec and test passed
changed date make it change by 3 instead
moved memory shifting into the loop
moved memory last seen outside of the loop
no_bulk_receive! ensures that agents which connect to an external only receive
one event per job this ensures that failed and retried jobs do not run twice
for the same event.
Fixes#1226
