GitHub-Mirror/huginn
2
0
Fork 0
mirror of https://github.com/Fishwaldo/huginn.git synced 2025-03-15 19:31:26 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #116 from cantino/do_not_symbolize

Browse source 
Do not symbolize
This commit is contained in:
Andrew Cantino 2014-01-01 13:39:21 -08:00
commit 3d8393357e
60 changed files with 984 additions and 822 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
CHANGES.md
View file

@ -1,8 +1,9 @@
# Changes
* 0.3 (Jan 1, 2014) - Remove symbolization of memory, options, and payloads; convert memory, options, and payloads to JSON from YAML. Migration will perform conversion and adjust tables to be UTF-8. Recommend making a DB backup before migrating.
* 0.2 (Nov 6, 2013) - PeakDetectorAgent now uses `window_duration_in_days` and `min_peak_spacing_in_days`. Additionally, peaks trigger when the time series rises over the standard deviation multiple, not after it starts to fall.
* June 29, 2013 - Removed rails\_admin because it was causing deployment issues. Better to have people install their favorite admin tool if they want one.
* June, 2013 - A number of new agents have been contributed, including interfaces to Weibo, Twitter, and Twilio, as well as Agents for translation, sentiment analysis, and for posting and receiving webhooks.
* March 24, 2013 (0.1) - Refactored loading of Agents for `check` and `receive` to use ids instead of full objects. This should fix the too-large delayed_job issues. Added `system_timer` and `fastercsv` to the Gemfile for the Ruby 1.8 platform.
* March 24, 2013 (0.1) - Refactored loading of Agents for `check` and `receive` to use ids instead of full objects. This should fix the too-large delayed\_job issues. Added `system_timer` and `fastercsv` to the Gemfile for the Ruby 1.8 platform.
* March 18, 2013 - Added Wiki page about the [Agent API](https://github.com/cantino/huginn/wiki/Creating-a-new-agent).
* March 17, 2013 - Switched to JSONPath for defining paths through JSON structures. The WebsiteAgent can now scrape and parse JSON.

6
README.md
View file

@ -13,7 +13,10 @@ Huginn is a system for building agents that perform automated tasks for you onli
* Track the weather and get an email when it's going to rain (or snow) tomorrow
* Follow your project names on Twitter and get updates when people mention them
* Scrape websites and receive emails when they change
* Compose digest emails about things you care about to be sent at specific times of the day
* Track counts of high frequency events and SMS on changes, such as the term "san francisco emergency"
* Track your location over time
* Create Amazon Mechanical Turk tasks as the input, or output, of events
Follow [@tectonic](https://twitter.com/tectonic) for updates as Huginn evolves, and join us in \#huginn on Freenode IRC to discuss the project.
@ -84,7 +87,7 @@ In order to use the WeatherAgent you need an [API key with Wunderground](http://
You can use [Post Location](https://github.com/cantino/post_location) on your iPhone to post your location to an instance of the UserLocationAgent. Make a new one to see instructions.
#### Enable DelayedJobWeb for handy delayed_job monitoring and control
#### Enable DelayedJobWeb for handy delayed\_job monitoring and control
* Edit `config.ru`, uncomment the DelayedJobWeb section, and change the DelayedJobWeb username and password.
* Uncomment `match "/delayed_job" => DelayedJobWeb, :anchor => false` in `config/routes.rb`.
@ -110,6 +113,5 @@ Please fork, add specs, and send pull requests!
[![Build Status](https://travis-ci.org/cantino/huginn.png)](https://travis-ci.org/cantino/huginn) [![Code Climate](https://codeclimate.com/github/cantino/huginn.png)](https://codeclimate.com/github/cantino/huginn)
[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/cantino/huginn/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

2
VERSION
View file

@ -1 +1 @@
0.2
0.3

1
app/assets/javascripts/application.js.coffee.erb
View file

@ -102,6 +102,7 @@ $(document).ready ->
$("#logs .refresh, #logs .clear").hide()
$.post "/agents/#{agentId}/logs/clear", { "_method": "DELETE" }, (html) =>
$("#logs .logs").html html
$("#show-tabs li a.recent-errors").removeClass 'recent-errors'
$("#logs .spinner").stop(true, true).fadeOut ->
$("#logs .refresh, #logs .clear").show()

4
app/assets/stylesheets/application.css.scss.erb
View file

@ -122,3 +122,7 @@ span.not-applicable:after {
margin: 0 10px;
}
}
#show-tabs li a.recent-errors {
font-weight: bold;
}

6
app/concerns/email_concern.rb
View file

@ -1,18 +1,18 @@
module EmailConcern
extend ActiveSupport::Concern
MAIN_KEYS = %w[title message text main value].map(&:to_sym)
MAIN_KEYS = %w[title message text main value]
included do
self.validate :validate_email_options
end
def validate_email_options
errors.add(:base, "subject and expected_receive_period_in_days are required") unless options[:subject].present? && options[:expected_receive_period_in_days].present?
errors.add(:base, "subject and expected_receive_period_in_days are required") unless options['subject'].present? && options['expected_receive_period_in_days'].present?
end
def working?
last_receive_at && last_receive_at > options[:expected_receive_period_in_days].to_i.days.ago && !recent_error_logs?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def present(payload)

16
app/concerns/twitter_concern.rb
View file

@ -7,20 +7,20 @@ module TwitterConcern
end
def validate_twitter_options
unless options[:consumer_key].present? &&
options[:consumer_secret].present? &&
options[:oauth_token].present? &&
options[:oauth_token_secret].present?
unless options['consumer_key'].present? &&
options['consumer_secret'].present? &&
options['oauth_token'].present? &&
options['oauth_token_secret'].present?
errors.add(:base, "consumer_key, consumer_secret, oauth_token and oauth_token_secret are required to authenticate with the Twitter API")
end
end
def configure_twitter
Twitter.configure do |config|
config.consumer_key = options[:consumer_key]
config.consumer_secret = options[:consumer_secret]
config.oauth_token = options[:oauth_token] || options[:access_key]
config.oauth_token_secret = options[:oauth_token_secret] || options[:access_secret]
config.consumer_key = options['consumer_key']
config.consumer_secret = options['consumer_secret']
config.oauth_token = options['oauth_token'] || options['access_key']
config.oauth_token_secret = options['oauth_token_secret'] || options['access_secret']
end
end

12
app/concerns/weibo_concern.rb
View file

@ -6,19 +6,19 @@ module WeiboConcern
end
def validate_weibo_options
unless options[:app_key].present? &&
options[:app_secret].present? &&
options[:access_token].present?
unless options['app_key'].present? &&
options['app_secret'].present? &&
options['access_token'].present?
errors.add(:base, "app_key, app_secret and access_token are required")
end
end
def weibo_client
unless @weibo_client
WeiboOAuth2::Config.api_key = options[:app_key] # WEIBO_APP_KEY
WeiboOAuth2::Config.api_secret = options[:app_secret] # WEIBO_APP_SECRET
WeiboOAuth2::Config.api_key = options['app_key'] # WEIBO_APP_KEY
WeiboOAuth2::Config.api_secret = options['app_secret'] # WEIBO_APP_SECRET
@weibo_client = WeiboOAuth2::Client.new
@weibo_client.get_token_from_hash :access_token => options[:access_token]
@weibo_client.get_token_from_hash :access_token => options['access_token']
end
@weibo_client
end

2
app/controllers/events_controller.rb
View file

@ -6,7 +6,7 @@ class EventsController < ApplicationController
@agent = current_user.agents.find(params[:agent])
@events = @agent.events.page(params[:page])
else
@events = current_user.events.page(params[:page])
@events = current_user.events.preload(:agent).page(params[:page])
end
respond_to do |format|

2
app/controllers/logs_controller.rb
View file

@ -7,7 +7,7 @@ class LogsController < ApplicationController
end
def clear
@agent.logs.delete_all
@agent.delete_logs!
index
end

29
app/models/agent.rb
View file

@ -1,14 +1,13 @@
require 'serialize_and_symbolize'
require 'json_serialized_field'
require 'assignable_types'
require 'markdown_class_attributes'
require 'utils'
class Agent < ActiveRecord::Base
include SerializeAndSymbolize
include AssignableTypes
include MarkdownClassAttributes
include JSONSerializedField
serialize_and_symbolize :options, :memory
markdown_class_attributes :description, :event_description
load_types_in "Agents"
@ -18,9 +17,12 @@ class Agent < ActiveRecord::Base
attr_accessible :options, :memory, :name, :type, :schedule, :source_ids
json_serialize :options, :memory
validates_presence_of :name, :user
validate :sources_are_owned
validate :validate_schedule
validate :validate_options
after_initialize :set_default_schedule
before_validation :set_default_schedule
@ -32,7 +34,6 @@ class Agent < ActiveRecord::Base
has_many :events, :dependent => :delete_all, :inverse_of => :agent, :order => "events.id desc"
has_one :most_recent_event, :inverse_of => :agent, :class_name => "Event", :order => "events.id desc"
has_many :logs, :dependent => :delete_all, :inverse_of => :agent, :class_name => "AgentLog", :order => "agent_logs.id desc"
has_one :most_recent_log, :inverse_of => :agent, :class_name => "AgentLog", :order => "agent_logs.id desc"
has_many :received_events, :through => :sources, :class_name => "Event", :source => :events, :order => "events.id desc"
has_many :links_as_source, :dependent => :delete_all, :foreign_key => "source_id", :class_name => "Link", :inverse_of => :source
has_many :links_as_receiver, :dependent => :delete_all, :foreign_key => "receiver_id", :class_name => "Link", :inverse_of => :receiver
@ -74,13 +75,16 @@ class Agent < ActiveRecord::Base
raise "Implement me in your subclass"
end
def event_created_within(days)
event = most_recent_event
event && event.created_at > days.to_i.days.ago && event.payload.present? && event
def validate_options
# Implement me in your subclass to test for valid options.
end
def event_created_within?(days)
last_event_at && last_event_at > days.to_i.days.ago
end
def recent_error_logs?
most_recent_log.try(:level) == 4
last_event_at && last_error_log_at && last_error_log_at > (last_event_at - 2.minutes)
end
def sources_are_owned
@ -120,10 +124,6 @@ class Agent < ActiveRecord::Base
self.schedule = nil if cannot_be_scheduled?
end
def last_event_at
@memoized_last_event_at ||= most_recent_event.try(:created_at)
end
def default_schedule
self.class.default_schedule
end
@ -158,6 +158,11 @@ class Agent < ActiveRecord::Base
end
end
def delete_logs!
logs.delete_all
update_column :last_error_log_at, nil
end
def log(message, options = {})
puts "Agent##{id}: #{message}" unless Rails.env.test?
AgentLog.log_for_agent(self, message, options)

3
app/models/agent_log.rb
View file

@ -14,6 +14,9 @@ class AgentLog < ActiveRecord::Base
oldest_id_to_keep = agent.logs.limit(1).offset(log_length - 1).pluck("agent_logs.id")
agent.logs.where("agent_logs.id < ?", oldest_id_to_keep).delete_all
end
agent.update_column :last_error_log_at, Time.now if log.level >= 4
log
end

26
app/models/agents/adioso_agent.rb
View file

@ -29,22 +29,22 @@ module Agents
def default_options
{
:start_date => Date.today.httpdate[0..15],
:end_date => Date.today.plus_with_duration(100).httpdate[0..15],
:from => "New York",
:to => "Chicago",
:username => "xx",
:password => "xx",
:expected_update_period_in_days => "1"
'start_date' => Date.today.httpdate[0..15],
'end_date' => Date.today.plus_with_duration(100).httpdate[0..15],
'from' => "New York",
'to' => "Chicago",
'username' => "xx",
'password' => "xx",
'expected_update_period_in_days' => "1"
}
end
def working?
event_created_within(options[:expected_update_period_in_days]) && !recent_error_logs?
event_created_within?(options['expected_update_period_in_days']) && !recent_error_logs?
end
def validate_options
unless %w[start_date end_date from to username password expected_update_period_in_days].all? { |field| options[field.to_sym].present? }
unless %w[start_date end_date from to username password expected_update_period_in_days].all? { |field| options[field].present? }
errors.add(:base, "All fields are required")
end
end
@ -54,9 +54,9 @@ module Agents
end
def check
auth_options = {:basic_auth => {:username =>options[:username], :password=>options[:password]}}
parse_response = HTTParty.get "http://api.adioso.com/v2/search/parse?q=#{URI.encode(options[:from])}+to+#{URI.encode(options[:to])}", auth_options
fare_request = parse_response["search_url"].gsub /(end=)(\d*)([^\d]*)(\d*)/, "\\1#{date_to_unix_epoch(options[:end_date])}\\3#{date_to_unix_epoch(options[:start_date])}"
auth_options = {:basic_auth => {:username =>options[:username], :password=>options['password']}}
parse_response = HTTParty.get "http://api.adioso.com/v2/search/parse?q=#{URI.encode(options['from'])}+to+#{URI.encode(options['to'])}", auth_options
fare_request = parse_response["search_url"].gsub /(end=)(\d*)([^\d]*)(\d*)/, "\\1#{date_to_unix_epoch(options['end_date'])}\\3#{date_to_unix_epoch(options['start_date'])}"
fare = HTTParty.get fare_request, auth_options
if fare["warnings"]
@ -64,7 +64,7 @@ module Agents
else
event = fare["results"].min {|a,b| a["cost"] <=> b["cost"]}
event["date"] = Time.at(event["date"]).to_date.httpdate[0..15]
event["route"] = "#{options[:from]} to #{options[:to]}"
event["route"] = "#{options['from']} to #{options['to']}"
create_event :payload => event
end
end

28
app/models/agents/digest_email_agent.rb
View file

@ -9,7 +9,7 @@ module Agents
description <<-MD
The DigestEmailAgent collects any Events sent to it and sends them all via email when run.
The email will be sent to your account's address and will have a `subject` and an optional `headline` before
listing the Events. If the Events' payloads contain a `:message`, that will be highlighted, otherwise everything in
listing the Events. If the Events' payloads contain a `message`, that will be highlighted, otherwise everything in
their payloads will be shown.
Set `expected_receive_period_in_days` to the maximum amount of time that you'd expect to pass between Events being received by this Agent.
@ -17,29 +17,29 @@ module Agents
def default_options
{
:subject => "You have some notifications!",
:headline => "Your notifications:",
:expected_receive_period_in_days => "2"
'subject' => "You have some notifications!",
'headline' => "Your notifications:",
'expected_receive_period_in_days' => "2"
}
end
def receive(incoming_events)
incoming_events.each do |event|
self.memory[:queue] ||= []
self.memory[:queue] << event.payload
self.memory[:events] ||= []
self.memory[:events] << event.id
self.memory['queue'] ||= []
self.memory['queue'] << event.payload
self.memory['events'] ||= []
self.memory['events'] << event.id
end
end
def check
if self.memory[:queue] && self.memory[:queue].length > 0
ids = self.memory[:events].join(",")
groups = self.memory[:queue].map { |payload| present(payload) }
if self.memory['queue'] && self.memory['queue'].length > 0
ids = self.memory['events'].join(",")
groups = self.memory['queue'].map { |payload| present(payload) }
log "Sending digest mail to #{user.email} with events [#{ids}]"
SystemMailer.delay.send_message(:to => user.email, :subject => options[:subject], :headline => options[:headline], :groups => groups)
self.memory[:queue] = []
self.memory[:events] = []
SystemMailer.delay.send_message(:to => user.email, :subject => options['subject'], :headline => options['headline'], :groups => groups)
self.memory['queue'] = []
self.memory['events'] = []
end
end
end

8
app/models/agents/email_agent.rb
View file

@ -16,16 +16,16 @@ module Agents
def default_options
{
:subject => "You have a notification!",
:headline => "Your notification:",
:expected_receive_period_in_days => "2"
'subject' => "You have a notification!",
'headline' => "Your notification:",
'expected_receive_period_in_days' => "2"
}
end
def receive(incoming_events)
incoming_events.each do |event|
log "Sending digest mail to #{user.email} with event #{event.id}"
SystemMailer.delay.send_message(:to => user.email, :subject => options[:subject], :headline => options[:headline], :groups => [present(event.payload)])
SystemMailer.delay.send_message(:to => user.email, :subject => options['subject'], :headline => options['headline'], :groups => [present(event.payload)])
end
end
end

44
app/models/agents/event_formatting_agent.rb
View file

@ -8,20 +8,20 @@ module Agents
For example, here is a possible Event:
{
:high => {
:celsius => "18",
:fahreinheit => "64"
"high": {
"celsius": "18",
"fahreinheit": "64"
},
:conditions => "Rain showers",
:data => "This is some data"
"conditions": "Rain showers",
"data": "This is some data"
}
You may want to send this event to another Agent, for example a Twilio Agent, which expects a `message` key.
You can use an Event Formatting Agent's `instructions` setting to do this in the following way:
instructions: {
message: "Today's conditions look like <$.conditions> with a high temperature of <$.high.celsius> degrees Celsius.",
subject: "$.data"
"instructions": {
"message": "Today's conditions look like <$.conditions> with a high temperature of <$.high.celsius> degrees Celsius.",
"subject": "$.data"
}
JSONPaths must be between < and > . Make sure that you don't use these symbols anywhere else.
@ -29,8 +29,8 @@ module Agents
Events generated by this possible Event Formatting Agent will look like:
{
:message => "Today's conditions look like Rain showers with a high temperature of 18 degrees Celsius.",
:subject => "This is some data"
"message": "Today's conditions look like Rain showers with a high temperature of 18 degrees Celsius.",
"subject": "This is some data"
}
If you want to retain original contents of events and only add new keys, then set `mode` to `merge`, otherwise set it to `clean`.
@ -40,25 +40,25 @@ module Agents
To CGI escape output (for example when creating a link), prefix with `escape`, like so:
{
:message => "A peak was on Twitter in <$.group_by>. Search: https://twitter.com/search?q=<escape $.group_by>"
"message": "A peak was on Twitter in <$.group_by>. Search: https://twitter.com/search?q=<escape $.group_by>"
}
MD
event_description "User defined"
def validate_options
errors.add(:base, "instructions, mode, skip_agent, and skip_created_at all need to be present.") unless options[:instructions].present? and options[:mode].present? and options[:skip_agent].present? and options[:skip_created_at].present?
errors.add(:base, "instructions, mode, skip_agent, and skip_created_at all need to be present.") unless options['instructions'].present? and options['mode'].present? and options['skip_agent'].present? and options['skip_created_at'].present?
end
def default_options
{
:instructions => {
:message => "You received a text <$.text> from <$.fields.from>",
:some_other_field => "Looks like the weather is going to be <$.fields.weather>"
'instructions' => {
'message' => "You received a text <$.text> from <$.fields.from>",
'some_other_field' => "Looks like the weather is going to be <$.fields.weather>"
},
:mode => "clean",
:skip_agent => "false",
:skip_created_at => "false"
'mode' => "clean",
'skip_agent' => "false",
'skip_created_at' => "false"
}
end
@ -68,10 +68,10 @@ module Agents
def receive(incoming_events)
incoming_events.each do |event|
formatted_event = options[:mode].to_s == "merge" ? event.payload : {}
options[:instructions].each_pair {|key, value| formatted_event[key] = Utils.interpolate_jsonpaths(value, event.payload) }
formatted_event[:agent] = Agent.find(event.agent_id).type.slice!(8..-1) unless options[:skip_agent].to_s == "true"
formatted_event[:created_at] = event.created_at unless options[:skip_created_at].to_s == "true"
formatted_event = options['mode'].to_s == "merge" ? event.payload : {}
options['instructions'].each_pair {|key, value| formatted_event[key] = Utils.interpolate_jsonpaths(value, event.payload) }
formatted_event['agent'] = Agent.find(event.agent_id).type.slice!(8..-1) unless options['skip_agent'].to_s == "true"
formatted_event['created_at'] = event.created_at unless options['skip_created_at'].to_s == "true"
create_event :payload => formatted_event
end
end

237
app/models/agents/human_task_agent.rb
View file

@ -128,73 +128,73 @@ module Agents
MD
def validate_options
options[:hit] ||= {}
options[:hit][:questions] ||= []
options['hit'] ||= {}
options['hit']['questions'] ||= []
errors.add(:base, "'trigger_on' must be one of 'schedule' or 'event'") unless %w[schedule event].include?(options[:trigger_on])
errors.add(:base, "'hit.assignments' should specify the number of HIT assignments to create") unless options[:hit][:assignments].present? && options[:hit][:assignments].to_i > 0
errors.add(:base, "'hit.title' must be provided") unless options[:hit][:title].present?
errors.add(:base, "'hit.description' must be provided") unless options[:hit][:description].present?
errors.add(:base, "'hit.questions' must be provided") unless options[:hit][:questions].present? && options[:hit][:questions].length > 0
errors.add(:base, "'trigger_on' must be one of 'schedule' or 'event'") unless %w[schedule event].include?(options['trigger_on'])
errors.add(:base, "'hit.assignments' should specify the number of HIT assignments to create") unless options['hit']['assignments'].present? && options['hit']['assignments'].to_i > 0
errors.add(:base, "'hit.title' must be provided") unless options['hit']['title'].present?
errors.add(:base, "'hit.description' must be provided") unless options['hit']['description'].present?
errors.add(:base, "'hit.questions' must be provided") unless options['hit']['questions'].present? && options['hit']['questions'].length > 0
if options[:trigger_on] == "event"
errors.add(:base, "'expected_receive_period_in_days' is required when 'trigger_on' is set to 'event'") unless options[:expected_receive_period_in_days].present?
elsif options[:trigger_on] == "schedule"
errors.add(:base, "'submission_period' must be set to a positive number of hours when 'trigger_on' is set to 'schedule'") unless options[:submission_period].present? && options[:submission_period].to_i > 0
if options['trigger_on'] == "event"
errors.add(:base, "'expected_receive_period_in_days' is required when 'trigger_on' is set to 'event'") unless options['expected_receive_period_in_days'].present?
elsif options['trigger_on'] == "schedule"
errors.add(:base, "'submission_period' must be set to a positive number of hours when 'trigger_on' is set to 'schedule'") unless options['submission_period'].present? && options['submission_period'].to_i > 0
end
if options[:hit][:questions].any? { |question| [:key, :name, :required, :type, :question].any? {|k| !question[k].present? } }
if options['hit']['questions'].any? { |question| %w[key name required type question].any? {|k| !question[k].present? } }
errors.add(:base, "all questions must set 'key', 'name', 'required', 'type', and 'question'")
end
if options[:hit][:questions].any? { |question| question[:type] == "selection" && (!question[:selections].present? || question[:selections].length == 0 || !question[:selections].all? {|s| s[:key].present? } || !question[:selections].all? { |s| s[:text].present? })}
if options['hit']['questions'].any? { |question| question['type'] == "selection" && (!question['selections'].present? || question['selections'].length == 0 || !question['selections'].all? {|s| s['key'].present? } || !question['selections'].all? { |s| s['text'].present? })}
errors.add(:base, "all questions of type 'selection' must have a selections array with selections that set 'key' and 'name'")
end
if take_majority? && options[:hit][:questions].any? { |question| question[:type] != "selection" }
if take_majority? && options['hit']['questions'].any? { |question| question['type'] != "selection" }
errors.add(:base, "all questions must be of type 'selection' to use the 'take_majority' option")
end
if create_poll?
errors.add(:base, "poll_options is required when combination_mode is set to 'poll' and must have the keys 'title', 'instructions', 'row_template', and 'assignments'") unless options[:poll_options].is_a?(Hash) && options[:poll_options][:title].present? && options[:poll_options][:instructions].present? && options[:poll_options][:row_template].present? && options[:poll_options][:assignments].to_i > 0
errors.add(:base, "poll_options is required when combination_mode is set to 'poll' and must have the keys 'title', 'instructions', 'row_template', and 'assignments'") unless options['poll_options'].is_a?(Hash) && options['poll_options']['title'].present? && options['poll_options']['instructions'].present? && options['poll_options']['row_template'].present? && options['poll_options']['assignments'].to_i > 0
end
end
def default_options
{
:expected_receive_period_in_days => 2,
:trigger_on => "event",
:hit =>
'expected_receive_period_in_days' => 2,
'trigger_on' => "event",
'hit' =>
{
:assignments => 1,
:title => "Sentiment evaluation",
:description => "Please rate the sentiment of this message: '<$.message>'",
:reward => 0.05,
:lifetime_in_seconds => 24 * 60 * 60,
:questions =>
'assignments' => 1,
'title' => "Sentiment evaluation",
'description' => "Please rate the sentiment of this message: '<$.message>'",
'reward' => 0.05,
'lifetime_in_seconds' => 24 * 60 * 60,
'questions' =>
[
{
:type => "selection",
:key => "sentiment",
:name => "Sentiment",
:required => "true",
:question => "Please select the best sentiment value:",
:selections =>
'type' => "selection",
'key' => "sentiment",
'name' => "Sentiment",
'required' => "true",
'question' => "Please select the best sentiment value:",
'selections' =>
[
{ :key => "happy", :text => "Happy" },
{ :key => "sad", :text => "Sad" },
{ :key => "neutral", :text => "Neutral" }
{ 'key' => "happy", 'text' => "Happy" },
{ 'key' => "sad", 'text' => "Sad" },
{ 'key' => "neutral", 'text' => "Neutral" }
]
},
{
:type => "free_text",
:key => "feedback",
:name => "Have any feedback for us?",
:required => "false",
:question => "Feedback",
:default => "Type here...",
:min_length => "2",
:max_length => "2000"
'type' => "free_text",
'key' => "feedback",
'name' => "Have any feedback for us?",
'required' => "false",
'question' => "Feedback",
'default' => "Type here...",
'min_length' => "2",
'max_length' => "2000"
}
]
}
@ -202,20 +202,20 @@ module Agents
end
def working?
last_receive_at && last_receive_at > options[:expected_receive_period_in_days].to_i.days.ago && !recent_error_logs?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def check
review_hits
if options[:trigger_on] == "schedule" && (memory[:last_schedule] || 0) <= Time.now.to_i - options[:submission_period].to_i * 60 * 60
memory[:last_schedule] = Time.now.to_i
if options['trigger_on'] == "schedule" && (memory['last_schedule'] || 0) <= Time.now.to_i - options['submission_period'].to_i * 60 * 60
memory['last_schedule'] = Time.now.to_i
create_basic_hit
end
end
def receive(incoming_events)
if options[:trigger_on] == "event"
if options['trigger_on'] == "event"
incoming_events.each do |event|
create_basic_hit event
end
@ -225,33 +225,32 @@ module Agents
protected
def take_majority?
options[:combination_mode] == "take_majority" || options[:take_majority] == "true"
options['combination_mode'] == "take_majority" || options['take_majority'] == "true"
end
def create_poll?
options[:combination_mode] == "poll"
options['combination_mode'] == "poll"
end
def event_for_hit(hit_id)
if memory[:hits][hit_id.to_sym].is_a?(Hash)
Event.find_by_id(memory[:hits][hit_id.to_sym][:event_id])
if memory['hits'][hit_id].is_a?(Hash)
Event.find_by_id(memory['hits'][hit_id]['event_id'])
else
nil
end
end
def hit_type(hit_id)
# Fix this: the Ruby process will slowly run out of RAM by symbolizing these unique keys.
if memory[:hits][hit_id.to_sym].is_a?(Hash) && memory[:hits][hit_id.to_sym][:type]
memory[:hits][hit_id.to_sym][:type].to_sym
if memory['hits'][hit_id].is_a?(Hash) && memory['hits'][hit_id]['type']
memory['hits'][hit_id]['type']
else
:user
'user'
end
end
def review_hits
reviewable_hit_ids = RTurk::GetReviewableHITs.create.hit_ids
my_reviewed_hit_ids = reviewable_hit_ids & (memory[:hits] || {}).keys.map(&:to_s)
my_reviewed_hit_ids = reviewable_hit_ids & (memory['hits'] || {}).keys
if reviewable_hit_ids.length > 0
log "MTurk reports #{reviewable_hit_ids.length} HITs, of which I own [#{my_reviewed_hit_ids.to_sentence}]"
end
@ -264,7 +263,7 @@ module Agents
if assignments.length == hit.max_assignments && assignments.all? { |assignment| assignment.status == "Submitted" }
inbound_event = event_for_hit(hit_id)
if hit_type(hit_id) == :poll
if hit_type(hit_id) == 'poll'
# handle completed polls
log "Handling a poll: #{hit_id}"
@ -280,35 +279,35 @@ module Agents
top_answer = scores.to_a.sort {|b, a| a.last <=> b.last }.first.first
payload = {
:answers => memory[:hits][hit_id.to_sym][:answers],
:poll => assignments.map(&:answers),
:best_answer => memory[:hits][hit_id.to_sym][:answers][top_answer.to_i - 1]
'answers' => memory['hits'][hit_id]['answers'],
'poll' => assignments.map(&:answers),
'best_answer' => memory['hits'][hit_id]['answers'][top_answer.to_i - 1]
}
event = create_event :payload => payload
log "Event emitted with answer(s) for poll", :outbound_event => event, :inbound_event => inbound_event
else
# handle normal completed HITs
payload = { :answers => assignments.map(&:answers) }
payload = { 'answers' => assignments.map(&:answers) }
if take_majority?
counts = {}
options[:hit][:questions].each do |question|
question_counts = question[:selections].inject({}) { |memo, selection| memo[selection[:key]] = 0; memo }
options['hit']['questions'].each do |question|
question_counts = question['selections'].inject({}) { |memo, selection| memo[selection['key']] = 0; memo }
assignments.each do |assignment|
answers = ActiveSupport::HashWithIndifferentAccess.new(assignment.answers)
answer = answers[question[:key]]
answer = answers[question['key']]
question_counts[answer] += 1
end
counts[question[:key]] = question_counts
counts[question['key']] = question_counts
end
payload[:counts] = counts
payload['counts'] = counts
majority_answer = counts.inject({}) do |memo, (key, question_counts)|
memo[key] = question_counts.to_a.sort {|a, b| a.last <=> b.last }.last.first
memo
end
payload[:majority_answer] = majority_answer
payload['majority_answer'] = majority_answer
if all_questions_are_numeric?
average_answer = counts.inject({}) do |memo, (key, question_counts)|
@ -320,35 +319,35 @@ module Agents
memo[key] = sum / divisor.to_f
memo
end
payload[:average_answer] = average_answer
payload['average_answer'] = average_answer
end
end
if create_poll?
questions = []
selections = 5.times.map { |i| { :key => i+1, :text => i+1 } }.reverse
selections = 5.times.map { |i| { 'key' => i+1, 'text' => i+1 } }.reverse
assignments.length.times do |index|
questions << {
:type => "selection",
:name => "Item #{index + 1}",
:key => index,
:required => "true",
:question => Utils.interpolate_jsonpaths(options[:poll_options][:row_template], assignments[index].answers),
:selections => selections
'type' => "selection",
'name' => "Item #{index + 1}",
'key' => index,
'required' => "true",
'question' => Utils.interpolate_jsonpaths(options['poll_options']['row_template'], assignments[index].answers),
'selections' => selections
}
end
poll_hit = create_hit :title => options[:poll_options][:title],
:description => options[:poll_options][:instructions],
:questions => questions,
:assignments => options[:poll_options][:assignments],
:lifetime_in_seconds => options[:poll_options][:lifetime_in_seconds],
:reward => options[:poll_options][:reward],
:payload => inbound_event && inbound_event.payload,
:metadata => { :type => :poll,
:original_hit => hit_id,
:answers => assignments.map(&:answers),
:event_id => inbound_event && inbound_event.id }
poll_hit = create_hit 'title' => options['poll_options']['title'],
'description' => options['poll_options']['instructions'],
'questions' => questions,
'assignments' => options['poll_options']['assignments'],
'lifetime_in_seconds' => options['poll_options']['lifetime_in_seconds'],
'reward' => options['poll_options']['reward'],
'payload' => inbound_event && inbound_event.payload,
'metadata' => { 'type' => 'poll',
'original_hit' => hit_id,
'answers' => assignments.map(&:answers),
'event_id' => inbound_event && inbound_event.id }
log "Poll HIT created with ID #{poll_hit.id} and URL #{poll_hit.url}. Original HIT: #{hit_id}", :inbound_event => inbound_event
else
@ -360,47 +359,47 @@ module Agents
assignments.each(&:approve!)
hit.dispose!
memory[:hits].delete(hit_id.to_sym)
memory['hits'].delete(hit_id)
end
end
end
def all_questions_are_numeric?
options[:hit][:questions].all? do |question|
question[:selections].all? do |selection|
selection[:key] == selection[:key].to_f.to_s || selection[:key] == selection[:key].to_i.to_s
options['hit']['questions'].all? do |question|
question['selections'].all? do |selection|
selection['key'] == selection['key'].to_f.to_s || selection['key'] == selection['key'].to_i.to_s
end
end
end
def create_basic_hit(event = nil)
hit = create_hit :title => options[:hit][:title],
:description => options[:hit][:description],
:questions => options[:hit][:questions],
:assignments => options[:hit][:assignments],
:lifetime_in_seconds => options[:hit][:lifetime_in_seconds],
:reward => options[:hit][:reward],
:payload => event && event.payload,
:metadata => { :event_id => event && event.id }
hit = create_hit 'title' => options['hit']['title'],
'description' => options['hit']['description'],
'questions' => options['hit']['questions'],
'assignments' => options['hit']['assignments'],
'lifetime_in_seconds' => options['hit']['lifetime_in_seconds'],
'reward' => options['hit']['reward'],
'payload' => event && event.payload,
'metadata' => { 'event_id' => event && event.id }
log "HIT created with ID #{hit.id} and URL #{hit.url}", :inbound_event => event
end
def create_hit(opts = {})
payload = opts[:payload] || {}
title = Utils.interpolate_jsonpaths(opts[:title], payload).strip
description = Utils.interpolate_jsonpaths(opts[:description], payload).strip
questions = Utils.recursively_interpolate_jsonpaths(opts[:questions], payload)
payload = opts['payload'] || {}
title = Utils.interpolate_jsonpaths(opts['title'], payload).strip
description = Utils.interpolate_jsonpaths(opts['description'], payload).strip
questions = Utils.recursively_interpolate_jsonpaths(opts['questions'], payload)
hit = RTurk::Hit.create(:title => title) do |hit|
hit.max_assignments = (opts[:assignments] || 1).to_i
hit.max_assignments = (opts['assignments'] || 1).to_i
hit.description = description
hit.lifetime = (opts[:lifetime_in_seconds] || 24 * 60 * 60).to_i
hit.lifetime = (opts['lifetime_in_seconds'] || 24 * 60 * 60).to_i
hit.question_form AgentQuestionForm.new(:title => title, :description => description, :questions => questions)
hit.reward = (opts[:reward] || 0.05).to_f
hit.reward = (opts['reward'] || 0.05).to_f
#hit.qualifications.add :approval_rate, { :gt => 80 }
end
memory[:hits] ||= {}
memory[:hits][hit.id] = opts[:metadata] || {}
memory['hits'] ||= {}
memory['hits'][hit.id] = opts['metadata'] || {}
hit
end
@ -422,34 +421,34 @@ module Agents
@questions.each.with_index do |question, index|
Question do
QuestionIdentifier do
text question[:key] || "question_#{index}"
text question['key'] || "question_#{index}"
end
DisplayName do
text question[:name] || "Question ##{index}"
text question['name'] || "Question ##{index}"
end
IsRequired do
text question[:required] || 'true'
text question['required'] || 'true'
end
QuestionContent do
Text do
text question[:question]
text question['question']
end
end
AnswerSpecification do
if question[:type] == "selection"
if question['type'] == "selection"
SelectionAnswer do
StyleSuggestion do
text 'radiobutton'
end
Selections do
question[:selections].each do |selection|
question['selections'].each do |selection|
Selection do
SelectionIdentifier do
text selection[:key]
text selection['key']
end
Text do
text selection[:text]
text selection['text']
end
end
end
@ -459,18 +458,18 @@ module Agents
else
FreeTextAnswer do
if question[:min_length].present? || question[:max_length].present?
if question['min_length'].present? || question['max_length'].present?
Constraints do
lengths = {}
lengths[:minLength] = question[:min_length].to_s if question[:min_length].present?
lengths[:maxLength] = question[:max_length].to_s if question[:max_length].present?
lengths['minLength'] = question['min_length'].to_s if question['min_length'].present?
lengths['maxLength'] = question['max_length'].to_s if question['max_length'].present?
Length lengths
end
end
if question[:default].present?
if question['default'].present?
DefaultText do
text question[:default]
text question['default']
end
end
end
@ -482,4 +481,4 @@ module Agents
end
end
end
end
end

4
app/models/agents/manual_event_agent.rb
View file

@ -14,8 +14,8 @@ module Agents
end
def handle_details_post(params)
if params[:payload]
create_event(:payload => params[:payload])
if params['payload']
create_event(:payload => params['payload'])
{ :success => true }
else
{ :success => false, :error => "You must provide a JSON payload" }

56
app/models/agents/peak_detector_agent.rb
View file

@ -28,22 +28,22 @@ module Agents
MD
def validate_options
unless options[:expected_receive_period_in_days].present? && options[:message].present? && options[:value_path].present?
unless options['expected_receive_period_in_days'].present? && options['message'].present? && options['value_path'].present?
errors.add(:base, "expected_receive_period_in_days, value_path, and message are required")
end
end
def default_options
{
:expected_receive_period_in_days => "2",
:group_by_path => "filter",
:value_path => "count",
:message => "A peak was found"
'expected_receive_period_in_days' => "2",
'group_by_path' => "filter",
'value_path' => "count",
'message' => "A peak was found"
}
end
def working?
last_receive_at && last_receive_at > options[:expected_receive_period_in_days].to_i.days.ago && !recent_error_logs?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def receive(incoming_events)
@ -57,25 +57,23 @@ module Agents
private
def check_for_peak(group, event)
memory[:peaks] ||= {}
memory[:peaks][group] ||= []
memory['peaks'] ||= {}
memory['peaks'][group] ||= []
if memory[:data][group].length > 4 && (memory[:peaks][group].empty? || memory[:peaks][group].last < event.created_at.to_i - peak_spacing)
if memory['data'][group].length > 4 && (memory['peaks'][group].empty? || memory['peaks'][group].last < event.created_at.to_i - peak_spacing)
average_value, standard_deviation = stats_for(group, :skip_last => 1)
newest_value, newest_time = memory[:data][group][-1].map(&:to_f)
#p [newest_value, average_value, average_value + std_multiple * standard_deviation, standard_deviation]
newest_value, newest_time = memory['data'][group][-1].map(&:to_f)
if newest_value > average_value + std_multiple * standard_deviation
memory[:peaks][group] << newest_time
memory[:peaks][group].reject! { |p| p <= newest_time - window_duration }
create_event :payload => {:message => options[:message], :peak => newest_value, :peak_time => newest_time, :grouped_by => group.to_s}
memory['peaks'][group] << newest_time
memory['peaks'][group].reject! { |p| p <= newest_time - window_duration }
create_event :payload => { 'message' => options['message'], 'peak' => newest_value, 'peak_time' => newest_time, 'grouped_by' => group.to_s }
end
end
end
def stats_for(group, options = {})
data = memory[:data][group].map { |d| d.first.to_f }
data = memory['data'][group].map { |d| d.first.to_f }
data = data[0...(data.length - (options[:skip_last] || 0))]
length = data.length.to_f
mean = 0
@ -94,39 +92,39 @@ module Agents
end
def window_duration
if options[:window_duration].present? # The older option
options[:window_duration].to_i
if options['window_duration'].present? # The older option
options['window_duration'].to_i
else
(options[:window_duration_in_days] || 14).to_f.days
(options['window_duration_in_days'] || 14).to_f.days
end
end
def std_multiple
(options[:std_multiple] || 3).to_f
(options['std_multiple'] || 3).to_f
end
def peak_spacing
if options[:peak_spacing].present? # The older option
options[:peak_spacing].to_i
if options['peak_spacing'].present? # The older option
options['peak_spacing'].to_i
else
(options[:min_peak_spacing_in_days] || 2).to_f.days
(options['min_peak_spacing_in_days'] || 2).to_f.days
end
end
def group_for(event)
((options[:group_by_path].present? && Utils.value_at(event.payload, options[:group_by_path])) || 'no_group').to_sym
((options['group_by_path'].present? && Utils.value_at(event.payload, options['group_by_path'])) || 'no_group')
end
def remember(group, event)
memory[:data] ||= {}
memory[:data][group] ||= []
memory[:data][group] << [Utils.value_at(event.payload, options[:value_path]), event.created_at.to_i]
memory['data'] ||= {}
memory['data'][group] ||= []
memory['data'][group] << [ Utils.value_at(event.payload, options['value_path']), event.created_at.to_i ]
cleanup group
end
def cleanup(group)
newest_time = memory[:data][group].last.last
memory[:data][group].reject! { |value, time| time <= newest_time - window_duration }
newest_time = memory['data'][group].last.last
memory['data'][group].reject! { |value, time| time <= newest_time - window_duration }
end
end
end

8
app/models/agents/post_agent.rb
View file

@ -11,17 +11,17 @@ module Agents
def default_options
{
:post_url => "http://www.example.com",
:expected_receive_period_in_days => 1
'post_url' => "http://www.example.com",
'expected_receive_period_in_days' => 1
}
end
def working?
last_receive_at && last_receive_at > options[:expected_receive_period_in_days].to_i.days.ago && !recent_error_logs?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def validate_options
unless options[:post_url].present? && options[:expected_receive_period_in_days].present?
unless options['post_url'].present? && options['expected_receive_period_in_days'].present?
errors.add(:base, "post_url and expected_receive_period_in_days are required fields")
end
end

20
app/models/agents/sentiment_agent.rb
View file

@ -28,31 +28,31 @@ module Agents
def default_options
{
:content => "$.message.text[*]",
:expected_receive_period_in_days => 1
'content' => "$.message.text[*]",
'expected_receive_period_in_days' => 1
}
end
def working?
last_receive_at && last_receive_at > options[:expected_receive_period_in_days].to_i.days.ago && !recent_error_logs?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def receive(incoming_events)
anew = self.class.sentiment_hash
incoming_events.each do |event|
Utils.values_at(event.payload, options[:content]).each do |content|
Utils.values_at(event.payload, options['content']).each do |content|
sent_values = sentiment_values anew, content
create_event :payload => { :content => content,
:valence => sent_values[0],
:arousal => sent_values[1],
:dominance => sent_values[2],
:original_event => event.payload }
create_event :payload => { 'content' => content,
'valence' => sent_values[0],
'arousal' => sent_values[1],
'dominance' => sent_values[2],
'original_event' => event.payload }
end
end
end
def validate_options
errors.add(:base, "content and expected_receive_period_in_days must be present") unless options[:content].present? && options[:expected_receive_period_in_days].present?
errors.add(:base, "content and expected_receive_period_in_days must be present") unless options['content'].present? && options['expected_receive_period_in_days'].present?
end
def self.sentiment_hash

30
app/models/agents/translation_agent.rb
View file

@ -17,26 +17,26 @@ module Agents
def default_options
{
:client_id => "xxxxxx",
:client_secret => "xxxxxx",
:to => "fi",
:expected_receive_period_in_days => 1,
:content => {
:text => "$.message.text",
:content => "$.xyz"
'client_id' => "xxxxxx",
'client_secret' => "xxxxxx",
'to' => "fi",
'expected_receive_period_in_days' => 1,
'content' => {
'text' => "$.message.text",
'content' => "$.xyz"
}
}
end
def working?
last_receive_at && last_receive_at > options[:expected_receive_period_in_days].to_i.days.ago && !recent_error_logs?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def translate(text, to, access_token)
translate_uri = URI 'http://api.microsofttranslator.com/v2/Ajax.svc/Translate'
params = {
:text => text,
:to => to
'text' => text,
'to' => to
}
translate_uri.query = URI.encode_www_form params
request = Net::HTTP::Get.new translate_uri.request_uri
@ -47,7 +47,7 @@ module Agents
end
def validate_options
unless options[:client_id].present? && options[:client_secret].present? && options[:to].present? && options[:content].present? && options[:expected_receive_period_in_days].present?
unless options['client_id'].present? && options['client_secret'].present? && options['to'].present? && options['content'].present? && options['expected_receive_period_in_days'].present?
errors.add :base, "client_id,client_secret,to,expected_receive_period_in_days and content are all required"
end
end
@ -60,16 +60,16 @@ module Agents
def receive(incoming_events)
auth_uri = URI "https://datamarket.accesscontrol.windows.net/v2/OAuth2-13"
response = postform auth_uri, :client_id => options[:client_id],
:client_secret => options[:client_secret],
response = postform auth_uri, :client_id => options['client_id'],
:client_secret => options['client_secret'],
:scope => "http://api.microsofttranslator.com",
:grant_type => "client_credentials"
access_token = JSON.parse(response.body)["access_token"]
incoming_events.each do |event|
translated_event = {}
options[:content].each_pair do |key, value|
options['content'].each_pair do |key, value|
to_be_translated = Utils.values_at event.payload, value
translated_event[key] = translate to_be_translated.first, options[:to], access_token
translated_event[key] = translate to_be_translated.first, options['to'], access_token
end
create_event :payload => translated_event
end

48
app/models/agents/trigger_agent.rb
View file

@ -23,57 +23,57 @@ module Agents
MD
def validate_options
unless options[:expected_receive_period_in_days].present? && options[:message].present? && options[:rules].present? &&
options[:rules].all? { |rule| rule[:type].present? && VALID_COMPARISON_TYPES.include?(rule[:type]) && rule[:value].present? && rule[:path].present? }
unless options['expected_receive_period_in_days'].present? && options['message'].present? && options['rules'].present? &&
options['rules'].all? { |rule| rule['type'].present? && VALID_COMPARISON_TYPES.include?(rule['type']) && rule['value'].present? && rule['path'].present? }
errors.add(:base, "expected_receive_period_in_days, message, and rules, with a type, value, and path for every rule, are required")
end
end
def default_options
{
:expected_receive_period_in_days => "2",
:rules => [{
:type => "regex",
:value => "foo\\d+bar",
:path => "topkey.subkey.subkey.goal",
}],
:message => "Looks like your pattern matched in '<value>'!"
'expected_receive_period_in_days' => "2",
'rules' => [{
'type' => "regex",
'value' => "foo\\d+bar",
'path' => "topkey.subkey.subkey.goal",
}],
'message' => "Looks like your pattern matched in '<value>'!"
}
end
def working?
last_receive_at && last_receive_at > options[:expected_receive_period_in_days].to_i.days.ago && !recent_error_logs?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def receive(incoming_events)
incoming_events.each do |event|
match = options[:rules].all? do |rule|
value_at_path = Utils.value_at(event[:payload], rule[:path])
case rule[:type]
match = options['rules'].all? do |rule|
value_at_path = Utils.value_at(event['payload'], rule['path'])
case rule['type']
when "regex"
value_at_path.to_s =~ Regexp.new(rule[:value], Regexp::IGNORECASE)
value_at_path.to_s =~ Regexp.new(rule['value'], Regexp::IGNORECASE)
when "!regex"
value_at_path.to_s !~ Regexp.new(rule[:value], Regexp::IGNORECASE)
value_at_path.to_s !~ Regexp.new(rule['value'], Regexp::IGNORECASE)
when "field>value"
value_at_path.to_f > rule[:value].to_f
value_at_path.to_f > rule['value'].to_f
when "field>=value"
value_at_path.to_f >= rule[:value].to_f
value_at_path.to_f >= rule['value'].to_f
when "field<value"
value_at_path.to_f < rule[:value].to_f
value_at_path.to_f < rule['value'].to_f
when "field<=value"
value_at_path.to_f <= rule[:value].to_f
value_at_path.to_f <= rule['value'].to_f
when "field==value"
value_at_path.to_s == rule[:value].to_s
value_at_path.to_s == rule['value'].to_s
when "field!=value"
value_at_path.to_s != rule[:value].to_s
value_at_path.to_s != rule['value'].to_s
else
raise "Invalid :type of #{rule[:type]} in TriggerAgent##{id}"
raise "Invalid type of #{rule['type']} in TriggerAgent##{id}"
end
end
if match
create_event :payload => { :message => make_message(event[:payload]) } # Maybe this should include the
# original event as well?
create_event :payload => { 'message' => make_message(event[:payload]) } # Maybe this should include the
# original event as well?
end
end
end

50
app/models/agents/twilio_agent.rb
View file

@ -9,7 +9,7 @@ module Agents
description <<-MD
The TwilioAgent receives and collects events and sends them via text message or gives you a call when scheduled.
It is assumed that events have a `:message`, `:text`, or `:sms` key, the value of which is sent as the content of the text message/call. You can use Event Formatting Agent if your event does not provide these keys.
It is assumed that events have a `message`, `text`, or `sms` key, the value of which is sent as the content of the text message/call. You can use Event Formatting Agent if your event does not provide these keys.
Set `receiver_cell` to the number to receive text messages/call and `sender_cell` to the number sending them.
@ -22,35 +22,35 @@ module Agents
def default_options
{
:account_sid => 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
:auth_token => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
:sender_cell => 'xxxxxxxxxx',
:receiver_cell => 'xxxxxxxxxx',
:server_url => 'http://somename.com:3000',
:receive_text => 'true',
:receive_call => 'false',
:expected_receive_period_in_days => '1'
'account_sid' => 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
'auth_token' => 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
'sender_cell' => 'xxxxxxxxxx',
'receiver_cell' => 'xxxxxxxxxx',
'server_url' => 'http://somename.com:3000',
'receive_text' => 'true',
'receive_call' => 'false',
'expected_receive_period_in_days' => '1'
}
end
def validate_options
unless options[:account_sid].present? && options[:auth_token].present? && options[:sender_cell].present? && options[:receiver_cell].present? && options[:expected_receive_period_in_days].present? && options[:receive_call].present? && options[:receive_text].present?
unless options['account_sid'].present? && options['auth_token'].present? && options['sender_cell'].present? && options['receiver_cell'].present? && options['expected_receive_period_in_days'].present? && options['receive_call'].present? && options['receive_text'].present?
errors.add(:base, 'account_sid, auth_token, sender_cell, receiver_cell, receive_text, receive_call and expected_receive_period_in_days are all required')
end
end
def receive(incoming_events)
@client = Twilio::REST::Client.new options[:account_sid], options[:auth_token]
memory[:pending_calls] ||= {}
@client = Twilio::REST::Client.new options['account_sid'], options['auth_token']
memory['pending_calls'] ||= {}
incoming_events.each do |event|
message = (event.payload[:message] || event.payload[:text] || event.payload[:sms]).to_s
message = (event.payload['message'] || event.payload['text'] || event.payload['sms']).to_s
if message != ""
if options[:receive_call].to_s == 'true'
if options['receive_call'].to_s == 'true'
secret = SecureRandom.hex 3
memory[:pending_calls][secret] = message
memory['pending_calls'][secret] = message
make_call secret
end
if options[:receive_text].to_s == 'true'
if options['receive_text'].to_s == 'true'
message = message.slice 0..160
send_message message
end
@ -59,19 +59,19 @@ module Agents
end
def working?
last_receive_at && last_receive_at > options[:expected_receive_period_in_days].to_i.days.ago && !recent_error_logs?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def send_message(message)
@client.account.sms.messages.create :from => options[:sender_cell],
:to => options[:receiver_cell],
@client.account.sms.messages.create :from => options['sender_cell'],
:to => options['receiver_cell'],
:body => message
end
def make_call(secret)
@client.account.calls.create :from => options[:sender_cell],
:to => options[:receiver_cell],
:url => post_url(options[:server_url],secret)
@client.account.calls.create :from => options['sender_cell'],
:to => options['receiver_cell'],
:url => post_url(options['server_url'],secret)
end
def post_url(server_url,secret)
@ -79,9 +79,9 @@ module Agents
end
def receive_webhook(params)
if memory[:pending_calls].has_key? params[:secret].to_sym
response = Twilio::TwiML::Response.new {|r| r.Say memory[:pending_calls][params[:secret].to_sym], :voice => 'woman'}
memory[:pending_calls].delete params[:secret].to_sym
if memory['pending_calls'].has_key? params['secret']
response = Twilio::TwiML::Response.new {|r| r.Say memory['pending_calls'][params['secret']], :voice => 'woman'}
memory['pending_calls'].delete params['secret']
[response.text, 200]
end
end

40
app/models/agents/twitter_publish_agent.rb
View file

@ -19,25 +19,25 @@ module Agents
MD
def validate_options
unless options[:username].present? &&
options[:expected_update_period_in_days].present?
unless options['username'].present? &&
options['expected_update_period_in_days'].present?
errors.add(:base, "username and expected_update_period_in_days are required")
end
end
def working?
(event = event_created_within(options[:expected_update_period_in_days])) && event.payload[:success] == true && !recent_error_logs?
event_created_within?(options['expected_update_period_in_days']) && most_recent_event.payload['success'] == true && !recent_error_logs?
end
def default_options
{
:username => "",
:expected_update_period_in_days => "10",
:consumer_key => "---",
:consumer_secret => "---",
:oauth_token => "---",
:oauth_token_secret => "---",
:message_path => "text"
'username' => "",
'expected_update_period_in_days' => "10",
'consumer_key' => "---",
'consumer_secret' => "---",
'oauth_token' => "---",
'oauth_token_secret' => "---",
'message_path' => "text"
}
end
@ -47,22 +47,22 @@ module Agents
incoming_events = incoming_events.first(20)
end
incoming_events.each do |event|
tweet_text = Utils.value_at(event.payload, options[:message_path])
tweet_text = Utils.value_at(event.payload, options['message_path'])
begin
publish_tweet tweet_text
create_event :payload => {
:success => true,
:published_tweet => tweet_text,
:agent_id => event.agent_id,
:event_id => event.id
'success' => true,
'published_tweet' => tweet_text,
'agent_id' => event.agent_id,
'event_id' => event.id
}
rescue Twitter::Error => e
create_event :payload => {
:success => false,
:error => e.message,
:failed_tweet => tweet_text,
:agent_id => event.agent_id,
:event_id => event.id
'success' => false,
'error' => e.message,
'failed_tweet' => tweet_text,
'agent_id' => event.agent_id,
'event_id' => event.id
}
end
end

46
app/models/agents/twitter_stream_agent.rb
View file

@ -54,26 +54,26 @@ module Agents
default_schedule "11pm"
def validate_options
unless options[:filters].present? &&
options[:expected_update_period_in_days].present? &&
options[:generate].present?
unless options['filters'].present? &&
options['expected_update_period_in_days'].present? &&
options['generate'].present?
errors.add(:base, "expected_update_period_in_days, generate, and filters are required fields")
end
end
def working?
event_created_within(options[:expected_update_period_in_days]) && !recent_error_logs?
event_created_within?(options['expected_update_period_in_days']) && !recent_error_logs?
end
def default_options
{
:consumer_key => "---",
:consumer_secret => "---",
:oauth_token => "---",
:oauth_token_secret => "---",
:filters => %w[keyword1 keyword2],
:expected_update_period_in_days => "2",
:generate => "events"
'consumer_key' => "---",
'consumer_secret' => "---",
'oauth_token' => "---",
'oauth_token_secret' => "---",
'filters' => %w[keyword1 keyword2],
'expected_update_period_in_days' => "2",
'generate' => "events"
}
end
@ -81,33 +81,33 @@ module Agents
filter = lookup_filter(filter)
if filter
if options[:generate] == "counts"
if options['generate'] == "counts"
# Avoid memory pollution by reloading the Agent.
agent = Agent.find(id)
agent.memory[:filter_counts] ||= {}
agent.memory[:filter_counts][filter.to_sym] ||= 0
agent.memory[:filter_counts][filter.to_sym] += 1
remove_unused_keys!(agent, :filter_counts)
agent.memory['filter_counts'] ||= {}
agent.memory['filter_counts'][filter] ||= 0
agent.memory['filter_counts'][filter] += 1
remove_unused_keys!(agent, 'filter_counts')
agent.save!
else
create_event :payload => status.merge(:filter => filter.to_s)
create_event :payload => status.merge('filter' => filter)
end
end
end
def check
if options[:generate] == "counts" && memory[:filter_counts] && memory[:filter_counts].length > 0
memory[:filter_counts].each do |filter, count|
create_event :payload => { :filter => filter.to_s, :count => count, :time => Time.now.to_i }
if options['generate'] == "counts" && memory['filter_counts'] && memory['filter_counts'].length > 0
memory['filter_counts'].each do |filter, count|
create_event :payload => { 'filter' => filter, 'count' => count, 'time' => Time.now.to_i }
end
end
memory[:filter_counts] = {}
memory['filter_counts'] = {}
end
protected
def lookup_filter(filter)
options[:filters].each do |known_filter|
options['filters'].each do |known_filter|
if known_filter == filter
return filter
elsif known_filter.is_a?(Array)
@ -120,7 +120,7 @@ module Agents
def remove_unused_keys!(agent, base)
if agent.memory[base]
(agent.memory[base].keys - agent.options[:filters].map {|f| f.is_a?(Array) ? f.first.to_sym : f.to_sym }).each do |removed_key|
(agent.memory[base].keys - agent.options['filters'].map {|f| f.is_a?(Array) ? f.first.to_s : f.to_s }).each do |removed_key|
agent.memory[base].delete(removed_key)
end
end

24
app/models/agents/twitter_user_agent.rb
View file

@ -41,36 +41,36 @@ module Agents
default_schedule "every_1h"
def validate_options
unless options[:username].present? &&
options[:expected_update_period_in_days].present?
unless options['username'].present? &&
options['expected_update_period_in_days'].present?
errors.add(:base, "username and expected_update_period_in_days are required")
end
end
def working?
event_created_within(options[:expected_update_period_in_days]) && !recent_error_logs?
event_created_within?(options['expected_update_period_in_days']) && !recent_error_logs?
end
def default_options
{
:username => "tectonic",
:expected_update_period_in_days => "2",
:consumer_key => "---",
:consumer_secret => "---",
:oauth_token => "---",
:oauth_token_secret => "---"
'username' => "tectonic",
'expected_update_period_in_days' => "2",
'consumer_key' => "---",
'consumer_secret' => "---",
'oauth_token' => "---",
'oauth_token_secret' => "---"
}
end
def check
since_id = memory[:since_id] || nil
since_id = memory['since_id'] || nil
opts = {:count => 200, :include_rts => true, :exclude_replies => false, :include_entities => true, :contributor_details => true}
opts.merge! :since_id => since_id unless since_id.nil?
tweets = Twitter.user_timeline(options[:username], opts)
tweets = Twitter.user_timeline(options['username'], opts)
tweets.each do |tweet|
memory[:since_id] = tweet.id if !memory[:since_id] || (tweet.id > memory[:since_id])
memory['since_id'] = tweet.id if !memory['since_id'] || (tweet.id > memory['since_id'])
create_event :payload => tweet.attrs
end

6
app/models/agents/user_location_agent.rb
View file

@ -30,15 +30,15 @@ module Agents
MD
def working?
event_created_within(2) && !recent_error_logs?
event_created_within?(2) && !recent_error_logs?
end
def default_options
{ :secret => SecureRandom.hex(7) }
{ 'secret' => SecureRandom.hex(7) }
end
def validate_options
errors.add(:base, "secret is required and must be longer than 4 characters") unless options[:secret].present? && options[:secret].length > 4
errors.add(:base, "secret is required and must be longer than 4 characters") unless options['secret'].present? && options['secret'].length > 4
end
end
end

18
app/models/agents/weather_agent.rb
View file

@ -41,34 +41,34 @@ module Agents
default_schedule "8pm"
def working?
event_created_within(2) && !recent_error_logs?
event_created_within?(2) && !recent_error_logs?
end
def wunderground
Wunderground.new(options[:api_key]) if key_setup?
Wunderground.new(options['api_key']) if key_setup?
end
def key_setup?
options[:api_key] && options[:api_key] != "your-key"
options['api_key'] && options['api_key'] != "your-key"
end
def default_options
{
:api_key => "your-key",
:location => "94103"
'api_key' => "your-key",
'location' => "94103"
}
end
def validate_options
errors.add(:base, "location is required") unless options[:location].present? || options[:zipcode].present?
errors.add(:base, "api_key is required") unless options[:api_key].present?
errors.add(:base, "location is required") unless options['location'].present? || options['zipcode'].present?
errors.add(:base, "api_key is required") unless options['api_key'].present?
end
def check
if key_setup?
wunderground.forecast_for(options[:location] || options[:zipcode])["forecast"]["simpleforecast"]["forecastday"].each do |day|
wunderground.forecast_for(options['location'] || options['zipcode'])["forecast"]["simpleforecast"]["forecastday"].each do |day|
if is_tomorrow?(day)
create_event :payload => day.merge(:location => options[:location] || options[:zipcode])
create_event :payload => day.merge('location' => options['location'] || options['zipcode'])
end
end
end

16
app/models/agents/webhook_agent.rb
View file

@ -18,7 +18,7 @@ module Agents
* `secret` - A token that the host will provide for authentication.
* `expected_receive_period_in_days` - How often you expect to receive
events this way. Used to determine if the agent is working.
* `payload_path` - JSONPath of the attribute of the POST body to be
* `payload_path` - JSONPath of the attribute in the POST body to be
used as the Event payload.
MD
end
@ -26,7 +26,7 @@ module Agents
event_description do
<<-MD
The event payload is base on the value of the `payload_path` option,
which is set to `#{options[:payload_path]}`.
which is set to `#{options['payload_path']}`.
MD
end
@ -37,8 +37,8 @@ module Agents
end
def receive_webhook(params)
secret = params.delete(:secret)
return ["Not Authorized", 401] unless secret == options[:secret]
secret = params.delete('secret')
return ["Not Authorized", 401] unless secret == options['secret']
create_event(:payload => payload_for(params))
@ -46,17 +46,17 @@ module Agents
end
def working?
event_created_within(options[:expected_receive_period_in_days]) && !recent_error_logs?
event_created_within(options['expected_receive_period_in_days']) && !recent_error_logs?
end
def validate_options
unless options[:secret].present?
errors.add(:base, "Must specify a :secret for 'Authenticating' requests")
unless options['secret'].present?
errors.add(:base, "Must specify a secret for 'Authenticating' requests")
end
end
def payload_for(params)
Utils.values_at(params, options[:payload_path]) || {}
Utils.value_at(params, options['payload_path']) || {}
end
end
end

78
app/models/agents/website_agent.rb
View file

@ -15,19 +15,19 @@ module Agents
To tell the Agent how to parse the content, specify `extract` as a hash with keys naming the extractions and values of hashes.
When parsing HTML or XML, these sub-hashes specify how to extract with a `:css` CSS selector and either `:text => true` or `attr` pointing to an attribute name to grab. An example:
When parsing HTML or XML, these sub-hashes specify how to extract with a `css` CSS selector and either `'text': true` or `attr` pointing to an attribute name to grab. An example:
:extract => {
:url => { :css => "#comic img", :attr => "src" },
:title => { :css => "#comic img", :attr => "title" },
:body_text => { :css => "div.main", :text => true }
'extract': {
'url': { 'css': "#comic img", 'attr': "src" },
'title': { 'css': "#comic img", 'attr': "title" },
'body_text': { 'css': "div.main", 'text': true }
}
When parsing JSON, these sub-hashes specify [JSONPaths](http://goessner.net/articles/JsonPath/) to the values that you care about. For example:
:extract => {
:title => { :path => "results.data[*].title" },
:description => { :path => "results.data[*].description" }
'extract': {
'title': { 'path': "results.data[*].title" },
'description': { 'path': "results.data[*].description" }
}
Note that for all of the formats, whatever you extract MUST have the same number of matches for each extractor. E.g., if you're extracting rows, all extractors must match all rows. For generating CSS selectors, something like [SelectorGadget](http://selectorgadget.com) may be helpful.
@ -36,7 +36,7 @@ module Agents
MD
event_description do
"Events will have the fields you specified. Your options look like:\n\n #{Utils.pretty_print options[:extract]}"
"Events will have the fields you specified. Your options look like:\n\n #{Utils.pretty_print options['extract']}"
end
default_schedule "every_12h"
@ -44,33 +44,33 @@ module Agents
UNIQUENESS_LOOK_BACK = 30
def working?
event_created_within(options[:expected_update_period_in_days]) && !recent_error_logs?
event_created_within?(options['expected_update_period_in_days']) && !recent_error_logs?
end
def default_options
{
:expected_update_period_in_days => "2",
:url => "http://xkcd.com",
:type => "html",
:mode => :on_change,
:extract => {
:url => {:css => "#comic img", :attr => "src"},
:title => {:css => "#comic img", :attr => "title"}
'expected_update_period_in_days' => "2",
'url' => "http://xkcd.com",
'type' => "html",
'mode' => :on_change,
'extract' => {
'url' => {'css' => "#comic img", 'attr' => "src"},
'title' => {'css' => "#comic img", 'attr' => "title"}
}
}
end
def validate_options
errors.add(:base, "url and expected_update_period_in_days are required") unless options[:expected_update_period_in_days].present? && options[:url].present?
if !options[:extract].present? && extraction_type != "json"
errors.add(:base, "url and expected_update_period_in_days are required") unless options['expected_update_period_in_days'].present? && options['url'].present?
if !options['extract'].present? && extraction_type != "json"
errors.add(:base, "extract is required for all types except json")
end
end
def check
hydra = Typhoeus::Hydra.new
log "Fetching #{options[:url]}"
request = Typhoeus::Request.new(options[:url], :followlocation => true)
log "Fetching #{options['url']}"
request = Typhoeus::Request.new(options['url'], :followlocation => true)
request.on_failure do |response|
error "Failed: #{response.inspect}"
end
@ -85,37 +85,37 @@ module Agents
end
else
output = {}
options[:extract].each do |name, extraction_details|
options['extract'].each do |name, extraction_details|
result = if extraction_type == "json"
output[name] = Utils.values_at(doc, extraction_details[:path])
output[name] = Utils.values_at(doc, extraction_details['path'])
else
output[name] = doc.css(extraction_details[:css]).map { |node|
if extraction_details[:attr]
node.attr(extraction_details[:attr])
elsif extraction_details[:text]
output[name] = doc.css(extraction_details['css']).map { |node|
if extraction_details['attr']
node.attr(extraction_details['attr'])
elsif extraction_details['text']
node.text()
else
error ":attr or :text is required on HTML or XML extraction patterns"
error "'attr' or 'text' is required on HTML or XML extraction patterns"
return
end
}
end
log "Extracting #{extraction_type} at #{extraction_details[:path] || extraction_details[:css]}: #{result}"
log "Extracting #{extraction_type} at #{extraction_details['path'] || extraction_details['css']}: #{result}"
end
num_unique_lengths = options[:extract].keys.map { |name| output[name].length }.uniq
num_unique_lengths = options['extract'].keys.map { |name| output[name].length }.uniq
if num_unique_lengths.length != 1
error "Got an uneven number of matches for #{options[:name]}: #{options[:extract].inspect}"
error "Got an uneven number of matches for #{options['name']}: #{options['extract'].inspect}"
return
end
num_unique_lengths.first.times do |index|
result = {}
options[:extract].keys.each do |name|
options['extract'].keys.each do |name|
result[name] = output[name][index]
if name.to_s == 'url'
result[name] = URI.join(options[:url], result[name]).to_s if (result[name] =~ URI::DEFAULT_PARSER.regexp[:ABS_URI]).nil?
result[name] = URI.join(options['url'], result[name]).to_s if (result[name] =~ URI::DEFAULT_PARSER.regexp[:ABS_URI]).nil?
end
end
@ -133,22 +133,22 @@ module Agents
private
def store_payload? result
!options[:mode] || options[:mode].to_s == "all" || (options[:mode].to_s == "on_change" && !previous_payloads.include?(result.to_json))
!options['mode'] || options['mode'].to_s == "all" || (options['mode'].to_s == "on_change" && !previous_payloads.include?(result.to_json))
end
def previous_payloads
events.order("id desc").limit(UNIQUENESS_LOOK_BACK).pluck(:payload).map(&:to_json) if options[:mode].to_s == "on_change"
events.order("id desc").limit(UNIQUENESS_LOOK_BACK).pluck(:payload).map(&:to_json) if options['mode'].to_s == "on_change"
end
def extract_full_json?
(!options[:extract].present? && extraction_type == "json")
(!options['extract'].present? && extraction_type == "json")
end
def extraction_type
(options[:type] || begin
if options[:url] =~ /\.(rss|xml)$/i
(options['type'] || begin
if options['url'] =~ /\.(rss|xml)$/i
"xml"
elsif options[:url] =~ /\.json$/i
elsif options['url'] =~ /\.json$/i
"json"
else
"html"

38
app/models/agents/weibo_publish_agent.rb
View file

@ -20,24 +20,24 @@ module Agents
MD
def validate_options
unless options[:uid].present? &&
options[:expected_update_period_in_days].present?
unless options['uid'].present? &&
options['expected_update_period_in_days'].present?
errors.add(:base, "expected_update_period_in_days and uid are required")
end
end
def working?
(event = event_created_within(options[:expected_update_period_in_days])) && event.payload[:success] == true && !recent_error_logs?
event_created_within?(options['expected_update_period_in_days']) && most_recent_event.payload['success'] == true && !recent_error_logs?
end
def default_options
{
:uid => "",
:access_token => "---",
:app_key => "---",
:app_secret => "---",
:expected_update_period_in_days => "10",
:message_path => "text"
'uid' => "",
'access_token' => "---",
'app_key' => "---",
'app_secret' => "---",
'expected_update_period_in_days' => "10",
'message_path' => "text"
}
end
@ -47,25 +47,25 @@ module Agents
incoming_events = incoming_events.first(20)
end
incoming_events.each do |event|
tweet_text = Utils.value_at(event.payload, options[:message_path])
tweet_text = Utils.value_at(event.payload, options['message_path'])
if event.agent.type == "Agents::TwitterUserAgent"
tweet_text = unwrap_tco_urls(tweet_text, event.payload)
end
begin
publish_tweet tweet_text
create_event :payload => {
:success => true,
:published_tweet => tweet_text,
:agent_id => event.agent_id,
:event_id => event.id
'success' => true,
'published_tweet' => tweet_text,
'agent_id' => event.agent_id,
'event_id' => event.id
}
rescue OAuth2::Error => e
create_event :payload => {
:success => false,
:error => e.message,
:failed_tweet => tweet_text,
:agent_id => event.agent_id,
:event_id => event.id
'success' => false,
'error' => e.message,
'failed_tweet' => tweet_text,
'agent_id' => event.agent_id,
'event_id' => event.id
}
end
end

22
app/models/agents/weibo_user_agent.rb
View file

@ -70,29 +70,29 @@ module Agents
default_schedule "every_1h"
def validate_options
unless options[:uid].present? &&
options[:expected_update_period_in_days].present?
unless options['uid'].present? &&
options['expected_update_period_in_days'].present?
errors.add(:base, "expected_update_period_in_days and uid are required")
end
end
def working?
event_created_within(options[:expected_update_period_in_days]) && !recent_error_logs?
event_created_within?(options['expected_update_period_in_days']) && !recent_error_logs?
end
def default_options
{
:uid => "",
:access_token => "---",
:app_key => "---",
:app_secret => "---",
:expected_update_period_in_days => "2"
'uid' => "",
'access_token' => "---",
'app_key' => "---",
'app_secret' => "---",
'expected_update_period_in_days' => "2"
}
end
def check
since_id = memory[:since_id] || nil
opts = {:uid => options[:uid].to_i}
since_id = memory['since_id'] || nil
opts = {:uid => options['uid'].to_i}
opts.merge! :since_id => since_id unless since_id.nil?
# http://open.weibo.com/wiki/2/statuses/user_timeline/en
@ -101,7 +101,7 @@ module Agents
resp[:statuses].each do |status|
memory[:since_id] = status.id if !memory[:since_id] || (status.id > memory[:since_id])
memory['since_id'] = status.id if !memory['since_id'] || (status.id > memory['since_id'])
create_event :payload => status.as_json
end

14
app/models/event.rb
View file

@ -1,23 +1,21 @@
require 'json_serialized_field'
class Event < ActiveRecord::Base
include JSONSerializedField
attr_accessible :lat, :lng, :payload, :user_id, :user, :expires_at
acts_as_mappable
serialize :payload
json_serialize :payload
belongs_to :user
belongs_to :agent, :counter_cache => true
before_save :symbolize_payload
belongs_to :agent, :counter_cache => true, :touch => :last_event_at
scope :recent, lambda { |timespan = 12.hours.ago|
where("events.created_at > ?", timespan)
}
def symbolize_payload
self.payload = payload.recursively_symbolize_keys if payload.is_a?(Hash)
end
def reemit!
agent.create_event :payload => payload, :lat => lat, :lng => lng
end

2
app/views/agents/show.html.erb
View file

@ -11,7 +11,7 @@
<li class='disabled'><a><i class='icon-picture'></i> Summary</a></li>
<li class='active'><a href="#details" data-toggle="tab"><i class='icon-indent-left'></i> Details</a></li>
<% end %>
<li><a href="#logs" data-toggle="tab" data-agent-id="<%= @agent.id %>"><i class='icon-list-alt'></i> Logs</a></li>
<li><a href="#logs" data-toggle="tab" data-agent-id="<%= @agent.id %>" class='<%= @agent.recent_error_logs? ? 'recent-errors' : '' %>'><i class='icon-list-alt'></i> Logs</a></li>
<% if @agent.can_create_events? && @agent.events.count > 0 %>
<li><%= link_to '<i class="icon-random"></i> Events'.html_safe, events_path(:agent => @agent.to_param) %></li>

2
app/views/layouts/application.html.erb
View file

@ -36,7 +36,7 @@
<script>
var agentPaths = {};
<% if current_user -%>
var myAgents = <%= Utils.jsonify(current_user.agents.inject({}) {|m, a| m[a.name] = agent_path(a) unless a.new_record?; m }) %>;
var myAgents = <%= Utils.jsonify(current_user.agents.select([:name, :id, :schedule]).inject({}) {|m, a| m[a.name] = agent_path(a); m }) %>;
$.extend(agentPaths, myAgents);
<% end -%>
agentPaths["All Agents Index"] = <%= Utils.jsonify agents_path %>;

7
config/initializers/recursively_symbolize_keys.rb
View file

@ -1,7 +0,0 @@
require 'utils'
class Hash
def recursively_symbolize_keys
Utils.recursively_symbolize_keys self
end
end

67
db/migrate/20131223032112_switch_to_json_serialization.rb Normal file
View file

@ -0,0 +1,67 @@
class SwitchToJsonSerialization < ActiveRecord::Migration
FIELDS = {
:agents => [:options, :memory],
:events => [:payload]
}
def up
if data_exists?
puts "This migration will update tables to use UTF-8 encoding and will update Agent and Event storage from YAML to JSON."
puts "It should work, but please make a backup before proceeding!"
print "Continue? (y/n) "
STDOUT.flush
exit unless STDIN.gets =~ /^y/i
set_to_utf8
translate YAML, JSON
end
end
def down
if data_exists?
translate JSON, YAML
end
end
def set_to_utf8
if mysql?
%w[agent_logs agents delayed_jobs events links users].each do |table_name|
quoted_table_name = ActiveRecord::Base.connection.quote_table_name(table_name)
execute "ALTER TABLE #{quoted_table_name} CONVERT TO CHARACTER SET utf8"
end
end
end
def mysql?
ActiveRecord::Base.connection.adapter_name =~ /mysql/i
end
def data_exists?
events = ActiveRecord::Base.connection.select_rows("SELECT count(*) FROM #{ActiveRecord::Base.connection.quote_table_name("events")}").first.first
agents = ActiveRecord::Base.connection.select_rows("SELECT count(*) FROM #{ActiveRecord::Base.connection.quote_table_name("agents")}").first.first
agents + events > 0
end
def translate(from, to)
FIELDS.each do |table, fields|
quoted_table_name = ActiveRecord::Base.connection.quote_table_name(table)
fields = fields.map { |f| ActiveRecord::Base.connection.quote_column_name(f) }
rows = ActiveRecord::Base.connection.select_rows("SELECT id, #{fields.join(", ")} FROM #{quoted_table_name}")
rows.each do |row|
id, *field_data = row
yaml_fields = field_data.map { |f| from.load(f) }.map { |f| to.dump(f) }
yaml_fields.map! {|f| f.encode('utf-8', 'binary', invalid: :replace, undef: :replace, replace: '??') }
update_sql = "UPDATE #{quoted_table_name} SET #{fields.map {|f| "#{f}=?"}.join(", ")} WHERE id = ?"
sanitized_update_sql = ActiveRecord::Base.send :sanitize_sql_array, [update_sql, *yaml_fields, id]
ActiveRecord::Base.connection.execute sanitized_update_sql
end
end
end
end

14
db/migrate/20131227000021_add_cached_dates_to_agent.rb Normal file
View file

@ -0,0 +1,14 @@
class AddCachedDatesToAgent < ActiveRecord::Migration
def up
add_column :agents, :last_event_at, :datetime
execute "UPDATE agents SET last_event_at = (SELECT created_at FROM events WHERE events.agent_id = agents.id ORDER BY id DESC LIMIT 1)"
add_column :agents, :last_error_log_at, :datetime
execute "UPDATE agents SET last_error_log_at = (SELECT created_at FROM agent_logs WHERE agent_logs.agent_id = agents.id AND agent_logs.level >= 4 ORDER BY id DESC LIMIT 1)"
end
def down
remove_column :agents, :last_event_at
remove_column :agents, :last_error_log_at
end
end

42
lib/json_serialized_field.rb Normal file
View file

@ -0,0 +1,42 @@
require 'json_with_indifferent_access'
module JSONSerializedField
extend ActiveSupport::Concern
module ClassMethods
def json_serialize(*fields)
fields.each do |field|
class_eval <<-CODE
serialize :#{field}, JSONWithIndifferentAccess
validate :#{field}_has_no_errors
def #{field}=(input)
@#{field}_assignment_error = false
case input
when String
if input.strip.length == 0
self[:#{field}] = ActiveSupport::HashWithIndifferentAccess.new
else
json = JSON.parse(input) rescue nil
if json
self[:#{field}] = ActiveSupport::HashWithIndifferentAccess.new(json)
else
@#{field}_assignment_error = "was assigned invalid JSON"
end
end
when Hash
self[:#{field}] = ActiveSupport::HashWithIndifferentAccess.new(input)
else
@#{field}_assignment_error = "cannot be set to an instance of \#{input.class}"
end
end
def #{field}_has_no_errors
errors.add(:#{field}, @#{field}_assignment_error) if @#{field}_assignment_error
end
CODE
end
end
end
end

9
lib/json_with_indifferent_access.rb Normal file
View file

@ -0,0 +1,9 @@
class JSONWithIndifferentAccess
def self.load(json)
ActiveSupport::HashWithIndifferentAccess.new(JSON.parse(json || '{}'))
end
def self.dump(hash)
JSON.dump(hash)
end
end

43
lib/serialize_and_symbolize.rb
View file

@ -1,43 +0,0 @@
module SerializeAndSymbolize
extend ActiveSupport::Concern
module ClassMethods
def serialize_and_symbolize(*column_names)
column_names.flatten.uniq.compact.map(&:to_sym).each do |column_name|
setup_name = "setup_#{column_name}".to_sym
symbolize_name = "symbolize_#{column_name}".to_sym
validate_name = "validate_#{column_name}".to_sym
serialize column_name
after_initialize setup_name
before_validation symbolize_name
before_save symbolize_name
validate validate_name
class_eval <<-RUBY
def #{setup_name}
self[:#{column_name}] ||= {}
end
def #{validate_name}
# Implement me in your subclass.
end
def #{symbolize_name}
self.#{column_name} = self[:#{column_name}]
end
def #{column_name}=(data)
if data.is_a?(String)
self[:#{column_name}] = JSON.parse(data).recursively_symbolize_keys rescue {}
elsif data.is_a?(Hash)
self[:#{column_name}] = data.recursively_symbolize_keys
else
self[:#{column_name}] = data
end
end
RUBY
end
end
end
end

11
lib/utils.rb
View file

@ -21,17 +21,6 @@ module Utils
end
end
def self.recursively_symbolize_keys(object)
case object
when Hash
object.inject({}) {|memo, (k, v)| memo[String === k ? k.to_sym : k] = recursively_symbolize_keys(v); memo }
when Array
object.map { |item| recursively_symbolize_keys item }
else
object
end
end
def self.interpolate_jsonpaths(value, data)
value.gsub(/<[^>]+>/).each { |jsonpath|
Utils.values_at(data, jsonpath[1..-2]).first.to_s

4
spec/controllers/agents_controller_spec.rb
View file

@ -5,7 +5,7 @@ describe AgentsController do
{
:type => "Agents::WebsiteAgent",
:name => "Something",
:options => agents(:bob_website_agent).options.to_json,
:options => agents(:bob_website_agent).options,
:source_ids => [agents(:bob_weather_agent).id, ""]
}.merge(options)
end
@ -23,7 +23,7 @@ describe AgentsController do
sign_in users(:bob)
post :handle_details_post, :id => agents(:bob_manual_event_agent).to_param, :payload => { :foo => "bar" }
JSON.parse(response.body).should == { "success" => true }
agents(:bob_manual_event_agent).events.last.payload.should == { :foo => "bar" }
agents(:bob_manual_event_agent).events.last.payload.should == { 'foo' => "bar" }
end
it "can only be accessed by the Agent's owner" do

4
spec/controllers/logs_controller_spec.rb
View file

@ -19,12 +19,14 @@ describe LogsController do
describe "DELETE clear" do
it "deletes all logs for a specific Agent" do
agents(:bob_weather_agent).last_error_log_at = 2.hours.ago
sign_in users(:bob)
lambda {
delete :clear, :agent_id => agents(:bob_weather_agent).id
}.should change { AgentLog.count }.by(-1 * agents(:bob_weather_agent).logs.count)
assigns(:logs).length.should == 0
agents(:bob_weather_agent).logs.count.should == 0
agents(:bob_weather_agent).reload.logs.count.should == 0
agents(:bob_weather_agent).last_error_log_at.should be_nil
end
it "only deletes logs for an Agent owned by the current user" do

6
spec/controllers/user_location_updates_controller_spec.rb
View file

@ -8,7 +8,7 @@ describe UserLocationUpdatesController do
it "should create events without requiring login" do
post :create, :user_id => users(:bob).to_param, :secret => "my_secret", :longitude => 123, :latitude => 45, :something => "else"
@agent.events.last.payload.should == { :longitude => "123", :latitude => "45", :something => "else" }
@agent.events.last.payload.should == { 'longitude' => "123", 'latitude' => "45", 'something' => "else" }
@agent.events.last.lat.should == 45
@agent.events.last.lng.should == 123
end
@ -18,7 +18,7 @@ describe UserLocationUpdatesController do
@jane_agent.save!
post :create, :user_id => users(:bob).to_param, :secret => "my_secret", :longitude => 123, :latitude => 45, :something => "else"
@agent.events.last.payload.should == { :longitude => "123", :latitude => "45", :something => "else" }
@agent.events.last.payload.should == { 'longitude' => "123", 'latitude' => "45", 'something' => "else" }
@jane_agent.events.should be_empty
end
@ -33,7 +33,7 @@ describe UserLocationUpdatesController do
lambda {
post :create, :user_id => users(:bob).to_param, :secret => "my_secret2", :longitude => 123, :latitude => 45, :something => "else"
@agent2.events.last.payload.should == { :longitude => "123", :latitude => "45", :something => "else" }
@agent2.events.last.payload.should == { 'longitude' => "123", 'latitude' => "45", 'something' => "else" }
}.should_not change { @agent.events.count }
end
end

4
spec/controllers/webhooks_controller_spec.rb
View file

@ -32,12 +32,12 @@ describe WebhooksController do
it "should call receive_webhook" do
post :create, :user_id => users(:bob).to_param, :agent_id => @agent.id, :secret => "my_secret", :key => "value", :another_key => "5"
@agent.reload.memory[:webhook_values].should == { :key => "value", :another_key => "5" }
@agent.reload.memory[:webhook_values].should == { 'key' => "value", 'another_key' => "5" }
response.body.should == "success"
response.should be_success
post :create, :user_id => users(:bob).to_param, :agent_id => @agent.id, :secret => "not_my_secret", :no => "go"
@agent.reload.memory[:webhook_values].should_not == { :no => "go" }
@agent.reload.memory[:webhook_values].should_not == { 'no' => "go" }
response.body.should == "failure"
response.should be_missing
end

14
spec/fixtures/agents.yml vendored
View file

@ -11,7 +11,7 @@ jane_website_agent:
:title => {:css => "item title", :text => true},
:url => {:css => "item link", :text => true}
}
}.to_yaml.inspect %>
}.to_json.inspect %>
bob_website_agent:
type: Agents::WebsiteAgent
@ -26,7 +26,7 @@ bob_website_agent:
:url => {:css => "#comic img", :attr => "src"},
:title => {:css => "#comic img", :attr => "title"}
}
}.to_yaml.inspect %>
}.to_json.inspect %>
bob_weather_agent:
type: Agents::WeatherAgent
@ -38,7 +38,7 @@ bob_weather_agent:
:lat => 37.779329,
:lng => -122.41915,
:api_key => 'test'
}.to_yaml.inspect %>
}.to_json.inspect %>
jane_weather_agent:
type: Agents::WeatherAgent
@ -50,7 +50,7 @@ jane_weather_agent:
:lat => 37.779329,
:lng => -122.41915,
:api_key => 'test'
}.to_yaml.inspect %>
}.to_json.inspect %>
jane_rain_notifier_agent:
type: Agents::TriggerAgent
@ -64,7 +64,7 @@ jane_rain_notifier_agent:
:path => "conditions"
}],
:message => "Just so you know, it looks like '<conditions>' tomorrow in <location>"
}.to_yaml.inspect %>
}.to_json.inspect %>
bob_rain_notifier_agent:
type: Agents::TriggerAgent
@ -78,7 +78,7 @@ bob_rain_notifier_agent:
:path => "conditions"
}],
:message => "Just so you know, it looks like '<conditions>' tomorrow in <location>"
}.to_yaml.inspect %>
}.to_json.inspect %>
bob_twitter_user_agent:
type: Agents::TwitterUserAgent
@ -91,7 +91,7 @@ bob_twitter_user_agent:
:consumer_secret => "---",
:oauth_token => "---",
:oauth_token_secret => "---"
}.to_yaml.inspect %>
}.to_json.inspect %>
bob_manual_event_agent:
type: Agents::ManualEventAgent

4
spec/fixtures/events.yml vendored
View file

@ -1,9 +1,9 @@
bob_website_agent_event:
user: bob
agent: bob_website_agent
payload: <%= [{ :title => "foo", :url => "http://foo.com" }].to_yaml.inspect %>
payload: <%= [{ :title => "foo", :url => "http://foo.com" }].to_json.inspect %>
jane_website_agent_event:
user: jane
agent: jane_website_agent
payload: <%= [{ :title => "foo", :url => "http://foo.com" }].to_yaml.inspect %>
payload: <%= [{ :title => "foo", :url => "http://foo.com" }].to_json.inspect %>

8
spec/models/agent_log_spec.rb
View file

@ -67,6 +67,14 @@ describe AgentLog do
agents(:jane_website_agent).logs.order("agent_logs.id desc").first.message.should == "message 6"
agents(:jane_website_agent).logs.order("agent_logs.id desc").last.message.should == "message 3"
end
it "updates Agents' last_error_log_at when an error is logged" do
AgentLog.log_for_agent(agents(:jane_website_agent), "some message", :level => 3, :outbound_event => events(:jane_website_agent_event))
agents(:jane_website_agent).reload.last_error_log_at.should be_nil
AgentLog.log_for_agent(agents(:jane_website_agent), "some message", :level => 4, :outbound_event => events(:jane_website_agent_event))
agents(:jane_website_agent).reload.last_error_log_at.to_i.should be_within(2).of(Time.now.to_i)
end
end
describe "#log_length" do

72
spec/models/agent_spec.rb
View file

@ -261,9 +261,51 @@ describe Agent do
it "symbolizes memory before validating" do
agent = Agents::SomethingSource.new(:name => "something")
agent.user = users(:bob)
agent.memory["bad"] = :hello
agent.memory["bad"] = 2
agent.save
agent.memory[:bad].should == :hello
agent.memory[:bad].should == 2
end
it "should work when assigned a hash or JSON string" do
agent = Agents::SomethingSource.new(:name => "something")
agent.memory = {}
agent.memory.should == {}
agent.memory["foo"].should be_nil
agent.memory = ""
agent.memory["foo"].should be_nil
agent.memory.should == {}
agent.memory = '{"hi": "there"}'
agent.memory.should == { "hi" => "there" }
agent.memory = '{invalid}'
agent.memory.should == { "hi" => "there" }
agent.should have(1).errors_on(:memory)
agent.memory = "{}"
agent.memory["foo"].should be_nil
agent.memory.should == {}
agent.should have(0).errors_on(:memory)
agent.options = "{}"
agent.options["foo"].should be_nil
agent.options.should == {}
agent.should have(0).errors_on(:options)
agent.options = '{"hi": 2}'
agent.options["hi"].should == 2
agent.should have(0).errors_on(:options)
agent.options = '{"hi": wut}'
agent.options["hi"].should == 2
agent.should have(1).errors_on(:options)
agent.errors_on(:options).should include("was assigned invalid JSON")
agent.options = 5
agent.options["hi"].should == 2
agent.should have(1).errors_on(:options)
agent.errors_on(:options).should include("cannot be set to an instance of Fixnum")
end
it "should not allow agents owned by other people" do
@ -279,6 +321,32 @@ describe Agent do
end
end
describe "recent_error_logs?" do
it "returns true if last_error_log_at is near last_event_at" do
agent = Agent.new
agent.last_error_log_at = 10.minutes.ago
agent.last_event_at = 10.minutes.ago
agent.recent_error_logs?.should be_true
agent.last_error_log_at = 11.minutes.ago
agent.last_event_at = 10.minutes.ago
agent.recent_error_logs?.should be_true
agent.last_error_log_at = 5.minutes.ago
agent.last_event_at = 10.minutes.ago
agent.recent_error_logs?.should be_true
agent.last_error_log_at = 15.minutes.ago
agent.last_event_at = 10.minutes.ago
agent.recent_error_logs?.should be_false
agent.last_error_log_at = 2.days.ago
agent.last_event_at = 10.minutes.ago
agent.recent_error_logs?.should be_false
end
end
describe "scopes" do
describe "of_type" do
it "should accept classes" do

10
spec/models/agents/digest_email_agent_spec.rb
View file

@ -19,16 +19,16 @@ describe Agents::DigestEmailAgent do
it "queues any payloads it receives" do
event1 = Event.new
event1.agent = agents(:bob_rain_notifier_agent)
event1.payload = "Something you should know about"
event1.payload = { :data => "Something you should know about" }
event1.save!
event2 = Event.new
event2.agent = agents(:bob_weather_agent)
event2.payload = "Something else you should know about"
event2.payload = { :data => "Something else you should know about" }
event2.save!
Agents::DigestEmailAgent.async_receive(@checker.id, [event1.id, event2.id])
@checker.reload.memory[:queue].should == ["Something you should know about", "Something else you should know about"]
@checker.reload.memory[:queue].should == [{ 'data' => "Something you should know about" }, { 'data' => "Something else you should know about" }]
end
end
@ -37,7 +37,7 @@ describe Agents::DigestEmailAgent do
Agents::DigestEmailAgent.async_check(@checker.id)
ActionMailer::Base.deliveries.should == []
@checker.memory[:queue] = ["Something you should know about",
@checker.memory[:queue] = [{ :data => "Something you should know about" },
{ :title => "Foo", :url => "http://google.com", :bar => 2 },
{ "message" => "hi", :woah => "there" },
{ "test" => 2 }]
@ -47,7 +47,7 @@ describe Agents::DigestEmailAgent do
Agents::DigestEmailAgent.async_check(@checker.id)
ActionMailer::Base.deliveries.last.to.should == ["bob@example.com"]
ActionMailer::Base.deliveries.last.subject.should == "something interesting"
get_message_part(ActionMailer::Base.deliveries.last, /plain/).strip.should == "Something you should know about\n\nFoo\n bar: 2\n url: http://google.com\n\nhi\n woah: there\n\nEvent\n test: 2"
get_message_part(ActionMailer::Base.deliveries.last, /plain/).strip.should == "Event\n data: Something you should know about\n\nFoo\n bar: 2\n url: http://google.com\n\nhi\n woah: there\n\nEvent\n test: 2"
@checker.reload.memory[:queue].should be_empty
end

8
spec/models/agents/email_agent_spec.rb
View file

@ -21,12 +21,12 @@ describe Agents::EmailAgent do
event1 = Event.new
event1.agent = agents(:bob_rain_notifier_agent)
event1.payload = "Something you should know about"
event1.payload = { :data => "Something you should know about" }
event1.save!
event2 = Event.new
event2.agent = agents(:bob_weather_agent)
event2.payload = "Something else you should know about"
event2.payload = { :data => "Something else you should know about" }
event2.save!
Agents::EmailAgent.async_receive(@checker.id, [event1.id])
@ -35,8 +35,8 @@ describe Agents::EmailAgent do
ActionMailer::Base.deliveries.count.should == 2
ActionMailer::Base.deliveries.last.to.should == ["bob@example.com"]
ActionMailer::Base.deliveries.last.subject.should == "something interesting"
get_message_part(ActionMailer::Base.deliveries.last, /plain/).strip.should == "Something else you should know about"
get_message_part(ActionMailer::Base.deliveries.first, /plain/).strip.should == "Something you should know about"
get_message_part(ActionMailer::Base.deliveries.last, /plain/).strip.should == "Event\n data: Something else you should know about"
get_message_part(ActionMailer::Base.deliveries.first, /plain/).strip.should == "Event\n data: Something you should know about"
end
it "can receive complex events and send them on" do

324
spec/models/agents/human_task_agent_spec.rb
View file

@ -9,8 +9,8 @@ describe Agents::HumanTaskAgent do
@event = Event.new
@event.agent = agents(:bob_rain_notifier_agent)
@event.payload = { :foo => { "bar" => { :baz => "a2b" } },
:name => "Joe" }
@event.payload = { 'foo' => { "bar" => { 'baz' => "a2b" } },
'name' => "Joe" }
@event.id = 345
@checker.should be_valid
@ -18,146 +18,146 @@ describe Agents::HumanTaskAgent do
describe "validations" do
it "validates that trigger_on is 'schedule' or 'event'" do
@checker.options[:trigger_on] = "foo"
@checker.options['trigger_on'] = "foo"
@checker.should_not be_valid
end
it "requires expected_receive_period_in_days when trigger_on is set to 'event'" do
@checker.options[:trigger_on] = "event"
@checker.options[:expected_receive_period_in_days] = nil
@checker.options['trigger_on'] = "event"
@checker.options['expected_receive_period_in_days'] = nil
@checker.should_not be_valid
@checker.options[:expected_receive_period_in_days] = 2
@checker.options['expected_receive_period_in_days'] = 2
@checker.should be_valid
end
it "requires a positive submission_period when trigger_on is set to 'schedule'" do
@checker.options[:trigger_on] = "schedule"
@checker.options[:submission_period] = nil
@checker.options['trigger_on'] = "schedule"
@checker.options['submission_period'] = nil
@checker.should_not be_valid
@checker.options[:submission_period] = 2
@checker.options['submission_period'] = 2
@checker.should be_valid
end
it "requires a hit.title" do
@checker.options[:hit][:title] = ""
@checker.options['hit']['title'] = ""
@checker.should_not be_valid
end
it "requires a hit.description" do
@checker.options[:hit][:description] = ""
@checker.options['hit']['description'] = ""
@checker.should_not be_valid
end
it "requires hit.assignments" do
@checker.options[:hit][:assignments] = ""
@checker.options['hit']['assignments'] = ""
@checker.should_not be_valid
@checker.options[:hit][:assignments] = 0
@checker.options['hit']['assignments'] = 0
@checker.should_not be_valid
@checker.options[:hit][:assignments] = "moose"
@checker.options['hit']['assignments'] = "moose"
@checker.should_not be_valid
@checker.options[:hit][:assignments] = "2"
@checker.options['hit']['assignments'] = "2"
@checker.should be_valid
end
it "requires hit.questions" do
old_questions = @checker.options[:hit][:questions]
@checker.options[:hit][:questions] = nil
old_questions = @checker.options['hit']['questions']
@checker.options['hit']['questions'] = nil
@checker.should_not be_valid
@checker.options[:hit][:questions] = []
@checker.options['hit']['questions'] = []
@checker.should_not be_valid
@checker.options[:hit][:questions] = [old_questions[0]]
@checker.options['hit']['questions'] = [old_questions[0]]
@checker.should be_valid
end
it "requires that all questions have key, name, required, type, and question" do
old_questions = @checker.options[:hit][:questions]
@checker.options[:hit][:questions].first[:key] = ""
old_questions = @checker.options['hit']['questions']
@checker.options['hit']['questions'].first['key'] = ""
@checker.should_not be_valid
@checker.options[:hit][:questions] = old_questions
@checker.options[:hit][:questions].first[:name] = ""
@checker.options['hit']['questions'] = old_questions
@checker.options['hit']['questions'].first['name'] = ""
@checker.should_not be_valid
@checker.options[:hit][:questions] = old_questions
@checker.options[:hit][:questions].first[:required] = nil
@checker.options['hit']['questions'] = old_questions
@checker.options['hit']['questions'].first['required'] = nil
@checker.should_not be_valid
@checker.options[:hit][:questions] = old_questions
@checker.options[:hit][:questions].first[:type] = ""
@checker.options['hit']['questions'] = old_questions
@checker.options['hit']['questions'].first['type'] = ""
@checker.should_not be_valid
@checker.options[:hit][:questions] = old_questions
@checker.options[:hit][:questions].first[:question] = ""
@checker.options['hit']['questions'] = old_questions
@checker.options['hit']['questions'].first['question'] = ""
@checker.should_not be_valid
end
it "requires that all questions of type 'selection' have a selections array with keys and text" do
@checker.options[:hit][:questions][0][:selections] = []
@checker.options['hit']['questions'][0]['selections'] = []
@checker.should_not be_valid
@checker.options[:hit][:questions][0][:selections] = [{}]
@checker.options['hit']['questions'][0]['selections'] = [{}]
@checker.should_not be_valid
@checker.options[:hit][:questions][0][:selections] = [{ :key => "", :text => "" }]
@checker.options['hit']['questions'][0]['selections'] = [{ 'key' => "", 'text' => "" }]
@checker.should_not be_valid
@checker.options[:hit][:questions][0][:selections] = [{ :key => "", :text => "hi" }]
@checker.options['hit']['questions'][0]['selections'] = [{ 'key' => "", 'text' => "hi" }]
@checker.should_not be_valid
@checker.options[:hit][:questions][0][:selections] = [{ :key => "hi", :text => "" }]
@checker.options['hit']['questions'][0]['selections'] = [{ 'key' => "hi", 'text' => "" }]
@checker.should_not be_valid
@checker.options[:hit][:questions][0][:selections] = [{ :key => "hi", :text => "hi" }]
@checker.options['hit']['questions'][0]['selections'] = [{ 'key' => "hi", 'text' => "hi" }]
@checker.should be_valid
@checker.options[:hit][:questions][0][:selections] = [{ :key => "hi", :text => "hi" }, {}]
@checker.options['hit']['questions'][0]['selections'] = [{ 'key' => "hi", 'text' => "hi" }, {}]
@checker.should_not be_valid
end
it "requires that 'poll_options' be present and populated when 'combination_mode' is set to 'poll'" do
@checker.options[:combination_mode] = "poll"
@checker.options['combination_mode'] = "poll"
@checker.should_not be_valid
@checker.options[:poll_options] = {}
@checker.options['poll_options'] = {}
@checker.should_not be_valid
@checker.options[:poll_options] = { :title => "Take a poll about jokes",
:instructions => "Rank these by how funny they are",
:assignments => 3,
:row_template => "<$.joke>" }
@checker.options['poll_options'] = { 'title' => "Take a poll about jokes",
'instructions' => "Rank these by how funny they are",
'assignments' => 3,
'row_template' => "<$.joke>" }
@checker.should be_valid
@checker.options[:poll_options] = { :instructions => "Rank these by how funny they are",
:assignments => 3,
:row_template => "<$.joke>" }
@checker.options['poll_options'] = { 'instructions' => "Rank these by how funny they are",
'assignments' => 3,
'row_template' => "<$.joke>" }
@checker.should_not be_valid
@checker.options[:poll_options] = { :title => "Take a poll about jokes",
:assignments => 3,
:row_template => "<$.joke>" }
@checker.options['poll_options'] = { 'title' => "Take a poll about jokes",
'assignments' => 3,
'row_template' => "<$.joke>" }
@checker.should_not be_valid
@checker.options[:poll_options] = { :title => "Take a poll about jokes",
:instructions => "Rank these by how funny they are",
:row_template => "<$.joke>" }
@checker.options['poll_options'] = { 'title' => "Take a poll about jokes",
'instructions' => "Rank these by how funny they are",
'row_template' => "<$.joke>" }
@checker.should_not be_valid
@checker.options[:poll_options] = { :title => "Take a poll about jokes",
:instructions => "Rank these by how funny they are",
:assignments => 3}
@checker.options['poll_options'] = { 'title' => "Take a poll about jokes",
'instructions' => "Rank these by how funny they are",
'assignments' => 3}
@checker.should_not be_valid
end
it "requires that all questions be of type 'selection' when 'combination_mode' is 'take_majority'" do
@checker.options[:combination_mode] = "take_majority"
@checker.options['combination_mode'] = "take_majority"
@checker.should_not be_valid
@checker.options[:hit][:questions][1][:type] = "selection"
@checker.options[:hit][:questions][1][:selections] = @checker.options[:hit][:questions][0][:selections]
@checker.options['hit']['questions'][1]['type'] = "selection"
@checker.options['hit']['questions'][1]['selections'] = @checker.options['hit']['questions'][0]['selections']
@checker.should be_valid
end
it "accepts 'take_majority': 'true' for legacy support" do
@checker.options[:take_majority] = "true"
@checker.options['take_majority'] = "true"
@checker.should_not be_valid
@checker.options[:hit][:questions][1][:type] = "selection"
@checker.options[:hit][:questions][1][:selections] = @checker.options[:hit][:questions][0][:selections]
@checker.options['hit']['questions'][1]['type'] = "selection"
@checker.options['hit']['questions'][1]['selections'] = @checker.options['hit']['questions'][0]['selections']
@checker.should be_valid
end
end
describe "when 'trigger_on' is set to 'schedule'" do
before do
@checker.options[:trigger_on] = "schedule"
@checker.options[:submission_period] = "2"
@checker.options.delete(:expected_receive_period_in_days)
@checker.options['trigger_on'] = "schedule"
@checker.options['submission_period'] = "2"
@checker.options.delete('expected_receive_period_in_days')
end
it "should check for reviewable HITs frequently" do
@ -187,7 +187,7 @@ describe Agents::HumanTaskAgent do
describe "when 'trigger_on' is set to 'event'" do
it "should not create HITs during check but should check for reviewable HITs" do
@checker.options[:submission_period] = "2"
@checker.options['submission_period'] = "2"
now = Time.now
stub(Time).now { now }
mock(@checker).review_hits.times(3)
@ -207,9 +207,9 @@ describe Agents::HumanTaskAgent do
describe "creating hits" do
it "can create HITs based on events, interpolating their values" do
@checker.options[:hit][:title] = "Hi <.name>"
@checker.options[:hit][:description] = "Make something for <.name>"
@checker.options[:hit][:questions][0][:name] = "<.name> Question 1"
@checker.options['hit']['title'] = "Hi <.name>"
@checker.options['hit']['description'] = "Make something for <.name>"
@checker.options['hit']['questions'][0]['name'] = "<.name> Question 1"
question_form = nil
hitInterface = OpenStruct.new
@ -219,8 +219,8 @@ describe Agents::HumanTaskAgent do
@checker.send :create_basic_hit, @event
hitInterface.max_assignments.should == @checker.options[:hit][:assignments]
hitInterface.reward.should == @checker.options[:hit][:reward]
hitInterface.max_assignments.should == @checker.options['hit']['assignments']
hitInterface.reward.should == @checker.options['hit']['reward']
hitInterface.description.should == "Make something for Joe"
xml = question_form.to_xml
@ -228,18 +228,18 @@ describe Agents::HumanTaskAgent do
xml.should include("<Text>Make something for Joe</Text>")
xml.should include("<DisplayName>Joe Question 1</DisplayName>")
@checker.memory[:hits][123][:event_id].should == @event.id
@checker.memory['hits'][123]['event_id'].should == @event.id
end
it "works without an event too" do
@checker.options[:hit][:title] = "Hi <.name>"
@checker.options['hit']['title'] = "Hi <.name>"
hitInterface = OpenStruct.new
hitInterface.id = 123
mock(hitInterface).question_form(instance_of Agents::HumanTaskAgent::AgentQuestionForm)
mock(RTurk::Hit).create(:title => "Hi").yields(hitInterface) { hitInterface }
@checker.send :create_basic_hit
hitInterface.max_assignments.should == @checker.options[:hit][:assignments]
hitInterface.reward.should == @checker.options[:hit][:reward]
hitInterface.max_assignments.should == @checker.options['hit']['assignments']
hitInterface.reward.should == @checker.options['hit']['reward']
end
end
@ -289,14 +289,14 @@ describe Agents::HumanTaskAgent do
it "should work on multiple HITs" do
event2 = Event.new
event2.agent = agents(:bob_rain_notifier_agent)
event2.payload = { :foo2 => { "bar2" => { :baz2 => "a2b2" } },
:name2 => "Joe2" }
event2.payload = { 'foo2' => { "bar2" => { 'baz2' => "a2b2" } },
'name2' => "Joe2" }
event2.id = 3452
# It knows about two HITs from two different events.
@checker.memory[:hits] = {}
@checker.memory[:hits][:"JH3132836336DHG"] = { :event_id => @event.id }
@checker.memory[:hits][:"JH39AA63836DHG"] = { :event_id => event2.id }
@checker.memory['hits'] = {}
@checker.memory['hits']["JH3132836336DHG"] = { 'event_id' => @event.id }
@checker.memory['hits']["JH39AA63836DHG"] = { 'event_id' => event2.id }
hit_ids = %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345]
mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { hit_ids } } # It sees 3 HITs.
@ -309,7 +309,7 @@ describe Agents::HumanTaskAgent do
end
it "shouldn't do anything if an assignment isn't ready" do
@checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
@checker.memory['hits'] = { "JH3132836336DHG" => { 'event_id' => @event.id } }
mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
assignments = [
FakeAssignment.new(:status => "Accepted", :answers => {}),
@ -324,11 +324,11 @@ describe Agents::HumanTaskAgent do
@checker.send :review_hits
assignments.all? {|a| a.approved == true }.should be_false
@checker.memory[:hits].should == { :"JH3132836336DHG" => { :event_id => @event.id } }
@checker.memory['hits'].should == { "JH3132836336DHG" => { 'event_id' => @event.id } }
end
it "shouldn't do anything if an assignment is missing" do
@checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
@checker.memory['hits'] = { "JH3132836336DHG" => { 'event_id' => @event.id } }
mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
assignments = [
FakeAssignment.new(:status => "Submitted", :answers => {"sentiment"=>"happy", "feedback"=>"Take 2"})
@ -342,11 +342,11 @@ describe Agents::HumanTaskAgent do
@checker.send :review_hits
assignments.all? {|a| a.approved == true }.should be_false
@checker.memory[:hits].should == { :"JH3132836336DHG" => { :event_id => @event.id } }
@checker.memory['hits'].should == { "JH3132836336DHG" => { 'event_id' => @event.id } }
end
it "should create events when all assignments are ready" do
@checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
@checker.memory['hits'] = { "JH3132836336DHG" => { 'event_id' => @event.id } }
mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
assignments = [
FakeAssignment.new(:status => "Submitted", :answers => {"sentiment"=>"neutral", "feedback"=>""}),
@ -363,32 +363,32 @@ describe Agents::HumanTaskAgent do
assignments.all? {|a| a.approved == true }.should be_true
hit.should be_disposed
@checker.events.last.payload[:answers].should == [
{:sentiment => "neutral", :feedback => ""},
{:sentiment => "happy", :feedback => "Take 2"}
@checker.events.last.payload['answers'].should == [
{'sentiment' => "neutral", 'feedback' => ""},
{'sentiment' => "happy", 'feedback' => "Take 2"}
]
@checker.memory[:hits].should == {}
@checker.memory['hits'].should == {}
end
describe "taking majority votes" do
before do
@checker.options[:combination_mode] = "take_majority"
@checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
@checker.options['combination_mode'] = "take_majority"
@checker.memory['hits'] = { "JH3132836336DHG" => { 'event_id' => @event.id } }
mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
end
it "should take the majority votes of all questions" do
@checker.options[:hit][:questions][1] = {
:type => "selection",
:key => "age_range",
:name => "Age Range",
:required => "true",
:question => "Please select your age range:",
:selections =>
@checker.options['hit']['questions'][1] = {
'type' => "selection",
'key' => "age_range",
'name' => "Age Range",
'required' => "true",
'question' => "Please select your age range:",
'selections' =>
[
{ :key => "<50", :text => "50 years old or younger" },
{ :key => ">50", :text => "Over 50 years old" }
{ 'key' => "<50", 'text' => "50 years old or younger" },
{ 'key' => ">50", 'text' => "Over 50 years old" }
]
}
@ -407,39 +407,39 @@ describe Agents::HumanTaskAgent do
assignments.all? {|a| a.approved == true }.should be_true
@checker.events.last.payload[:answers].should == [
{ :sentiment => "sad", :age_range => "<50" },
{ :sentiment => "neutral", :age_range => ">50" },
{ :sentiment => "happy", :age_range => ">50" },
{ :sentiment => "happy", :age_range => ">50" }
@checker.events.last.payload['answers'].should == [
{ 'sentiment' => "sad", 'age_range' => "<50" },
{ 'sentiment' => "neutral", 'age_range' => ">50" },
{ 'sentiment' => "happy", 'age_range' => ">50" },
{ 'sentiment' => "happy", 'age_range' => ">50" }
]
@checker.events.last.payload[:counts].should == { :sentiment => { :happy => 2, :sad => 1, :neutral => 1 }, :age_range => { :">50" => 3, :"<50" => 1 } }
@checker.events.last.payload[:majority_answer].should == { :sentiment => "happy", :age_range => ">50" }
@checker.events.last.payload.should_not have_key(:average_answer)
@checker.events.last.payload['counts'].should == { 'sentiment' => { 'happy' => 2, 'sad' => 1, 'neutral' => 1 }, 'age_range' => { ">50" => 3, "<50" => 1 } }
@checker.events.last.payload['majority_answer'].should == { 'sentiment' => "happy", 'age_range' => ">50" }
@checker.events.last.payload.should_not have_key('average_answer')
@checker.memory[:hits].should == {}
@checker.memory['hits'].should == {}
end
it "should also provide an average answer when all questions are numeric" do
# it should accept 'take_majority': 'true' as well for legacy support. Demonstrating that here.
@checker.options.delete :combination_mode
@checker.options[:take_majority] = "true"
@checker.options['take_majority'] = "true"
@checker.options[:hit][:questions] = [
@checker.options['hit']['questions'] = [
{
:type => "selection",
:key => "rating",
:name => "Rating",
:required => "true",
:question => "Please select a rating:",
:selections =>
'type' => "selection",
'key' => "rating",
'name' => "Rating",
'required' => "true",
'question' => "Please select a rating:",
'selections' =>
[
{ :key => "1", :text => "One" },
{ :key => "2", :text => "Two" },
{ :key => "3", :text => "Three" },
{ :key => "4", :text => "Four" },
{ :key => "5.1", :text => "Five Point One" }
{ 'key' => "1", 'text' => "One" },
{ 'key' => "2", 'text' => "Two" },
{ 'key' => "3", 'text' => "Three" },
{ 'key' => "4", 'text' => "Four" },
{ 'key' => "5.1", 'text' => "Five Point One" }
]
}
]
@ -460,37 +460,37 @@ describe Agents::HumanTaskAgent do
assignments.all? {|a| a.approved == true }.should be_true
@checker.events.last.payload[:answers].should == [
{ :rating => "1" },
{ :rating => "3" },
{ :rating => "5.1" },
{ :rating => "2" },
{ :rating => "2" }
@checker.events.last.payload['answers'].should == [
{ 'rating' => "1" },
{ 'rating' => "3" },
{ 'rating' => "5.1" },
{ 'rating' => "2" },
{ 'rating' => "2" }
]
@checker.events.last.payload[:counts].should == { :rating => { :"1" => 1, :"2" => 2, :"3" => 1, :"4" => 0, :"5.1" => 1 } }
@checker.events.last.payload[:majority_answer].should == { :rating => "2" }
@checker.events.last.payload[:average_answer].should == { :rating => (1 + 2 + 2 + 3 + 5.1) / 5.0 }
@checker.events.last.payload['counts'].should == { 'rating' => { "1" => 1, "2" => 2, "3" => 1, "4" => 0, "5.1" => 1 } }
@checker.events.last.payload['majority_answer'].should == { 'rating' => "2" }
@checker.events.last.payload['average_answer'].should == { 'rating' => (1 + 2 + 2 + 3 + 5.1) / 5.0 }
@checker.memory[:hits].should == {}
@checker.memory['hits'].should == {}
end
end
describe "creating and reviewing polls" do
before do
@checker.options[:combination_mode] = "poll"
@checker.options[:poll_options] = {
:title => "Hi!",
:instructions => "hello!",
:assignments => 2,
:row_template => "This is <.sentiment>"
@checker.options['combination_mode'] = "poll"
@checker.options['poll_options'] = {
'title' => "Hi!",
'instructions' => "hello!",
'assignments' => 2,
'row_template' => "This is <.sentiment>"
}
@event.save!
mock(RTurk::GetReviewableHITs).create { mock!.hit_ids { %w[JH3132836336DHG JH39AA63836DHG JH39AA63836DH12345] } }
end
it "creates a poll using the row_template, message, and correct number of assignments" do
@checker.memory[:hits] = { :"JH3132836336DHG" => { :event_id => @event.id } }
@checker.memory['hits'] = { "JH3132836336DHG" => { 'event_id' => @event.id } }
# Mock out the HIT's submitted assignments.
assignments = [
@ -502,7 +502,7 @@ describe Agents::HumanTaskAgent do
hit = FakeHit.new(:max_assignments => 4, :assignments => assignments)
mock(RTurk::Hit).new("JH3132836336DHG") { hit }
@checker.memory[:hits][:"JH3132836336DHG"].should be_present
@checker.memory['hits']["JH3132836336DHG"].should be_present
# Setup mocks for HIT creation
@ -525,8 +525,8 @@ describe Agents::HumanTaskAgent do
# it creates a new HIT for the poll
hitInterface.max_assignments.should == @checker.options[:poll_options][:assignments]
hitInterface.description.should == @checker.options[:poll_options][:instructions]
hitInterface.max_assignments.should == @checker.options['poll_options']['assignments']
hitInterface.description.should == @checker.options['poll_options']['instructions']
xml = question_form.to_xml
xml.should include("<Text>This is happy</Text>")
@ -535,28 +535,28 @@ describe Agents::HumanTaskAgent do
@checker.save
@checker.reload
@checker.memory[:hits][:"JH3132836336DHG"].should_not be_present
@checker.memory[:hits][:"JH39AA63836DH12345"].should be_present
@checker.memory[:hits][:"JH39AA63836DH12345"][:event_id].should == @event.id
@checker.memory[:hits][:"JH39AA63836DH12345"][:type].should == :poll
@checker.memory[:hits][:"JH39AA63836DH12345"][:original_hit].should == "JH3132836336DHG"
@checker.memory[:hits][:"JH39AA63836DH12345"][:answers].length.should == 4
@checker.memory['hits']["JH3132836336DHG"].should_not be_present
@checker.memory['hits']["JH39AA63836DH12345"].should be_present
@checker.memory['hits']["JH39AA63836DH12345"]['event_id'].should == @event.id
@checker.memory['hits']["JH39AA63836DH12345"]['type'].should == "poll"
@checker.memory['hits']["JH39AA63836DH12345"]['original_hit'].should == "JH3132836336DHG"
@checker.memory['hits']["JH39AA63836DH12345"]['answers'].length.should == 4
end
it "emits an event when all poll results are in, containing the data from the best answer, plus all others" do
original_answers = [
{:sentiment => "sad", :feedback => "This is my feedback 1"},
{:sentiment => "neutral", :feedback => "This is my feedback 2"},
{:sentiment => "happy", :feedback => "This is my feedback 3"},
{:sentiment => "happy", :feedback => "This is my feedback 4"}
{ 'sentiment' => "sad", 'feedback' => "This is my feedback 1"},
{ 'sentiment' => "neutral", 'feedback' => "This is my feedback 2"},
{ 'sentiment' => "happy", 'feedback' => "This is my feedback 3"},
{ 'sentiment' => "happy", 'feedback' => "This is my feedback 4"}
]
@checker.memory[:hits] = {
:JH39AA63836DH12345 => {
:type => :poll,
:original_hit => "JH3132836336DHG",
:answers => original_answers,
:event_id => 345
@checker.memory['hits'] = {
'JH39AA63836DH12345' => {
'type' => 'poll',
'original_hit' => "JH3132836336DHG",
'answers' => original_answers,
'event_id' => 345
}
}
@ -568,7 +568,7 @@ describe Agents::HumanTaskAgent do
hit = FakeHit.new(:max_assignments => 2, :assignments => assignments)
mock(RTurk::Hit).new("JH39AA63836DH12345") { hit }
@checker.memory[:hits][:"JH39AA63836DH12345"].should be_present
@checker.memory['hits']["JH39AA63836DH12345"].should be_present
lambda {
@checker.send :review_hits
@ -576,17 +576,17 @@ describe Agents::HumanTaskAgent do
# It emits an event
@checker.events.last.payload[:answers].should == original_answers
@checker.events.last.payload[:poll].should == [{:"1" => "2", :"2" => "5", :"3" => "3", :"4" => "2"}, {:"1" => "3", :"2" => "4", :"3" => "1", :"4" => "4"}]
@checker.events.last.payload[:best_answer].should == {:sentiment => "neutral", :feedback => "This is my feedback 2"}
@checker.events.last.payload['answers'].should == original_answers
@checker.events.last.payload['poll'].should == [{"1" => "2", "2" => "5", "3" => "3", "4" => "2"}, {"1" => "3", "2" => "4", "3" => "1", "4" => "4"}]
@checker.events.last.payload['best_answer'].should == {'sentiment' => "neutral", 'feedback' => "This is my feedback 2"}
# it approves the existing assignments
assignments.all? {|a| a.approved == true }.should be_true
hit.should be_disposed
@checker.memory[:hits].should be_empty
@checker.memory['hits'].should be_empty
end
end
end
end
end

58
spec/models/agents/peak_detector_agent_spec.rb
View file

@ -3,12 +3,12 @@ require 'spec_helper'
describe Agents::PeakDetectorAgent do
before do
@valid_params = {
:name => "my peak detector agent",
:options => {
:expected_receive_period_in_days => "2",
:group_by_path => "filter",
:value_path => "count",
:message => "A peak was found"
'name' => "my peak detector agent",
'options' => {
'expected_receive_period_in_days' => "2",
'group_by_path' => "filter",
'value_path' => "count",
'message' => "A peak was found"
}
}
@ -19,54 +19,54 @@ describe Agents::PeakDetectorAgent do
describe "#receive" do
it "tracks and groups by the group_by_path" do
events = build_events(:keys => [:count, :filter],
events = build_events(:keys => ['count', 'filter'],
:values => [[1, "something"], [2, "something"], [3, "else"]])
@agent.receive events
@agent.memory[:data][:something].map(&:first).should == [1, 2]
@agent.memory[:data][:something].last.last.should be_within(10).of((100 - 1).hours.ago.to_i)
@agent.memory[:data][:else].first.first.should == 3
@agent.memory[:data][:else].first.last.should be_within(10).of((100 - 2).hours.ago.to_i)
@agent.memory['data']['something'].map(&:first).should == [1, 2]
@agent.memory['data']['something'].last.last.should be_within(10).of((100 - 1).hours.ago.to_i)
@agent.memory['data']['else'].first.first.should == 3
@agent.memory['data']['else'].first.last.should be_within(10).of((100 - 2).hours.ago.to_i)
end
it "works without a group_by_path as well" do
@agent.options[:group_by_path] = ""
events = build_events(:keys => [:count], :values => [[1], [2]])
@agent.options['group_by_path'] = ""
events = build_events(:keys => ['count'], :values => [[1], [2]])
@agent.receive events
@agent.memory[:data][:no_group].map(&:first).should == [1, 2]
@agent.memory['data']['no_group'].map(&:first).should == [1, 2]
end
it "keeps a rolling window of data" do
@agent.options[:window_duration_in_days] = 5/24.0
@agent.receive build_events(:keys => [:count],
@agent.options['window_duration_in_days'] = 5/24.0
@agent.receive build_events(:keys => ['count'],
:values => [1, 2, 3, 4, 5, 6, 7, 8].map {|i| [i]},
:pattern => { :filter => "something" })
@agent.memory[:data][:something].map(&:first).should == [4, 5, 6, 7, 8]
:pattern => { 'filter' => "something" })
@agent.memory['data']['something'].map(&:first).should == [4, 5, 6, 7, 8]
end
it "finds peaks" do
build_events(:keys => [:count],
build_events(:keys => ['count'],
:values => [5, 6,
4, 5,
4, 5,
15, 11, # peak
8, 50, # ignored because it's too close to the first peak
4, 5].map {|i| [i]},
:pattern => { :filter => "something" }).each.with_index do |event, index|
:pattern => { 'filter' => "something" }).each.with_index do |event, index|
lambda {
@agent.receive([event])
}.should change { @agent.events.count }.by( index == 6 ? 1 : 0 )
end
@agent.events.last.payload[:peak].should == 15.0
@agent.memory[:peaks][:something].length.should == 1
@agent.events.last.payload['peak'].should == 15.0
@agent.memory['peaks']['something'].length.should == 1
end
it "keeps a rolling window of peaks" do
@agent.options[:min_peak_spacing_in_days] = 1/24.0
@agent.receive build_events(:keys => [:count],
@agent.options['min_peak_spacing_in_days'] = 1/24.0
@agent.receive build_events(:keys => ['count'],
:values => [1, 1, 1, 1, 1, 1, 10, 1, 1, 1, 1, 1, 1, 1, 10, 1].map {|i| [i]},
:pattern => { :filter => "something" })
@agent.memory[:peaks][:something].length.should == 2
:pattern => { 'filter' => "something" })
@agent.memory['peaks']['something'].length.should == 2
end
end
@ -76,17 +76,17 @@ describe Agents::PeakDetectorAgent do
end
it "should validate presence of message" do
@agent.options[:message] = nil
@agent.options['message'] = nil
@agent.should_not be_valid
end
it "should validate presence of expected_receive_period_in_days" do
@agent.options[:expected_receive_period_in_days] = ""
@agent.options['expected_receive_period_in_days'] = ""
@agent.should_not be_valid
end
it "should validate presence of value_path" do
@agent.options[:value_path] = ""
@agent.options['value_path'] = ""
@agent.should_not be_valid
end
end

2
spec/models/agents/sentiment_agent_spec.rb
View file

@ -53,7 +53,7 @@ describe Agents::SentimentAgent do
it "checks if content key is working fine" do
@checker.receive([@event])
Event.last.payload[:content].should == "value1"
Event.last.payload[:original_event].should == {:message => "value1"}
Event.last.payload[:original_event].should == { 'message' => "value1" }
end
it "should handle multiple events" do
event1 = Event.new

106
spec/models/agents/trigger_agent_spec.rb
View file

@ -3,16 +3,16 @@ require 'spec_helper'
describe Agents::TriggerAgent do
before do
@valid_params = {
:name => "my trigger agent",
:options => {
:expected_receive_period_in_days => 2,
:rules => [{
:type => "regex",
'value' => "a\\db",
:path => "foo.bar.baz",
}],
:message => "I saw '<foo.bar.baz>' from <name>"
}
'name' => "my trigger agent",
'options' => {
'expected_receive_period_in_days' => 2,
'rules' => [{
'type' => "regex",
'value' => "a\\db",
'path' => "foo.bar.baz",
}],
'message' => "I saw '<foo.bar.baz>' from <name>"
}
}
@checker = Agents::TriggerAgent.new(@valid_params)
@ -21,8 +21,8 @@ describe Agents::TriggerAgent do
@event = Event.new
@event.agent = agents(:bob_rain_notifier_agent)
@event.payload = { :foo => { "bar" => { :baz => "a2b" }},
:name => "Joe" }
@event.payload = { 'foo' => { "bar" => { 'baz' => "a2b" }},
'name' => "Joe" }
end
describe "validation" do
@ -31,22 +31,22 @@ describe Agents::TriggerAgent do
end
it "should validate presence of options" do
@checker.options[:message] = nil
@checker.options['message'] = nil
@checker.should_not be_valid
end
it "should validate the three fields in each rule" do
@checker.options[:rules] << { :path => "foo", :type => "fake", :value => "6" }
@checker.options['rules'] << { 'path' => "foo", 'type' => "fake", 'value' => "6" }
@checker.should_not be_valid
@checker.options[:rules].last[:type] = "field>=value"
@checker.options['rules'].last['type'] = "field>=value"
@checker.should be_valid
@checker.options[:rules].last.delete(:value)
@checker.options['rules'].last.delete('value')
@checker.should_not be_valid
end
end
describe "#working?" do
it "checks to see if the Agent has received any events in the last :expected_receive_period_in_days days" do
it "checks to see if the Agent has received any events in the last 'expected_receive_period_in_days' days" do
@event.save!
@checker.should_not be_working # no events have ever been received
@ -60,30 +60,30 @@ describe Agents::TriggerAgent do
describe "#receive" do
it "handles regex" do
@event.payload[:foo]["bar"][:baz] = "a222b"
@event.payload['foo']['bar']['baz'] = "a222b"
lambda {
@checker.receive([@event])
}.should_not change { Event.count }
@event.payload[:foo]["bar"][:baz] = "a2b"
@event.payload['foo']['bar']['baz'] = "a2b"
lambda {
@checker.receive([@event])
}.should change { Event.count }.by(1)
end
it "handles negated regex" do
@event.payload[:foo]["bar"][:baz] = "a2b"
@checker.options[:rules][0] = {
:type => "!regex",
:value => "a\\db",
:path => "foo.bar.baz",
}
@event.payload['foo']['bar']['baz'] = "a2b"
@checker.options['rules'][0] = {
'type' => "!regex",
'value' => "a\\db",
'path' => "foo.bar.baz",
}
lambda {
@checker.receive([@event])
}.should_not change { Event.count }
@event.payload[:foo]["bar"][:baz] = "a22b"
@event.payload['foo']['bar']['baz'] = "a22b"
lambda {
@checker.receive([@event])
}.should change { Event.count }.by(1)
@ -91,49 +91,49 @@ describe Agents::TriggerAgent do
it "puts can extract values into the message based on paths" do
@checker.receive([@event])
Event.last.payload[:message].should == "I saw 'a2b' from Joe"
Event.last.payload['message'].should == "I saw 'a2b' from Joe"
end
it "handles numerical comparisons" do
@event.payload[:foo]["bar"][:baz] = "5"
@checker.options[:rules].first[:value] = 6
@checker.options[:rules].first[:type] = "field<value"
@event.payload['foo']['bar']['baz'] = "5"
@checker.options['rules'].first['value'] = 6
@checker.options['rules'].first['type'] = "field<value"
lambda {
@checker.receive([@event])
}.should change { Event.count }.by(1)
@checker.options[:rules].first[:value] = 3
@checker.options['rules'].first['value'] = 3
lambda {
@checker.receive([@event])
}.should_not change { Event.count }
end
it "handles exact comparisons" do
@event.payload[:foo]["bar"][:baz] = "hello world"
@checker.options[:rules].first[:type] = "field==value"
@event.payload['foo']['bar']['baz'] = "hello world"
@checker.options['rules'].first['type'] = "field==value"
@checker.options[:rules].first[:value] = "hello there"
@checker.options['rules'].first['value'] = "hello there"
lambda {
@checker.receive([@event])
}.should_not change { Event.count }
@checker.options[:rules].first[:value] = "hello world"
@checker.options['rules'].first['value'] = "hello world"
lambda {
@checker.receive([@event])
}.should change { Event.count }.by(1)
end
it "handles negated comparisons" do
@event.payload[:foo]["bar"][:baz] = "hello world"
@checker.options[:rules].first[:type] = "field!=value"
@checker.options[:rules].first[:value] = "hello world"
@event.payload['foo']['bar']['baz'] = "hello world"
@checker.options['rules'].first['type'] = "field!=value"
@checker.options['rules'].first['value'] = "hello world"
lambda {
@checker.receive([@event])
}.should_not change { Event.count }
@checker.options[:rules].first[:value] = "hello there"
@checker.options['rules'].first['value'] = "hello there"
lambda {
@checker.receive([@event])
@ -141,20 +141,20 @@ describe Agents::TriggerAgent do
end
it "does fine without dots in the path" do
@event.payload = { :hello => "world" }
@checker.options[:rules].first[:type] = "field==value"
@checker.options[:rules].first[:path] = "hello"
@checker.options[:rules].first[:value] = "world"
@event.payload = { 'hello' => "world" }
@checker.options['rules'].first['type'] = "field==value"
@checker.options['rules'].first['path'] = "hello"
@checker.options['rules'].first['value'] = "world"
lambda {
@checker.receive([@event])
}.should change { Event.count }.by(1)
@checker.options[:rules].first[:path] = "foo"
@checker.options['rules'].first['path'] = "foo"
lambda {
@checker.receive([@event])
}.should_not change { Event.count }
@checker.options[:rules].first[:value] = "hi"
@checker.options['rules'].first['value'] = "hi"
lambda {
@checker.receive([@event])
}.should_not change { Event.count }
@ -163,11 +163,11 @@ describe Agents::TriggerAgent do
it "handles multiple events" do
event2 = Event.new
event2.agent = agents(:bob_weather_agent)
event2.payload = { :foo => { "bar" => { :baz => "a2b" }}}
event2.payload = { 'foo' => { 'bar' => { 'baz' => "a2b" }}}
event3 = Event.new
event3.agent = agents(:bob_weather_agent)
event3.payload = { :foo => { "bar" => { :baz => "a222b" }}}
event3.payload = { 'foo' => { 'bar' => { 'baz' => "a222b" }}}
lambda {
@checker.receive([@event, event2, event3])
@ -175,19 +175,19 @@ describe Agents::TriggerAgent do
end
it "handles ANDing rules together" do
@checker.options[:rules] << {
:type => "field>=value",
:value => "4",
:path => "foo.bing"
@checker.options['rules'] << {
'type' => "field>=value",
'value' => "4",
'path' => "foo.bing"
}
@event.payload[:foo]["bing"] = "5"
@event.payload['foo']["bing"] = "5"
lambda {
@checker.receive([@event])
}.should change { Event.count }.by(1)
@checker.options[:rules].last[:value] = 6
@checker.options['rules'].last['value'] = 6
lambda {
@checker.receive([@event])
}.should_not change { Event.count }

14
spec/models/agents/twitter_stream_agent_spec.rb
View file

@ -53,7 +53,7 @@ describe Agents::TwitterStreamAgent do
@agent.memory[:filter_counts] = {:keyword1 => 2, :keyword2 => 3, :keyword3 => 4}
@agent.save!
@agent.process_tweet('keyword1', {:text => "something", :user => {:name => "Mr. Someone"}})
@agent.reload.memory[:filter_counts].should == {:keyword1 => 3, :keyword2 => 3}
@agent.reload.memory[:filter_counts].should == { 'keyword1' => 3, 'keyword2' => 3 }
end
end
@ -64,9 +64,9 @@ describe Agents::TwitterStreamAgent do
}.should change { @agent.events.count }.by(1)
@agent.events.last.payload.should == {
:filter => 'keyword1',
:text => "something",
:user => {:name => "Mr. Someone"}
'filter' => 'keyword1',
'text' => "something",
'user' => { 'name' => "Mr. Someone" }
}
end
@ -79,9 +79,9 @@ describe Agents::TwitterStreamAgent do
}.should change { @agent.events.count }.by(1)
@agent.events.last.payload.should == {
:filter => 'keyword1-1',
:text => "something",
:user => {:name => "Mr. Someone"}
'filter' => 'keyword1-1',
'text' => "something",
'user' => { 'name' => "Mr. Someone" }
}
end
end

10
spec/models/agents/webhook_agent_spec.rb
View file

@ -3,29 +3,27 @@ require 'spec_helper'
describe Agents::WebhookAgent do
let(:agent) do
_agent = Agents::WebhookAgent.new(:name => 'webhook',
:options => {:secret => :foobar, :payload_path => '$'})
:options => { 'secret' => 'foobar', 'payload_path' => 'payload' })
_agent.user = users(:bob)
_agent.save!
_agent
end
let(:payload) { {'some' => 'info'} }
after { agent.destroy }
describe 'receive_webhook' do
it 'should create event if secret matches' do
out = nil
lambda {
out = agent.receive_webhook({:secret => :foobar, :payload => payload})
out = agent.receive_webhook('secret' => 'foobar', 'payload' => payload)
}.should change { Event.count }.by(1)
out.should eq(['Event Created', 201])
Event.last.payload.should eq([{'payload' => payload}])
Event.last.payload.should eq(payload)
end
it 'should not create event if secrets dont match' do
out = nil
lambda {
out = agent.receive_webhook({:secret => :bazbat, :payload => payload})
out = agent.receive_webhook('secret' => 'bazbat', 'payload' => payload)
}.should change { Event.count }.by(0)
out.should eq(['Not Authorized', 401])
end

14
spec/models/agents/website_agent_spec.rb
View file

@ -44,18 +44,22 @@ describe Agents::WebsiteAgent do
describe '#working?' do
it 'checks if events have been received within the expected receive period' do
stubbed_time = Time.now
stub(Time).now { stubbed_time }
@checker.should_not be_working # No events created
@checker.check
@checker.reload.should be_working # Just created events
@checker.error "oh no!"
@checker.reload.should_not be_working # The most recent log is an error
@checker.reload.should_not be_working # There is a recent error
@checker.log "ok now"
@checker.reload.should be_working # The most recent log is no longer an error
stubbed_time = 20.minutes.from_now
@checker.events.delete_all
@checker.check
@checker.reload.should be_working # There is a newer event now
two_days_from_now = 2.days.from_now
stub(Time).now { two_days_from_now }
stubbed_time = 2.days.from_now
@checker.reload.should_not be_working # Two days have passed without a new event having been created
end
end