From 9d584b6ba4957137dabce28012992cb69da81c9e Mon Sep 17 00:00:00 2001 From: Darren Cauthon Date: Mon, 25 Jul 2016 20:23:33 -0500 Subject: [PATCH] Liquid output agent (#1587) * Start with a stripped copy of the data output agent. * Run the data from the last event through a liquid template. * Flatten the secret logic to allow for an easier switch to FormConfigurable. * Switch to form configurable, and allow the content of the page to be configured. * Allow the mime type to be changed. * Cleanup. * Write how this template works. * Better default values. * Cleanup. * Refactor. * Start testing. * Test the validation. * Test receive. * Test the happy path through the receive web events. * Test the authentication. * This is actually a match. * Refactor. * Refactor. * Refactor for better testing. * Create a mode that lets the logic change. Start with a merge behavior. * Refactor. * Create a form configurable setting to change the mode. * Document how the modes work. * Wording change. * Go with a singular secret. * Fix typo. * Fix the tests. * Test cleanup. * If not one of two types that receive events, ignore all received events. * Set up these tests for the next set of changes. * Isolate the method that gets the data for the liquid template. * Look up past events to render through the liquid template. * Implement a limit of 2 events. * Extract a method. * Hook the limit to options. * Implement a limit of X events. * Implement a date limit. * Refactor the count limit. * Limit by date with sql, not in-memory objects. * This ordering is already built into the scope. * Refactor the dates a bit. * Put in a few checks around the date limits. * Add the last X event options to the form and the documentation. * Missed one bit of documentation. * Add a view for a liquid output agent that makes it easy to retrieve the generated URL. * This agent cannot accept events. * Hardcode the possibilities instead of inspecting the integer. * Do not be case sensitive on the date filter. * Hardcode a limit of 5000, just in case no limit was provided. * Better checks around the time period parsing. * Test the hardcodes, and rename for consistency. * Nevermind on that rename. * Do not be case sensitive on this mode. * Test that it works even when the casing on the mode is wrong. * Here is more descriptive default content. * Text change. * The if is no longer necessary. * Refactor. * Move the limit down to 1000. * Put a hard limit of 1000. * Note the new event limit... limit. * Validate for a valid event limit. * Do not throw an error if someone types in a non-integer into this field. * Text update. * Typo. * Add a link to the Liquid Templating engine. --- app/models/agents/liquid_output_agent.rb | 214 ++++++++ .../liquid_output_agent/_show.html.erb | 15 + .../models/agents/liquid_output_agent_spec.rb | 461 ++++++++++++++++++ 3 files changed, 690 insertions(+) create mode 100644 app/models/agents/liquid_output_agent.rb create mode 100644 app/views/agents/agent_views/liquid_output_agent/_show.html.erb create mode 100644 spec/models/agents/liquid_output_agent_spec.rb diff --git a/app/models/agents/liquid_output_agent.rb b/app/models/agents/liquid_output_agent.rb new file mode 100644 index 00000000..a26210f5 --- /dev/null +++ b/app/models/agents/liquid_output_agent.rb @@ -0,0 +1,214 @@ +module Agents + class LiquidOutputAgent < Agent + include WebRequestConcern + include FormConfigurable + + cannot_be_scheduled! + cannot_create_events! + + DATE_UNITS = %w[second seconds minute minutes hour hours day days week weeks month months year years] + + description do + <<-MD + The Liquid Output Agent outputs events through a Liquid template you provide. Use it to create a HTML page, or a json feed, or anything else that can be rendered as a string from your stream of Huginn data. + + This Agent will output data at: + + `https://#{ENV['DOMAIN']}#{Rails.application.routes.url_helpers.web_requests_path(agent_id: ':id', user_id: user_id, secret: ':secret', format: :any_extension)}` + + where `:secret` is the secret specified in your options. You can use any extension you wish. + + Options: + + * `secret` - A token that the requestor must provide for light-weight authentication. + * `expected_receive_period_in_days` - How often you expect data to be received by this Agent from other Agents. + * `content` - The content to display when someone requests this page. + * `mime_type` - The mime type to use when someone requests this page. + * `mode` - The behavior that determines what data is passed to the Liquid template. + * `event_limit` - A limit applied to the events passed to a template when in "Last X events" mode. Can be a count like "1", or an amount of time like "1 day" or "5 minutes". + + # Liquid Templating + + The content you provide will be run as a Liquid template. The data from the last event received will be used when processing the Liquid template. + + To learn more about Liquid templates, go here: [http://liquidmarkup.org](http://liquidmarkup.org "Liquid Templating") + + # Modes + + ### Merge events + + The data for incoming events will be merged. So if two events come in like this: + +``` +{ 'a' => 'b', 'c' => 'd'} +{ 'a' => 'bb', 'e' => 'f'} +``` + + The final result will be: + +``` +{ 'a' => 'bb', 'c' => 'd', 'e' => 'f'} +``` + + This merged version will be passed to the Liquid template. + + ### Last event in + + The data from the last event will be passed to the template. + + ### Last X events + + All of the events received by this agent will be passed to the template + as the ```events``` array. + + The number of events can be controlled via the ```event_limit``` option. + If ```event_limit``` is an integer X, the last X events will be passed + to the template. If ```event_limit``` is an integer with a unit of + measure like "1 day" or "5 minutes" or "9 years", a date filter will + be applied to the events passed to the template. If no ```event_limit``` + is provided, then all of the events for the agent will be passed to + the template. + + For performance, the maximum ```event_limit``` allowed is 1000. + + MD + end + + def default_options + content = < + {% for event in events %} + + {{ event.title }} + Click here to see + + {% endfor %} + +EOF + { + "secret" => "a-secret-key", + "expected_receive_period_in_days" => 2, + "mime_type" => 'text/html', + "mode" => 'Last event in', + "event_limit" => '', + "content" => content, + } + end + + form_configurable :secret + form_configurable :expected_receive_period_in_days + form_configurable :content, type: :text + form_configurable :mime_type + form_configurable :mode, type: :array, values: [ 'Last event in', 'Merge events', 'Last X events'] + form_configurable :event_limit + + def working? + last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs? + end + + def validate_options + if options['secret'].present? + case options['secret'] + when %r{[/.]} + errors.add(:base, "secret may not contain a slash or dot") + when String + else + errors.add(:base, "secret must be a string") + end + else + errors.add(:base, "Please specify one secret for 'authenticating' incoming feed requests") + end + + unless options['expected_receive_period_in_days'].present? && options['expected_receive_period_in_days'].to_i > 0 + errors.add(:base, "Please provide 'expected_receive_period_in_days' to indicate how many days can pass before this Agent is considered to be not working") + end + + if options['event_limit'].present? + if((Integer(options['event_limit']) rescue false) == false) + errors.add(:base, "Event limit must be an integer that is less than 1001.") + elsif (options['event_limit'].to_i > 1000) + errors.add(:base, "For performance reasons, you cannot have an event limit greater than 1000.") + end + else + end + end + + def receive(incoming_events) + return unless ['merge events', 'last event in'].include?(mode) + memory['last_event'] ||= {} + incoming_events.each do |event| + case mode + when 'merge events' + memory['last_event'] = memory['last_event'].merge(event.payload) + else + memory['last_event'] = event.payload + end + end + end + + def receive_web_request(params, method, format) + valid_authentication?(params) ? [liquified_content, 200, mime_type] + : [unauthorized_content(format), 401] + end + + private + + def mode + options['mode'].to_s.downcase + end + + def unauthorized_content(format) + format =~ /json/ ? { error: "Not Authorized" } + : "Not Authorized" + end + + def valid_authentication?(params) + interpolated['secret'] == params['secret'] + end + + def mime_type + options['mime_type'].presence || 'text/html' + end + + def liquified_content + template = Liquid::Template.parse(options['content'] || "") + template.render(data_for_liquid_template) + end + + def data_for_liquid_template + case mode + when 'last x events' + events = received_events + events = events.where('events.created_at > ?', date_limit) if date_limit + events = events.limit count_limit + events = events.to_a.map { |x| x.payload } + { 'events' => events } + else + memory['last_event'] || {} + end + end + + def count_limit + limit = Integer(options['event_limit']) rescue 1000 + limit <= 1000 ? limit : 1000 + end + + def date_limit + return nil unless options['event_limit'].to_s.include?(' ') + value, unit = options['event_limit'].split(' ') + value = Integer(value) rescue nil + return nil unless value + unit = unit.to_s.downcase + return nil unless DATE_UNITS.include?(unit) + value.send(unit.to_sym).ago + end + + end +end diff --git a/app/views/agents/agent_views/liquid_output_agent/_show.html.erb b/app/views/agents/agent_views/liquid_output_agent/_show.html.erb new file mode 100644 index 00000000..152d1f5f --- /dev/null +++ b/app/views/agents/agent_views/liquid_output_agent/_show.html.erb @@ -0,0 +1,15 @@ +

+ Data for this Agent is available at these URLs: +

+ + + + +

+ ... or any other extension you wish, as the extension does not change the content or mime type. +

diff --git a/spec/models/agents/liquid_output_agent_spec.rb b/spec/models/agents/liquid_output_agent_spec.rb new file mode 100644 index 00000000..9a73d808 --- /dev/null +++ b/spec/models/agents/liquid_output_agent_spec.rb @@ -0,0 +1,461 @@ +# encoding: utf-8 + +require 'rails_helper' + +describe Agents::LiquidOutputAgent do + let(:agent) do + _agent = Agents::LiquidOutputAgent.new(:name => 'My Data Output Agent') + _agent.options = _agent.default_options.merge('secret' => 'secret1', 'events_to_show' => 3) + _agent.options['secret'] = "a secret" + _agent.user = users(:bob) + _agent.sources << agents(:bob_website_agent) + _agent.save! + _agent + end + + describe "#working?" do + it "checks if events have been received within expected receive period" do + expect(agent).not_to be_working + Agents::LiquidOutputAgent.async_receive agent.id, [events(:bob_website_agent_event).id] + expect(agent.reload).to be_working + two_days_from_now = 2.days.from_now + stub(Time).now { two_days_from_now } + expect(agent.reload).not_to be_working + end + end + + describe "validation" do + before do + expect(agent).to be_valid + end + + it "should validate presence and length of secret" do + agent.options[:secret] = "" + expect(agent).not_to be_valid + agent.options[:secret] = "foo" + expect(agent).to be_valid + agent.options[:secret] = "foo/bar" + expect(agent).not_to be_valid + agent.options[:secret] = "foo.xml" + expect(agent).not_to be_valid + agent.options[:secret] = false + expect(agent).not_to be_valid + agent.options[:secret] = [] + expect(agent).not_to be_valid + agent.options[:secret] = ["foo.xml"] + expect(agent).not_to be_valid + agent.options[:secret] = ["hello", true] + expect(agent).not_to be_valid + agent.options[:secret] = ["hello"] + expect(agent).not_to be_valid + agent.options[:secret] = ["hello", "world"] + expect(agent).not_to be_valid + end + + it "should validate presence of expected_receive_period_in_days" do + agent.options[:expected_receive_period_in_days] = "" + expect(agent).not_to be_valid + agent.options[:expected_receive_period_in_days] = 0 + expect(agent).not_to be_valid + agent.options[:expected_receive_period_in_days] = -1 + expect(agent).not_to be_valid + end + + it "should validate the event_limit" do + agent.options[:event_limit] = "" + expect(agent).to be_valid + agent.options[:event_limit] = "1" + expect(agent).to be_valid + agent.options[:event_limit] = "1001" + expect(agent).not_to be_valid + agent.options[:event_limit] = "10000" + expect(agent).not_to be_valid + end + + it "should should not allow non-integer event limits" do + agent.options[:event_limit] = "abc1234" + expect(agent).not_to be_valid + end + end + + describe "#receive?" do + + let(:key) { SecureRandom.uuid } + let(:value) { SecureRandom.uuid } + + let(:incoming_events) do + last_payload = { key => value } + [Struct.new(:payload).new( { key => SecureRandom.uuid } ), + Struct.new(:payload).new( { key => SecureRandom.uuid } ), + Struct.new(:payload).new(last_payload)] + end + + describe "and the mode is last event in" do + + before { agent.options['mode'] = 'Last event in' } + + it "stores the last event in memory" do + agent.receive incoming_events + expect(agent.memory['last_event'][key]).to equal(value) + end + + describe "but the casing is wrong" do + before { agent.options['mode'] = 'LAST EVENT IN' } + + it "stores the last event in memory" do + agent.receive incoming_events + expect(agent.memory['last_event'][key]).to equal(value) + end + end + + end + + describe "but the mode is merge" do + + let(:second_key) { SecureRandom.uuid } + let(:second_value) { SecureRandom.uuid } + + before { agent.options['mode'] = 'Merge events' } + + let(:incoming_events) do + last_payload = { key => value } + [Struct.new(:payload).new( { key => SecureRandom.uuid, second_key => second_value } ), + Struct.new(:payload).new(last_payload)] + end + + it "should merge all of the events passed to it" do + agent.receive incoming_events + expect(agent.memory['last_event'][key]).to equal(value) + expect(agent.memory['last_event'][second_key]).to equal(second_value) + end + + describe "but the casing on the mode is wrong" do + + before { agent.options['mode'] = 'MERGE EVENTS' } + + it "should merge all of the events passed to it" do + agent.receive incoming_events + expect(agent.memory['last_event'][key]).to equal(value) + expect(agent.memory['last_event'][second_key]).to equal(second_value) + end + + end + + end + + describe "but the mode is anything else" do + + before { agent.options['mode'] = SecureRandom.uuid } + + let(:incoming_events) do + last_payload = { key => value } + [Struct.new(:payload).new(last_payload)] + end + + it "should do nothing" do + agent.receive incoming_events + expect(agent.memory.keys.count).to equal(0) + end + + end + + end + + describe "#count_limit" do + it "should have a default of 1000" do + agent.options['event_limit'] = nil + expect(agent.send(:count_limit)).to eq(1000) + + agent.options['event_limit'] = '' + expect(agent.send(:count_limit)).to eq(1000) + + agent.options['event_limit'] = ' ' + expect(agent.send(:count_limit)).to eq(1000) + end + + it "should convert string count limits to integers" do + agent.options['event_limit'] = '1' + expect(agent.send(:count_limit)).to eq(1) + + agent.options['event_limit'] = '2' + expect(agent.send(:count_limit)).to eq(2) + + agent.options['event_limit'] = 3 + expect(agent.send(:count_limit)).to eq(3) + end + + it "should default to 1000 with invalid values" do + agent.options['event_limit'] = SecureRandom.uuid + expect(agent.send(:count_limit)).to eq(1000) + + agent.options['event_limit'] = 'John Galt' + expect(agent.send(:count_limit)).to eq(1000) + end + + it "should not allow event limits above 1000" do + agent.options['event_limit'] = '1001' + expect(agent.send(:count_limit)).to eq(1000) + + agent.options['event_limit'] = '5000' + expect(agent.send(:count_limit)).to eq(1000) + end + end + + describe "#receive_web_request?" do + + let(:secret) { SecureRandom.uuid } + + let(:params) { { 'secret' => secret } } + + let(:method) { nil } + let(:format) { nil } + + let(:mime_type) { SecureRandom.uuid } + let(:content) { "The key is {{#{key}}}." } + + let(:key) { SecureRandom.uuid } + let(:value) { SecureRandom.uuid } + + before do + agent.options['secret'] = secret + agent.options['mime_type'] = mime_type + agent.options['content'] = content + agent.memory['last_event'] = { key => value } + agents(:bob_website_agent).events.destroy_all + end + + describe "and the mode is last event in" do + + before { agent.options['mode'] = 'Last event in' } + + it "should render the results as a liquid template from the last event in" do + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq("The key is #{value}.") + expect(result[1]).to eq(200) + expect(result[2]).to eq(mime_type) + end + + describe "but the casing is wrong" do + before { agent.options['mode'] = 'last event in' } + + it "should render the results as a liquid template from the last event in" do + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq("The key is #{value}.") + expect(result[1]).to eq(200) + expect(result[2]).to eq(mime_type) + end + end + + end + + describe "and the mode is merge events" do + + before { agent.options['mode'] = 'Merge events' } + + it "should render the results as a liquid template from the last event in" do + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq("The key is #{value}.") + expect(result[1]).to eq(200) + expect(result[2]).to eq(mime_type) + end + + end + + describe "and the mode is last X events" do + + before do + agent.options['mode'] = 'Last X events' + + agents(:bob_website_agent).create_event payload: { + "name" => "Dagny Taggart", + "book" => "Atlas Shrugged" + } + agents(:bob_website_agent).create_event payload: { + "name" => "John Galt", + "book" => "Atlas Shrugged" + } + agents(:bob_website_agent).create_event payload: { + "name" => "Howard Roark", + "book" => "The Fountainhead" + } + + agent.options['content'] = < + {% for event in events %} + + {{ event.name }} + {{ event.book }} + + {% endfor %} + +EOF + end + + it "should render the results as a liquid template from the last event in, limiting to 2" do + agent.options['event_limit'] = 2 + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq < + + + Howard Roark + The Fountainhead + + + + John Galt + Atlas Shrugged + + + +EOF + end + + it "should render the results as a liquid template from the last event in, limiting to 1" do + agent.options['event_limit'] = 1 + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq < + + + Howard Roark + The Fountainhead + + + +EOF + end + + it "should render the results as a liquid template from the last event in, allowing no limit" do + agent.options['event_limit'] = '' + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq < + + + Howard Roark + The Fountainhead + + + + John Galt + Atlas Shrugged + + + + Dagny Taggart + Atlas Shrugged + + + +EOF + end + + it "should allow the limiting by time, as well" do + + one_event = agent.received_events.select { |x| x.payload['name'] == 'John Galt' }.first + one_event.created_at = 2.days.ago + one_event.save! + + agent.options['event_limit'] = '1 day' + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq < + + + Howard Roark + The Fountainhead + + + + Dagny Taggart + Atlas Shrugged + + + +EOF + end + + it "should not be case sensitive when limiting on time" do + + one_event = agent.received_events.select { |x| x.payload['name'] == 'John Galt' }.first + one_event.created_at = 2.days.ago + one_event.save! + + agent.options['event_limit'] = '1 DaY' + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq < + + + Howard Roark + The Fountainhead + + + + Dagny Taggart + Atlas Shrugged + + + +EOF + end + + it "it should continue to work when the event limit is wrong" do + agent.options['event_limit'] = 'five days' + result = agent.receive_web_request params, method, format + + expect(result[0].include?("Howard Roark")).to eq(true) + expect(result[0].include?("Dagny Taggart")).to eq(true) + expect(result[0].include?("John Galt")).to eq(true) + + agent.options['event_limit'] = '5 quibblequarks' + result = agent.receive_web_request params, method, format + + expect(result[0].include?("Howard Roark")).to eq(true) + expect(result[0].include?("Dagny Taggart")).to eq(true) + expect(result[0].include?("John Galt")).to eq(true) + end + + describe "but the mode was set to last X events with the wrong casing" do + + before { agent.options['mode'] = 'LAST X EVENTS' } + + it "should still work as last x events" do + result = agent.receive_web_request params, method, format + expect(result[0].include?("Howard Roark")).to eq(true) + expect(result[0].include?("Dagny Taggart")).to eq(true) + expect(result[0].include?("John Galt")).to eq(true) + end + + end + + end + + describe "but the secret provided does not match" do + before { params['secret'] = SecureRandom.uuid } + + it "should return a 401 response" do + result = agent.receive_web_request params, method, format + + expect(result[0]).to eq("Not Authorized") + expect(result[1]).to eq(401) + end + + it "should return a 401 json response if the format is json" do + result = agent.receive_web_request params, method, 'json' + + expect(result[0][:error]).to eq("Not Authorized") + expect(result[1]).to eq(401) + end + end + end +end