Liquid output agent (#1587)

* Start with a stripped copy of the data output agent.

* Run the data from the last event through a liquid template.

* Flatten the secret logic to allow for an easier switch to FormConfigurable.

* Switch to form configurable, and allow the content of the page to be configured.

* Allow the mime type to be changed.

* Cleanup.

* Write how this template works.

* Better default values.

* Cleanup.

* Refactor.

* Start testing.

* Test the validation.

* Test receive.

* Test the happy path through the receive web events.

* Test the authentication.

* This is actually a match.

* Refactor.

* Refactor.

* Refactor for better testing.

* Create a mode that lets the logic change. Start with a merge behavior.

* Refactor.

* Create a form configurable setting to change the mode.

* Document how the modes work.

* Wording change.

* Go with a singular secret.

* Fix typo.

* Fix the tests.

* Test cleanup.

* If not one of two types that receive events, ignore all received events.

* Set up these tests for the next set of changes.

* Isolate the method that gets the data for the liquid template.

* Look up past events to render through the liquid template.

* Implement a limit of 2 events.

* Extract a method.

* Hook the limit to options.

* Implement a limit of X events.

* Implement a date limit.

* Refactor the count limit.

* Limit by date with sql, not in-memory objects.

* This ordering is already built into the scope.

* Refactor the dates a bit.

* Put in a few checks around the date limits.

* Add the last X event options to the form and the documentation.

* Missed one bit of documentation.

* Add a view for a liquid output agent that makes it easy to retrieve the generated URL.

* This agent cannot accept events.

* Hardcode the possibilities instead of inspecting the integer.

* Do not be case sensitive on the date filter.

* Hardcode a limit of 5000, just in case no limit was provided.

* Better checks around the time period parsing.

* Test the hardcodes, and rename for consistency.

* Nevermind on that rename.

* Do not be case sensitive on this mode.

* Test that it works even when the casing on the mode is wrong.

* Here is more descriptive default content.

* Text change.

* The if is no longer necessary.

* Refactor.

* Move the limit down to 1000.

* Put a hard limit of 1000.

* Note the new event limit... limit.

* Validate for a valid event limit.

* Do not throw an error if someone types in a non-integer into this field.

* Text update.

* Typo.

* Add a link to the Liquid Templating engine.
This commit is contained in:
Darren Cauthon 2016-07-25 20:23:33 -05:00 committed by Andrew Cantino
parent cc72f79e0b
commit 9d584b6ba4
3 changed files with 690 additions and 0 deletions

View file

@ -0,0 +1,214 @@
module Agents
class LiquidOutputAgent < Agent
include WebRequestConcern
include FormConfigurable
cannot_be_scheduled!
cannot_create_events!
DATE_UNITS = %w[second seconds minute minutes hour hours day days week weeks month months year years]
description do
<<-MD
The Liquid Output Agent outputs events through a Liquid template you provide. Use it to create a HTML page, or a json feed, or anything else that can be rendered as a string from your stream of Huginn data.
This Agent will output data at:
`https://#{ENV['DOMAIN']}#{Rails.application.routes.url_helpers.web_requests_path(agent_id: ':id', user_id: user_id, secret: ':secret', format: :any_extension)}`
where `:secret` is the secret specified in your options. You can use any extension you wish.
Options:
* `secret` - A token that the requestor must provide for light-weight authentication.
* `expected_receive_period_in_days` - How often you expect data to be received by this Agent from other Agents.
* `content` - The content to display when someone requests this page.
* `mime_type` - The mime type to use when someone requests this page.
* `mode` - The behavior that determines what data is passed to the Liquid template.
* `event_limit` - A limit applied to the events passed to a template when in "Last X events" mode. Can be a count like "1", or an amount of time like "1 day" or "5 minutes".
# Liquid Templating
The content you provide will be run as a Liquid template. The data from the last event received will be used when processing the Liquid template.
To learn more about Liquid templates, go here: [http://liquidmarkup.org](http://liquidmarkup.org "Liquid Templating")
# Modes
### Merge events
The data for incoming events will be merged. So if two events come in like this:
```
{ 'a' => 'b', 'c' => 'd'}
{ 'a' => 'bb', 'e' => 'f'}
```
The final result will be:
```
{ 'a' => 'bb', 'c' => 'd', 'e' => 'f'}
```
This merged version will be passed to the Liquid template.
### Last event in
The data from the last event will be passed to the template.
### Last X events
All of the events received by this agent will be passed to the template
as the ```events``` array.
The number of events can be controlled via the ```event_limit``` option.
If ```event_limit``` is an integer X, the last X events will be passed
to the template. If ```event_limit``` is an integer with a unit of
measure like "1 day" or "5 minutes" or "9 years", a date filter will
be applied to the events passed to the template. If no ```event_limit```
is provided, then all of the events for the agent will be passed to
the template.
For performance, the maximum ```event_limit``` allowed is 1000.
MD
end
def default_options
content = <<EOF
When you use the "Last event in" or "Merge events" option, you can use variables from the last event received, like this:
Name: {{name}}
Url: {{url}}
If you use the "Last X Events" mode, a set of events will be passed to your Liquid template. You can use them like this:
<table class="table">
{% for event in events %}
<tr>
<td>{{ event.title }}</td>
<td><a href="{{ event.url }}">Click here to see</a></td>
</tr>
{% endfor %}
</table>
EOF
{
"secret" => "a-secret-key",
"expected_receive_period_in_days" => 2,
"mime_type" => 'text/html',
"mode" => 'Last event in',
"event_limit" => '',
"content" => content,
}
end
form_configurable :secret
form_configurable :expected_receive_period_in_days
form_configurable :content, type: :text
form_configurable :mime_type
form_configurable :mode, type: :array, values: [ 'Last event in', 'Merge events', 'Last X events']
form_configurable :event_limit
def working?
last_receive_at && last_receive_at > options['expected_receive_period_in_days'].to_i.days.ago && !recent_error_logs?
end
def validate_options
if options['secret'].present?
case options['secret']
when %r{[/.]}
errors.add(:base, "secret may not contain a slash or dot")
when String
else
errors.add(:base, "secret must be a string")
end
else
errors.add(:base, "Please specify one secret for 'authenticating' incoming feed requests")
end
unless options['expected_receive_period_in_days'].present? && options['expected_receive_period_in_days'].to_i > 0
errors.add(:base, "Please provide 'expected_receive_period_in_days' to indicate how many days can pass before this Agent is considered to be not working")
end
if options['event_limit'].present?
if((Integer(options['event_limit']) rescue false) == false)
errors.add(:base, "Event limit must be an integer that is less than 1001.")
elsif (options['event_limit'].to_i > 1000)
errors.add(:base, "For performance reasons, you cannot have an event limit greater than 1000.")
end
else
end
end
def receive(incoming_events)
return unless ['merge events', 'last event in'].include?(mode)
memory['last_event'] ||= {}
incoming_events.each do |event|
case mode
when 'merge events'
memory['last_event'] = memory['last_event'].merge(event.payload)
else
memory['last_event'] = event.payload
end
end
end
def receive_web_request(params, method, format)
valid_authentication?(params) ? [liquified_content, 200, mime_type]
: [unauthorized_content(format), 401]
end
private
def mode
options['mode'].to_s.downcase
end
def unauthorized_content(format)
format =~ /json/ ? { error: "Not Authorized" }
: "Not Authorized"
end
def valid_authentication?(params)
interpolated['secret'] == params['secret']
end
def mime_type
options['mime_type'].presence || 'text/html'
end
def liquified_content
template = Liquid::Template.parse(options['content'] || "")
template.render(data_for_liquid_template)
end
def data_for_liquid_template
case mode
when 'last x events'
events = received_events
events = events.where('events.created_at > ?', date_limit) if date_limit
events = events.limit count_limit
events = events.to_a.map { |x| x.payload }
{ 'events' => events }
else
memory['last_event'] || {}
end
end
def count_limit
limit = Integer(options['event_limit']) rescue 1000
limit <= 1000 ? limit : 1000
end
def date_limit
return nil unless options['event_limit'].to_s.include?(' ')
value, unit = options['event_limit'].split(' ')
value = Integer(value) rescue nil
return nil unless value
unit = unit.to_s.downcase
return nil unless DATE_UNITS.include?(unit)
value.send(unit.to_sym).ago
end
end
end

View file

@ -0,0 +1,15 @@
<p>
Data for this Agent is available at these URLs:
</p>
<ul>
<% url = lambda { |format| web_requests_url(:agent_id => @agent.id, :user_id => current_user.id, :secret => @agent.options['secret'], :format => format) } %>
<li><%= link_to url.call(:html), url.call(:html), :target => :blank %></li>
<li><%= link_to url.call(:json), url.call(:json), :target => :blank %></li>
<li><%= link_to url.call(:xml), url.call(:xml), :target => :blank %></li>
</ul>
<p>
... or any other extension you wish, as the extension does not change the content or mime type.
</p>

View file

@ -0,0 +1,461 @@
# encoding: utf-8
require 'rails_helper'
describe Agents::LiquidOutputAgent do
let(:agent) do
_agent = Agents::LiquidOutputAgent.new(:name => 'My Data Output Agent')
_agent.options = _agent.default_options.merge('secret' => 'secret1', 'events_to_show' => 3)
_agent.options['secret'] = "a secret"
_agent.user = users(:bob)
_agent.sources << agents(:bob_website_agent)
_agent.save!
_agent
end
describe "#working?" do
it "checks if events have been received within expected receive period" do
expect(agent).not_to be_working
Agents::LiquidOutputAgent.async_receive agent.id, [events(:bob_website_agent_event).id]
expect(agent.reload).to be_working
two_days_from_now = 2.days.from_now
stub(Time).now { two_days_from_now }
expect(agent.reload).not_to be_working
end
end
describe "validation" do
before do
expect(agent).to be_valid
end
it "should validate presence and length of secret" do
agent.options[:secret] = ""
expect(agent).not_to be_valid
agent.options[:secret] = "foo"
expect(agent).to be_valid
agent.options[:secret] = "foo/bar"
expect(agent).not_to be_valid
agent.options[:secret] = "foo.xml"
expect(agent).not_to be_valid
agent.options[:secret] = false
expect(agent).not_to be_valid
agent.options[:secret] = []
expect(agent).not_to be_valid
agent.options[:secret] = ["foo.xml"]
expect(agent).not_to be_valid
agent.options[:secret] = ["hello", true]
expect(agent).not_to be_valid
agent.options[:secret] = ["hello"]
expect(agent).not_to be_valid
agent.options[:secret] = ["hello", "world"]
expect(agent).not_to be_valid
end
it "should validate presence of expected_receive_period_in_days" do
agent.options[:expected_receive_period_in_days] = ""
expect(agent).not_to be_valid
agent.options[:expected_receive_period_in_days] = 0
expect(agent).not_to be_valid
agent.options[:expected_receive_period_in_days] = -1
expect(agent).not_to be_valid
end
it "should validate the event_limit" do
agent.options[:event_limit] = ""
expect(agent).to be_valid
agent.options[:event_limit] = "1"
expect(agent).to be_valid
agent.options[:event_limit] = "1001"
expect(agent).not_to be_valid
agent.options[:event_limit] = "10000"
expect(agent).not_to be_valid
end
it "should should not allow non-integer event limits" do
agent.options[:event_limit] = "abc1234"
expect(agent).not_to be_valid
end
end
describe "#receive?" do
let(:key) { SecureRandom.uuid }
let(:value) { SecureRandom.uuid }
let(:incoming_events) do
last_payload = { key => value }
[Struct.new(:payload).new( { key => SecureRandom.uuid } ),
Struct.new(:payload).new( { key => SecureRandom.uuid } ),
Struct.new(:payload).new(last_payload)]
end
describe "and the mode is last event in" do
before { agent.options['mode'] = 'Last event in' }
it "stores the last event in memory" do
agent.receive incoming_events
expect(agent.memory['last_event'][key]).to equal(value)
end
describe "but the casing is wrong" do
before { agent.options['mode'] = 'LAST EVENT IN' }
it "stores the last event in memory" do
agent.receive incoming_events
expect(agent.memory['last_event'][key]).to equal(value)
end
end
end
describe "but the mode is merge" do
let(:second_key) { SecureRandom.uuid }
let(:second_value) { SecureRandom.uuid }
before { agent.options['mode'] = 'Merge events' }
let(:incoming_events) do
last_payload = { key => value }
[Struct.new(:payload).new( { key => SecureRandom.uuid, second_key => second_value } ),
Struct.new(:payload).new(last_payload)]
end
it "should merge all of the events passed to it" do
agent.receive incoming_events
expect(agent.memory['last_event'][key]).to equal(value)
expect(agent.memory['last_event'][second_key]).to equal(second_value)
end
describe "but the casing on the mode is wrong" do
before { agent.options['mode'] = 'MERGE EVENTS' }
it "should merge all of the events passed to it" do
agent.receive incoming_events
expect(agent.memory['last_event'][key]).to equal(value)
expect(agent.memory['last_event'][second_key]).to equal(second_value)
end
end
end
describe "but the mode is anything else" do
before { agent.options['mode'] = SecureRandom.uuid }
let(:incoming_events) do
last_payload = { key => value }
[Struct.new(:payload).new(last_payload)]
end
it "should do nothing" do
agent.receive incoming_events
expect(agent.memory.keys.count).to equal(0)
end
end
end
describe "#count_limit" do
it "should have a default of 1000" do
agent.options['event_limit'] = nil
expect(agent.send(:count_limit)).to eq(1000)
agent.options['event_limit'] = ''
expect(agent.send(:count_limit)).to eq(1000)
agent.options['event_limit'] = ' '
expect(agent.send(:count_limit)).to eq(1000)
end
it "should convert string count limits to integers" do
agent.options['event_limit'] = '1'
expect(agent.send(:count_limit)).to eq(1)
agent.options['event_limit'] = '2'
expect(agent.send(:count_limit)).to eq(2)
agent.options['event_limit'] = 3
expect(agent.send(:count_limit)).to eq(3)
end
it "should default to 1000 with invalid values" do
agent.options['event_limit'] = SecureRandom.uuid
expect(agent.send(:count_limit)).to eq(1000)
agent.options['event_limit'] = 'John Galt'
expect(agent.send(:count_limit)).to eq(1000)
end
it "should not allow event limits above 1000" do
agent.options['event_limit'] = '1001'
expect(agent.send(:count_limit)).to eq(1000)
agent.options['event_limit'] = '5000'
expect(agent.send(:count_limit)).to eq(1000)
end
end
describe "#receive_web_request?" do
let(:secret) { SecureRandom.uuid }
let(:params) { { 'secret' => secret } }
let(:method) { nil }
let(:format) { nil }
let(:mime_type) { SecureRandom.uuid }
let(:content) { "The key is {{#{key}}}." }
let(:key) { SecureRandom.uuid }
let(:value) { SecureRandom.uuid }
before do
agent.options['secret'] = secret
agent.options['mime_type'] = mime_type
agent.options['content'] = content
agent.memory['last_event'] = { key => value }
agents(:bob_website_agent).events.destroy_all
end
describe "and the mode is last event in" do
before { agent.options['mode'] = 'Last event in' }
it "should render the results as a liquid template from the last event in" do
result = agent.receive_web_request params, method, format
expect(result[0]).to eq("The key is #{value}.")
expect(result[1]).to eq(200)
expect(result[2]).to eq(mime_type)
end
describe "but the casing is wrong" do
before { agent.options['mode'] = 'last event in' }
it "should render the results as a liquid template from the last event in" do
result = agent.receive_web_request params, method, format
expect(result[0]).to eq("The key is #{value}.")
expect(result[1]).to eq(200)
expect(result[2]).to eq(mime_type)
end
end
end
describe "and the mode is merge events" do
before { agent.options['mode'] = 'Merge events' }
it "should render the results as a liquid template from the last event in" do
result = agent.receive_web_request params, method, format
expect(result[0]).to eq("The key is #{value}.")
expect(result[1]).to eq(200)
expect(result[2]).to eq(mime_type)
end
end
describe "and the mode is last X events" do
before do
agent.options['mode'] = 'Last X events'
agents(:bob_website_agent).create_event payload: {
"name" => "Dagny Taggart",
"book" => "Atlas Shrugged"
}
agents(:bob_website_agent).create_event payload: {
"name" => "John Galt",
"book" => "Atlas Shrugged"
}
agents(:bob_website_agent).create_event payload: {
"name" => "Howard Roark",
"book" => "The Fountainhead"
}
agent.options['content'] = <<EOF
<table>
{% for event in events %}
<tr>
<td>{{ event.name }}</td>
<td>{{ event.book }}</td>
</tr>
{% endfor %}
</table>
EOF
end
it "should render the results as a liquid template from the last event in, limiting to 2" do
agent.options['event_limit'] = 2
result = agent.receive_web_request params, method, format
expect(result[0]).to eq <<EOF
<table>
<tr>
<td>Howard Roark</td>
<td>The Fountainhead</td>
</tr>
<tr>
<td>John Galt</td>
<td>Atlas Shrugged</td>
</tr>
</table>
EOF
end
it "should render the results as a liquid template from the last event in, limiting to 1" do
agent.options['event_limit'] = 1
result = agent.receive_web_request params, method, format
expect(result[0]).to eq <<EOF
<table>
<tr>
<td>Howard Roark</td>
<td>The Fountainhead</td>
</tr>
</table>
EOF
end
it "should render the results as a liquid template from the last event in, allowing no limit" do
agent.options['event_limit'] = ''
result = agent.receive_web_request params, method, format
expect(result[0]).to eq <<EOF
<table>
<tr>
<td>Howard Roark</td>
<td>The Fountainhead</td>
</tr>
<tr>
<td>John Galt</td>
<td>Atlas Shrugged</td>
</tr>
<tr>
<td>Dagny Taggart</td>
<td>Atlas Shrugged</td>
</tr>
</table>
EOF
end
it "should allow the limiting by time, as well" do
one_event = agent.received_events.select { |x| x.payload['name'] == 'John Galt' }.first
one_event.created_at = 2.days.ago
one_event.save!
agent.options['event_limit'] = '1 day'
result = agent.receive_web_request params, method, format
expect(result[0]).to eq <<EOF
<table>
<tr>
<td>Howard Roark</td>
<td>The Fountainhead</td>
</tr>
<tr>
<td>Dagny Taggart</td>
<td>Atlas Shrugged</td>
</tr>
</table>
EOF
end
it "should not be case sensitive when limiting on time" do
one_event = agent.received_events.select { |x| x.payload['name'] == 'John Galt' }.first
one_event.created_at = 2.days.ago
one_event.save!
agent.options['event_limit'] = '1 DaY'
result = agent.receive_web_request params, method, format
expect(result[0]).to eq <<EOF
<table>
<tr>
<td>Howard Roark</td>
<td>The Fountainhead</td>
</tr>
<tr>
<td>Dagny Taggart</td>
<td>Atlas Shrugged</td>
</tr>
</table>
EOF
end
it "it should continue to work when the event limit is wrong" do
agent.options['event_limit'] = 'five days'
result = agent.receive_web_request params, method, format
expect(result[0].include?("Howard Roark")).to eq(true)
expect(result[0].include?("Dagny Taggart")).to eq(true)
expect(result[0].include?("John Galt")).to eq(true)
agent.options['event_limit'] = '5 quibblequarks'
result = agent.receive_web_request params, method, format
expect(result[0].include?("Howard Roark")).to eq(true)
expect(result[0].include?("Dagny Taggart")).to eq(true)
expect(result[0].include?("John Galt")).to eq(true)
end
describe "but the mode was set to last X events with the wrong casing" do
before { agent.options['mode'] = 'LAST X EVENTS' }
it "should still work as last x events" do
result = agent.receive_web_request params, method, format
expect(result[0].include?("Howard Roark")).to eq(true)
expect(result[0].include?("Dagny Taggart")).to eq(true)
expect(result[0].include?("John Galt")).to eq(true)
end
end
end
describe "but the secret provided does not match" do
before { params['secret'] = SecureRandom.uuid }
it "should return a 401 response" do
result = agent.receive_web_request params, method, format
expect(result[0]).to eq("Not Authorized")
expect(result[1]).to eq(401)
end
it "should return a 401 json response if the format is json" do
result = agent.receive_web_request params, method, 'json'
expect(result[0][:error]).to eq("Not Authorized")
expect(result[1]).to eq(401)
end
end
end
end