Commit graph

1638 commits

Author SHA1 Message Date
Akinori MUSHA
898e3d8edb Revert the special treatment for CDATA introduced in #1071 2016-11-02 19:35:16 +09:00
Akinori MUSHA
e50b8e0d5c Document that url in a created event is automatically resolved 2016-11-02 11:14:02 +09:00
Akinori MUSHA
cc16e854b3 Fix a problem in resolving the url key of a created event
The `url` parameter of handle_data() could hold a string or nil when
invoked from handle_event_data(), in which case resolving `url` in a
created event would fail with a type error.  Moreover, the logic did not
have any guard for URI errors.  This commit should fix them.

Fixes #1765.
2016-11-02 11:14:02 +09:00
Akinori MUSHA
91f096b16f Merge pull request #1743 from cantino/website_agent_can_interpolate_after_extraction
WebsiteAgent can interpolate after extraction

Incorporating feedback from @cantino and @dsander.
2016-11-01 20:20:37 +09:00
Alex Jordan
77da54ea0c Convert a bunch of HTTP links to HTTPS (#1757) 2016-10-31 19:21:03 -04:00
Akinori MUSHA
58fabb885c Add a new Liquid filter rebase_hrefs 2016-10-29 20:40:52 +09:00
Akinori MUSHA
8b897f5da3 Add Liquid variables _response_.url and _url_ to WebsiteAgent 2016-10-29 20:40:51 +09:00
Akinori MUSHA
fe35df8752 Add a new option template to WebsiteAgent
If given, it is used as a Liquid template for each event created by the
Agent, instead of directly emitting the results of extraction as events.

An existing spec needs to be fixed because WebsiteAgent now has the
`template` option, which may not be a hash of hashes.
2016-10-29 20:40:51 +09:00
Andrew Cantino
9a3290ef40 Language changes 2016-10-28 19:05:46 -04:00
Akinori MUSHA
50b5833a3f Improve encoding detection in WebsiteAgent
Previously, WebsiteAgent always assumed that a content with no charset
specified in the Content-Type header would be encoded in UTF-8.  This
enhancement is to make use of the encoding detector implemented in
Nokogiri for HTML/XML documents, instead of blindly falling back to
UTF-8.

When the document `type` is `html` or `xml`, WebsiteAgent tries to
detect the encoding of a fetched document from the presence of a BOM,
XML declaration, or HTML `meta` tag.

This fixes #1742.
2016-10-27 13:00:37 +09:00
Akinori MUSHA
7ed40a6901 Use the XPath expression string(.) instead of .//text()
That is the correct way to extract a raw string with all text nodes
concatenated without entity escaping.
2016-10-21 00:23:00 +09:00
Akinori MUSHA
0fcd8e285e Normalize URL in to_uri and uri_expand liquid filters 2016-10-17 15:02:57 +09:00
Dominik Sander
bf35a49f6c Fix delayed_job_active_record overriding defaults
Because `delayed_job_active_record` was loading active_record to early it used to override defaults set in our
initializers.
2016-10-15 12:45:23 +02:00
Dominik Sander
005f01a4ad Merge pull request #1716 from dsander/liquid-as-object
Add as_object Liquid filter
2016-10-14 12:53:24 +02:00
Dominik Sander
d2cbd04ac8 Add as_object Liquid filter
The `as_object` returns the received data/object as is without casting it to a string like liquid normally does. It
can be used as a JSONPath replacement or to emit result of a Liquid filter chain as an array.

`catch` and `throw` needs to be used to break out of Liquid render chain. Liquid aggregates the output of every
expression an array and [joins](https://github.com/Shopify/liquid/blob/v3.0.6/lib/liquid/block.rb#L147) it together that
join makes it impossible to get anything else than a string out of a Liquid template.
2016-10-14 12:33:31 +02:00
YURA BUYLOV
864b0708a7 Agent form: ace-editor highlighting and theme (#1727) 2016-10-11 19:25:12 -04:00
Enfop
8efe50ad99 manual event agent validate JSON field before form submit (#1728) 2016-10-11 07:53:56 -04:00
Andrew Cantino
005be58d30 Update forecast_io gem and language (#1722) 2016-10-09 10:49:12 -04:00
Akinori MUSHA
12cecb8392 Merge branch 'data_output_agent_limits_events_after_ordering' 2016-10-07 19:22:42 +09:00
Akinori MUSHA
63c7ec5fdd Reorder received_events by id asc
The implementation of latest_events() assumed that while it wasn't
actually the case; the association of received_events has a default
scope of `order(id: :desc)`.
2016-10-07 19:22:00 +09:00
Akinori MUSHA
d63f714ae9 RssAgent: Migrate from FeedNormalizer to Feedjira
FeedNormalizer is no longer maintained, and its Atom support has flaws
in that it throws away what RSS::Parser returns and falls back to using
SimpleRSS which is not capable of handling XML entities, resulting in
getting ususable URLs such as ones including `&`.

Feedjira is highly customizable as it implements parsers for various
feed formats on its own using sax-machine.

The `clean` option is reimplemented using the loofah gem.
`Feedjira::FeedUtilities#sanitize_entries!` is not used because it
tries to sanitize non-HTML properties too.

A new option `include_feed_info` is added, with which turned on feed
information is added to each event payload.

A new key `links` is added, which lists all `link` elements.

A new key `enclosure` is added.
2016-10-06 11:49:34 +09:00
Dominik Sander
bade1a1a32 Work around counter cache bug when not selecting the cached column 2016-10-05 22:47:38 +02:00
Dominik Sander
5cfca6bf68 Ensure helper classes are available in production
Due to eager_loading being active in production helper classes in lib need to be explicitly required.
2016-10-05 22:47:38 +02:00
Dominik Sander
b651f08cc5 Fix the specs on rails5 2016-10-05 22:47:38 +02:00
Dominik Sander
26e878c61a Fix the rest of the deprecation warnings 2016-10-05 22:47:38 +02:00
Dominik Sander
5fbeb105b2 Update gems and configs to Rails5 2016-10-05 22:47:38 +02:00
Akinori MUSHA
44efff5a46 Fix grammar 2016-10-05 09:28:24 +09:00
Akinori MUSHA
a47c5c8a80 PushoverAgent: Treat parameter options as templates rather than default values
This should fix #1718.

While at it, character limit implementation is fixed: `slice(0..n)`
would take the first n+1 characters instead of n.
2016-10-04 21:57:30 +09:00
Dominik Sander
4309460896 Fix escape characters of events when dry running
JSONEditor double escapes escape characters to properly display them in the UI, however when dry running Agents we
actually need to send the Event's characters as they were.
Before an event containing new lines like this `Line A\nLineB` would be send as `Line A\\nLineB` to the backend. By
replacing the double escaped slashes we ensure the event is handled as it would be when not dry running.

 #1677
2016-09-30 17:06:50 +02:00
bobbysteel
1497712e04 Clarify path for a simple body_text event. (#1705)
* Clarify path for a simple body_text event. 

It's not really clear with regard to a normal agent that this is simply the name of the field.

* updated text
2016-09-23 15:36:18 -04:00
Dominik Sander
e0fa7f00d0 Merge pull request #1695 from kreuzwerker/missing-agent-warning
Rescue from AR:SubclassNotFound and allow to delete agents
2016-09-21 21:04:46 +02:00
Dominik Sander
d2b26a0684 Merge pull request #1698 from kreuzwerker/fix-form-completion
Replace jquery.serializeObject with new implementation
2016-09-21 20:58:03 +02:00
Dominik Sander
42b132017f Rescue from AR:SubclassNotFound and allow to delete agents
Especially when using Agent gems it happens that an Agent was deleted from the gem or the user removed a gem from the
configuration. Instead of failing with an Internal Server Error the user now is offered to delete all undefined agents.

`type.constantize` is needed to invoke the Rails auto_loader in development since not all Agent classes are loaded when
the application boots.
2016-09-21 20:10:49 +02:00
Dominik Sander
517e42c9d2 Replace jquery.serializeObject with new implementation
The old serializeObject function serialized checked checkboxes as an array which was not correct and with
strong_parameters enabled it triggered a `UnpermittedParameters` error.

In the AgentEditPage javascript we only need to initialize `select2` when then `#agent_type` field is a select
box (trying to call `select2` on normal text fields caused a javascript error in the feature specs)
2016-09-21 20:05:18 +02:00
Andrew Cantino
0dd1c78ae0 Ignore the fixture API key in WeatherAgent (#1694) 2016-09-20 20:31:44 -04:00
Dominik Sander
663eebc080 Add multipart file upload to PostAgent
This allows the PostAgent to receive `file_pointer` events and upload the data to a website.
2016-09-19 10:39:48 +02:00
Thiago Talma
4fdd69c6f4 interpolated response (#1682) 2016-09-15 19:15:46 -04:00
Dominik Sander
58f388d530 service_id is a valid part of the agent_params
Fixes #1685
2016-09-15 16:17:46 +02:00
Dominik Sander
b2f031003f Use strong_parameters and drop protected_attributes
* Ensure admin attribute can not be set at sign up and the agents user_id is not changeable
* Remove ACCESSIBLE_ATTRIBUTES from the User model
* Remove side effect on agent_params from AgentsController#build_agent
2016-09-12 15:54:26 +02:00
Andrew Cantino
7550274578 Remove unused contact model (#1680) 2016-09-10 10:24:23 -04:00
Thiago Talma
c3d5380f98 Option to set response code for webhook agent (#1676)
* setting for webhook response code

* some adjustments
2016-09-09 09:53:29 -04:00
Karol Bucek
c6a17c90fe can_enqueue? propagation detection that does not depend on Rails (#1672) 2016-09-09 09:50:32 -04:00
Dominik Sander
f42ad96ed6 Merge pull request #1645 from dsander/retry-twitter-action-agent
Allow TwitterUserAgent to retry failed actions
2016-09-08 20:44:20 +02:00
Dominik Sander
3e76fa0e41 Add cache_response option to completable form fields
When `cache_response` is set to `false` for completable fields the returned options will not be
cached on the client side. The default is `true`.

Clearing the cache is needed when the completable response depends on a different option of the Agent.

Usage Example:

```
form_configurable :models, roles: :completable, cache_response: false
```

* Disable the completion callback when `select2` is opened programmatically to avoid an infinite loop
* Moved the completion ajax call from `select2-open` to `select2-opening` to clear the cached results before
 `completableDefaultOptions` is called (shows the "Loading ..." message instead of the previous results)
2016-09-08 12:21:49 +02:00
Karol Bucek
8ca39a5802 minor tweaks (#1669)
* avoid unnecesary  method pollution

* production.rake typo fix

* remove un-used local variable assignment
2016-09-06 13:36:10 -04:00
Will Read
6b459ef412 Extract service option prviders for non-standard Omniauth payloads (#1655)
* Extract option prviders for non-standard Omniauth payloads

- should enable huginn_agent gem developers to add their own providers
- devs would reopen OmniauthCallbacksController and define their specific action, supplying their own option provider

* Creates a registry for option providers on the Service class

- enables huginn_agent gem developers to register their own omniauth option interpreters

* Cleans up changes to the global Service class after the spec
2016-09-04 19:39:49 -04:00
Albert Sun
6bb61065ec Allow admins to login as another user (#1659)
* switch_to_user and switch_back routes

* Uses session to store original user id
2016-09-02 15:32:38 -04:00
Géza Búza
c3ab764efc Feature/add language option to weather agent (#1544)
* Add language parameter to the weather agent

* Add text forecast for Wunderground data provider

Wunderground API returns forecast for the next few days. Returned JSON response has two types of data under the "forecast" key:
- "txt_forecast": pre-formatted text representation
- "simpleforecast": raw data representation

The "period" key contains which day the node belongs to.

Currently only "simpleforecast" is available to users of the Weather agent.
This commit appends the "txt_forecast" representation to the current data set.
2016-08-31 08:37:51 -07:00
Dominik Sander
49d3611273 Allow TwitterUserAgent to retry failed actions
Add new `emit_error_events` option which is set to `false` for all new `TwitterActionAgent`s. This will ensure the Agent
is retried when the action failed by raising an exception.

For all existing `TwitterUserAgent`s the option is set to `true` to keep the previous behavior.
2016-08-25 19:49:31 +02:00
Will Read
a6fe9743f6 EmailDigestAgent relies on received events, rather in memory (#1624)
* EmailDigestAgent relies on received events, rather than storing them in memory

- Addressing a problem where 40,000+ events were stored in one agent's memory, and the agent never loaded from the database

* Cleans up references to memory["queue"]

- incorporates suggested style and safeguards

* Adds clarification on event inclusion in email

* Use `reorder` rather than stripping away all scoping
2016-08-16 23:05:16 -04:00