Previously, WebsiteAgent always assumed that a content with no charset
specified in the Content-Type header would be encoded in UTF-8. This
enhancement is to make use of the encoding detector implemented in
Nokogiri for HTML/XML documents, instead of blindly falling back to
UTF-8.
When the document `type` is `html` or `xml`, WebsiteAgent tries to
detect the encoding of a fetched document from the presence of a BOM,
XML declaration, or HTML `meta` tag.
This fixes#1742.
The `as_object` returns the received data/object as is without casting it to a string like liquid normally does. It
can be used as a JSONPath replacement or to emit result of a Liquid filter chain as an array.
`catch` and `throw` needs to be used to break out of Liquid render chain. Liquid aggregates the output of every
expression an array and [joins](https://github.com/Shopify/liquid/blob/v3.0.6/lib/liquid/block.rb#L147) it together that
join makes it impossible to get anything else than a string out of a Liquid template.
The implementation of latest_events() assumed that while it wasn't
actually the case; the association of received_events has a default
scope of `order(id: :desc)`.
FeedNormalizer is no longer maintained, and its Atom support has flaws
in that it throws away what RSS::Parser returns and falls back to using
SimpleRSS which is not capable of handling XML entities, resulting in
getting ususable URLs such as ones including `&`.
Feedjira is highly customizable as it implements parsers for various
feed formats on its own using sax-machine.
The `clean` option is reimplemented using the loofah gem.
`Feedjira::FeedUtilities#sanitize_entries!` is not used because it
tries to sanitize non-HTML properties too.
A new option `include_feed_info` is added, with which turned on feed
information is added to each event payload.
A new key `links` is added, which lists all `link` elements.
A new key `enclosure` is added.
* Improve Utils.normalize_uri
Globally replacing generally unsafe characters in a URL would not fix
invalid authorities and paths, so use Addressable::URI to normalize them
when necessary.
This should fix#1701.
* Remove an unused function
* Fix the test case to make sure an IPv6 address is supported
JSONEditor double escapes escape characters to properly display them in the UI, however when dry running Agents we
actually need to send the Event's characters as they were.
Before an event containing new lines like this `Line A\nLineB` would be send as `Line A\\nLineB` to the backend. By
replacing the double escaped slashes we ensure the event is handled as it would be when not dry running.
#1677
* Clarify path for a simple body_text event.
It's not really clear with regard to a normal agent that this is simply the name of the field.
* updated text
