GitHub-Mirror/huginn
2
0
Fork 0
mirror of https://github.com/Fishwaldo/huginn.git synced 2025-03-16 11:51:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
2632 commits 15 branches 0 tags 8.2 MiB
Commit graph

2632 commits

This branch
This branch
All branches
Author SHA1 Message Date
Akinori MUSHA
5f5f3cd38f Do not err if headers is a valid headers hash 2016-11-27 13:38:23 +09:00
Akinori MUSHA
9074f3115e Make sure status is an integer when set 2016-11-27 13:38:23 +09:00
Akinori MUSHA
a94cd7fd6d Allow an empty or null base URI 2016-11-27 13:38:23 +09:00
Akinori MUSHA
3a0c9e6274 Disable automatic URL normalization and absolutization on url 
This was discussed in #1766.

For backward compatibility, existing WebsiteAgents with a key named
`url` will be given a `template` to resolve `url`.
 2016-11-27 13:38:23 +09:00
Akinori MUSHA
3d91469733 Make WebsiteAgent merge template with the results of extract (#1816) 
A new extraction option `hidden` is added so that keys with it gets
excluded from the final payloads while they can be used in `template`.
 2016-11-27 13:33:24 +09:00
bobbysteel
f530305edc Add class of service chooser for Google Flights Agent (#1778) 
* Add class of service chooser

* Add cabin chooser test

* Fix preferredCabin

* Per @cantino feedback taking out check
 2016-11-26 15:13:00 -05:00
Akinori MUSHA
7ac691652b Spec that force_encoding works with encoding declaration in RssAgent 2016-11-26 13:26:09 +09:00
Akinori MUSHA
b8d88aa9a3 Merge pull request #1813 from cantino/fix_decoding_in_rss_agent 
Fix a double-decoding problem in RssAgent
 2016-11-23 09:05:46 +09:00
Dominik Sander
5d69bd2d93 Merge pull request #1804 from dsander/cache-agent-type-dropdown 
Cache Agent type select options in Agent#new
 2016-11-22 21:04:29 +01:00
Dominik Sander
6daa6cc75e Cache Agent type select options in Agent#new 
Agent#new was by far the slowest action of all controllers. When using
many Agent gems the response time goes up to a point where the user
starts to wonder if something is going wrong. By caching the Agent
description the response time goes down from about 1 second to 100ms
in development.
 2016-11-22 20:02:27 +01:00
Akinori MUSHA
aa6d8be697 Assume that an XML declaration is at the beginning of a document 2016-11-23 01:17:46 +09:00
Akinori MUSHA
0b3700999b Fix a double-decoding problem in RssAgent 
The SAX parser Feedjira uses (Nokogiri::XML::SAX) tries to detect the
encoding of a document from the content even if it is already known
and given.  This results in a content being decoded twice by
WebRequestConcern and the SAX parser if its encoding is declared in
both the Content-Type header and the XML declaration.

This commit makes RssAgent remove the `encoding` attribute from the
XML declaration of a document if the encoding is already known by the
Content-Type header.

Fixes #1797.
 2016-11-22 12:14:28 +09:00
Andrew Cantino
6fb8fe2292 Increase default database pool size to 20 (#1805) 2016-11-20 09:48:34 -05:00
Akinori MUSHA
c575af959b Merge pull request #1769 from cantino/website_agent_repeat_option 
Add a `repeat` option for extractors to WebsiteAgent
 2016-11-20 19:16:36 +09:00
The Doctor
5e1191534c Fixed the online documentation for the Weather Agent class. (#1803) 
Signed-off-by: The Doctor <drwho@virtadpt.net>
 2016-11-19 10:39:15 -05:00
Akinori MUSHA
bd9455d5d0 Add a repeat option to extractors 
This allows user to include a value that only appears once in a content
in all events created from the content.
 2016-11-18 18:47:32 +09:00
Akinori MUSHA
3a66c152ef Make extract_each prepare a storage for results 2016-11-18 16:34:45 +09:00
Dominik Sander
e26976be76 Merge pull request #1792 from bugdone/master 
Fix typos in docker documentation
 2016-11-14 23:04:50 +01:00
bugdone
a51d8169dc Fix typos in docker documentation 2016-11-14 22:24:35 +02:00
Andrew Cantino
770706463b Prevent submit from disabling on invalid json (#1790) 2016-11-13 15:21:43 -05:00
Andrew Cantino
085473263f Remove additional nitrous files (#1791) 2016-11-13 15:21:30 -05:00
Andrew Cantino
fca06d6ec2 Nitrous.io is shutting down (#1789) 2016-11-13 14:35:25 -05:00
Akinori MUSHA
b6c1e908c8 Update nokogiri to 1.6.8.1 2016-11-11 15:35:24 +09:00
Akinori MUSHA
74077b0ad4 Auto-focus on Agent Type when creating an agent 2016-11-09 23:25:55 +09:00
Akinori MUSHA
486246e63c Add "image" to Event Description 2016-11-07 12:47:13 +09:00
Akinori MUSHA
1e70b31e7f Merge pull request #1770 from cantino/revert-1071 
Revert the special treatment for CDATA introduced in #1071
 2016-11-03 12:14:42 +09:00
Akinori MUSHA
50123dca53 Fix event_description broken in full JSON mode or without a template 2016-11-02 21:47:01 +09:00
Akinori MUSHA
898e3d8edb Revert the special treatment for CDATA introduced in #1071 2016-11-02 19:35:16 +09:00
Akinori MUSHA
07effe5eb4 Merge pull request #1766 from cantino/fix_website_agent_url_handling 
Fix `url` handling of WebsiteAgent
 2016-11-02 13:36:38 +09:00
Akinori MUSHA
e50b8e0d5c Document that url in a created event is automatically resolved 2016-11-02 11:14:02 +09:00
Akinori MUSHA
cc16e854b3 Fix a problem in resolving the url key of a created event 
The `url` parameter of handle_data() could hold a string or nil when
invoked from handle_event_data(), in which case resolving `url` in a
created event would fail with a type error.  Moreover, the logic did not
have any guard for URI errors.  This commit should fix them.

Fixes #1765.
 2016-11-02 11:14:02 +09:00
Akinori MUSHA
4fe35b2a1f Reproduce #1765 2016-11-01 22:29:56 +09:00
Akinori MUSHA
91f096b16f Merge pull request #1743 from cantino/website_agent_can_interpolate_after_extraction 
WebsiteAgent can interpolate after extraction

Incorporating feedback from @cantino and @dsander.
 2016-11-01 20:20:37 +09:00
Dominik Sander
e3f1429a37 Merge pull request #1764 from strugee/http-to-https 
Fix Stubhub test failures
 2016-11-01 09:43:22 +01:00
Alex Jordan
651eb50729 Fix another Stubhub HTTP URL 2016-10-31 20:58:06 -07:00
Alex Jordan
77da54ea0c Convert a bunch of HTTP links to HTTPS (#1757) 2016-10-31 19:21:03 -04:00
Akinori MUSHA
58fabb885c Add a new Liquid filter rebase_hrefs 2016-10-29 20:40:52 +09:00
Akinori MUSHA
8b897f5da3 Add Liquid variables _response_.url and _url_ to WebsiteAgent 2016-10-29 20:40:51 +09:00
Akinori MUSHA
fe35df8752 Add a new option template to WebsiteAgent 
If given, it is used as a Liquid template for each event created by the
Agent, instead of directly emitting the results of extraction as events.

An existing spec needs to be fixed because WebsiteAgent now has the
`template` option, which may not be a hash of hashes.
 2016-10-29 20:40:51 +09:00
Andrew Cantino
9a3290ef40 Language changes 2016-10-28 19:05:46 -04:00
Akinori MUSHA
faa2789a0c Fix the order of receivers in the DotHelper specs 
This should fix occasional build failure on CI.
 2016-10-27 16:31:24 +09:00
Akinori MUSHA
4f93db60e7 Merge pull request #1754 from cantino/ignore_empty_author 
Ignore empty author and link entries in RssAgent.

Fixes #1753.
 2016-10-27 13:07:21 +09:00
Akinori MUSHA
1e14358648 Merge pull request #1751 from cantino/encoding_detection 
Improve encoding detection in WebsiteAgent
 2016-10-27 13:00:56 +09:00
Akinori MUSHA
50b5833a3f Improve encoding detection in WebsiteAgent 
Previously, WebsiteAgent always assumed that a content with no charset
specified in the Content-Type header would be encoded in UTF-8.  This
enhancement is to make use of the encoding detector implemented in
Nokogiri for HTML/XML documents, instead of blindly falling back to
UTF-8.

When the document `type` is `html` or `xml`, WebsiteAgent tries to
detect the encoding of a fetched document from the presence of a BOM,
XML declaration, or HTML `meta` tag.

This fixes #1742.
 2016-10-27 13:00:37 +09:00
Akinori MUSHA
4d10132709 Fix a bug where an empty <link> is wrongly parsed 
Due to a problem in sax-machine's internals, an empty `<link/>` in RSS
would be parsed to JSON as `{ "href": "no_buffer" }`.  Now empty
`<link/>` elements in RSS and ATOM are simply ignored just like other
collection elements like `<category>`.
 2016-10-27 09:17:46 +09:00
Akinori MUSHA
2bb97b53bc Add failing specs for empty <link> elements 2016-10-27 09:17:40 +09:00
Akinori MUSHA
5f5e246552 Use Struct#each_pair 2016-10-27 08:13:43 +09:00
Akinori MUSHA
e5c938aa85 Exclude empty entries from authors 2016-10-27 08:12:48 +09:00
Akinori MUSHA
852f39d480 Rescue error from Mail::Address#name and #address 
`Mail::Address.new('')` does not raise any error but calling `name` on
the created instance does.
 2016-10-27 08:11:10 +09:00
Akinori MUSHA
445665ee3a Add a failing test for #1753 2016-10-27 08:09:57 +09:00