Commit graph

139 commits

Author SHA1 Message Date
Akinori MUSHA
3612ba7333 Add podcast tags to events emitted by RssAgent
The keys are only added when the feed is a podcast.
2016-11-29 23:59:32 +09:00
Akinori MUSHA
58fabb885c Add a new Liquid filter rebase_hrefs 2016-10-29 20:40:52 +09:00
Akinori MUSHA
4d10132709 Fix a bug where an empty <link> is wrongly parsed
Due to a problem in sax-machine's internals, an empty `<link/>` in RSS
would be parsed to JSON as `{ "href": "no_buffer" }`.  Now empty
`<link/>` elements in RSS and ATOM are simply ignored just like other
collection elements like `<category>`.
2016-10-27 09:17:46 +09:00
Akinori MUSHA
5f5e246552 Use Struct#each_pair 2016-10-27 08:13:43 +09:00
Akinori MUSHA
e5c938aa85 Exclude empty entries from authors 2016-10-27 08:12:48 +09:00
Akinori MUSHA
852f39d480 Rescue error from Mail::Address#name and #address
`Mail::Address.new('')` does not raise any error but calling `name` on
the created instance does.
2016-10-27 08:11:10 +09:00
Akinori MUSHA
ba0d2ee0de Retire ar_mysql_column_charset
Rails 5 has native support for charset/collate.
2016-10-11 22:25:54 +09:00
Akinori MUSHA
df6fb7e5fc Fix super which is not defined yet here 2016-10-07 19:32:02 +09:00
Akinori MUSHA
d63f714ae9 RssAgent: Migrate from FeedNormalizer to Feedjira
FeedNormalizer is no longer maintained, and its Atom support has flaws
in that it throws away what RSS::Parser returns and falls back to using
SimpleRSS which is not capable of handling XML entities, resulting in
getting ususable URLs such as ones including `&amp;`.

Feedjira is highly customizable as it implements parsers for various
feed formats on its own using sax-machine.

The `clean` option is reimplemented using the loofah gem.
`Feedjira::FeedUtilities#sanitize_entries!` is not used because it
tries to sanitize non-HTML properties too.

A new option `include_feed_info` is added, with which turned on feed
information is added to each event payload.

A new key `links` is added, which lists all `link` elements.

A new key `enclosure` is added.
2016-10-06 11:49:34 +09:00
Akinori MUSHA
7e79d576b5 Improve Utils.normalize_uri (#1719)
* Improve Utils.normalize_uri

Globally replacing generally unsafe characters in a URL would not fix
invalid authorities and paths, so use Addressable::URI to normalize them
when necessary.

This should fix #1701.

* Remove an unused function

* Fix the test case to make sure an IPv6 address is supported
2016-10-05 14:17:04 +09:00
Karol Bucek
8ca39a5802 minor tweaks (#1669)
* avoid unnecesary  method pollution

* production.rake typo fix

* remove un-used local variable assignment
2016-09-06 13:36:10 -04:00
Dominik Sander
811f9c27d1 Allow specifing Agent gem in the Gemfile 2016-06-20 10:37:33 +02:00
Judy Ngai
1abcd947f5 Uploadicons (#1427)
* checking in work for glyph upload

* checking in icon selections

* added more icon options

* work on upload icon

* fix issue

* switched to options for select

* removed wrapper and removed commment

* add migration to support icon

* remove right floating

* rm params session add it to model

* rebuilt the menu

* rm extra div

* converted js to coffee

* quick change in indentation

* using the view helper

* submitting icons and loading into form

* added icon to both scenario export and import

* added icon to spec

* low level caching the icons

* tentative work in making a new coffee class

* renamed js page

* switched to cache in memory and worked on coffee class

* made pathsmatching compatible for all scenarios

* switched to rails root

* fixed all rubocop stylings

* fixed line is too long styling

* rm private method definition

* patch to avoid having no icon

* blank or nil?

* reordering the lines
2016-06-15 23:52:50 -07:00
Pablo Carranza
74a3c88608 Add time tracking to http status agent (#1517)
* Add time tracking to http status agent

* Replace time tracking to monotonic

* Add tests for time tracker
2016-05-29 19:13:40 -04:00
Dominik Sander
a6a06c1cd1 Separate feature and non feature spec runs on travis
Add database_cleaner gem
Set up coveralls parralel builds in travisci
Require capybara_helper in rails_helper to make clear it affects all tests
2016-05-12 10:03:23 +02:00
Dominik Sander
73bcd927e3 Do not run/enqueue event propagation when a AgentPropagateJob is already enqueued
Fixes the creation of duplicated events caused by Agent.receive! running multiple times concurrently.
2016-04-17 17:15:50 +02:00
Brian Petro
b941f740e5 replace delayed_job queue methods with ActiveJob counterparts 2016-04-12 09:49:49 -04:00
Dominik Sander
38f1044078 Add LocalFileAgent and introduce concept to handle files
The idea is to have source agents (`LocalFile` and `S3Agent` for now), which just emit a "file_pointer" and consuming
agents (`ReadFile` and `CsvAgent`) which get access to an `IO` object to actually access the file contents. This way we
do not have to pass megabytes of data through the database and the consuming agents can work with relatively unlimited
file sizes.
2016-03-18 20:43:28 +01:00
Dominik Sander
ee1ebea830 Allow to configure various devise settings in .env 2016-03-17 20:47:45 +01:00
Andrew Cantino
67fa70553d Allow a key with credential instead of a key_file 2016-02-01 20:29:40 -08:00
Akinori MUSHA
96b756ca11 Default ENV['RAILS_ENV'] to 'development' 2016-01-05 16:23:45 +09:00
Dominik Sander
daf45f7f0e Do not try to load .env file on heroku 2015-12-10 20:54:11 +01:00
eric
d233946feb Export Agents in GUID order so that the result is amenable to version control tracking 2015-12-07 10:30:14 -07:00
Dominik Sander
9c16b0fb51 Allow usage of PostgreSQL without exporting ON_HEROKU
Vendorizing the dotenv gem allows access to the `.env` configuration
in the `Gemfile`

Manual installation: Configure before installing gems
2015-12-04 21:31:45 +01:00
Andrew Cantino
9bdbdfa10d Merge remote-tracking branch 'origin/master' into prevent_extra_threads_in_specs 2015-11-20 08:56:30 -08:00
Andrew Cantino
60d880e8be remove p statements 2015-11-20 08:19:22 -08:00
Andrew Cantino
ff05700458 Prevent buildup of threads in specs 2015-11-19 21:39:03 -08:00
Akinori MUSHA
bde6a7269d Merge pull request #1125 from cantino/website_agent_normalize_uri
Introduce Utils.normalize_uri and use it in WebsiteAgent
2015-11-15 01:53:18 +09:00
Akinori MUSHA
cdfdc7f3c5 Introduce Utils.normalize_uri and use it in WebsiteAgent
This fixes #938, and the specs are from #958. (Thanks @irfancharania!)
2015-11-15 01:41:19 +09:00
Dominik Sander
28983a7378 Merge pull request #1142 from dsander/fix-twitter-agent-restart-leak
Do check restarting LongRunnable::Worker for liveliness
2015-11-12 17:26:08 +01:00
Dominik Sander
e303239281 Do check restarting LongRunnable::Worker for liveliness
Fixes a memory leak caused by starting a new Worker when the old one is still restarting.
2015-11-12 00:07:30 +01:00
Akinori MUSHA
43a6a025a4 Remove SchedulerAgent#check! which name is confusing 2015-11-06 17:11:09 +09:00
oroce
4487a9270d force-stopping services 2015-10-21 07:30:25 +02:00
Andrew Cantino
797e7e3b3c When payloads get truncated in the DB due to encoding issues, at least do not blow up on reading them 2015-09-30 20:28:30 -07:00
Dominik Sander
06b022f161 Merge pull request #819 from dsander/agent-runner
Add AgentRunner and LongRunnable to support long running agents
2015-09-09 16:36:56 +02:00
Dominik Sander
e0f14698ba Ensure database connection are checked back into the pool by wrapping calls in with_connection blocks 2015-09-07 15:52:54 +02:00
Dominik Sander
44ac1ed909 Restart TwitterStreamAgent::Worker every hour to prevent stalling 2015-09-07 15:52:54 +02:00
Dominik Sander
08028aa9ea Revert TwitterStreamAgent back to the twitter-stream gem 2015-09-07 15:52:48 +02:00
Dominik Sander
991777ca29 Background worker load Dotenv if APP_SECRET_TOKEN is not set
Unified background worker initialization into pre_runner_boot

Set Rails.configuration.cache_classes = true for background workers
2015-08-23 16:58:44 +02:00
Dominik Sander
0f64e4e6b9 Add AgentRunner and LongRunnable to support long running agents
This changes HuginnScheduler and TwitterStream to use the new
AgentRunner

TwitterStream now uses the `twitter` gem to access the streaming API
which removes the `eventmachine` dependency (in production).

Make JabberAgent FormConfigurable and LongRunnable

Expect LongRunnable::Worker#run to be overridden
2015-08-23 16:58:44 +02:00
Ian Blenke
7b952fa8ac Merged master to re-converge after merge of PR #983 2015-08-21 21:57:12 -04:00
Dominik Sander
1361c5693b Moved deployment configuration files and changed the repository
Update ruby to 2.2.3
2015-08-19 22:12:50 +02:00
Dominik Sander
cec02c1bd6 Use runit instead of upstart as init system 2015-08-19 00:13:45 +02:00
Dominik Sander
0e4c9be723 First version of the manual deployment guide 2015-08-19 00:13:44 +02:00
Ian Blenke
16b4c6cad1 Fixed conflict in docker/scripts/init with bind to 0.0.0.0 2015-08-16 20:58:56 -04:00
Akinori MUSHA
51e8414257 Retire lib/prepend.rb
We do not support Ruby 1.9 any more.
2015-08-01 16:55:04 +09:00
Akinori MUSHA
8c2b11a567 Make TupleSorter#sort! never fail
If comparison of two objects fails, they are compared as strings.
2015-07-30 02:06:56 +09:00
Akinori MUSHA
06554eaf31 Implement Utils.sort_tuples! for user-defined sorting 2015-07-22 23:13:40 +09:00
Akinori MUSHA
66e72eff75 Do not activate the emulation of Module#prepend on ruby 2.0 2015-07-22 23:13:14 +09:00
Andrew Cantino
9ad49fbf03 Merge remote-tracking branch 'origin/master' into scenarios_contain_control_links 2015-07-20 20:50:38 -07:00