Gergely Brautigam
073ae10332
Removing obsolete test and environment property.
2017-05-29 20:48:50 +02:00
Dominik Sander
e1f702adf1
Upgrade liquid ( #1982 )
...
* Update Liquid to 4.0
* Handle Liquid renaming before_method to liquid_method_missing
* Adopt `regex_replace` tags to new Liquid BlockBody class
2017-04-26 22:09:21 +02:00
Dominik Sander
d3cbd14dd5
Simplify cronlines
2016-12-29 21:52:25 +01:00
Dominik Sander
1f4d54a112
Run every_X schedules on fixed times
...
Makes the schedules more predictable by running them on fixes hours/minutes instead of X seconds after startup.
Changed the "every 7d" schedule to "every monday" night to make it clearer when it runs because every 7 days is ambiguous
without a explicit start date.
Fixes #1103
Possibly related to #1840
Example runtimes:
```
every_1m (*/1 * * * *)
2016-12-29 11:28:00 +0100
2016-12-29 11:29:00 +0100
2016-12-29 11:30:00 +0100
2016-12-29 11:31:00 +0100
2016-12-29 11:32:00 +0100
2016-12-29 11:33:00 +0100
2016-12-29 11:34:00 +0100
2016-12-29 11:35:00 +0100
2016-12-29 11:36:00 +0100
2016-12-29 11:37:00 +0100
every_2m (*/2 * * * *)
2016-12-29 11:28:00 +0100
2016-12-29 11:30:00 +0100
2016-12-29 11:32:00 +0100
2016-12-29 11:34:00 +0100
2016-12-29 11:36:00 +0100
2016-12-29 11:38:00 +0100
2016-12-29 11:40:00 +0100
2016-12-29 11:42:00 +0100
2016-12-29 11:44:00 +0100
2016-12-29 11:46:00 +0100
every_5m (*/5 * * * *)
2016-12-29 11:30:00 +0100
2016-12-29 11:35:00 +0100
2016-12-29 11:40:00 +0100
2016-12-29 11:45:00 +0100
2016-12-29 11:50:00 +0100
2016-12-29 11:55:00 +0100
2016-12-29 12:00:00 +0100
2016-12-29 12:05:00 +0100
2016-12-29 12:10:00 +0100
2016-12-29 12:15:00 +0100
every_10m (*/10 * * * *)
2016-12-29 11:30:00 +0100
2016-12-29 11:40:00 +0100
2016-12-29 11:50:00 +0100
2016-12-29 12:00:00 +0100
2016-12-29 12:10:00 +0100
2016-12-29 12:20:00 +0100
2016-12-29 12:30:00 +0100
2016-12-29 12:40:00 +0100
2016-12-29 12:50:00 +0100
2016-12-29 13:00:00 +0100
every_30m (*/30 * * * *)
2016-12-29 11:30:00 +0100
2016-12-29 12:00:00 +0100
2016-12-29 12:30:00 +0100
2016-12-29 13:00:00 +0100
2016-12-29 13:30:00 +0100
2016-12-29 14:00:00 +0100
2016-12-29 14:30:00 +0100
2016-12-29 15:00:00 +0100
2016-12-29 15:30:00 +0100
2016-12-29 16:00:00 +0100
every_1h (0 */1 * * *)
2016-12-29 12:00:00 +0100
2016-12-29 13:00:00 +0100
2016-12-29 14:00:00 +0100
2016-12-29 15:00:00 +0100
2016-12-29 16:00:00 +0100
2016-12-29 17:00:00 +0100
2016-12-29 18:00:00 +0100
2016-12-29 19:00:00 +0100
2016-12-29 20:00:00 +0100
2016-12-29 21:00:00 +0100
every_2h (0 */2 * * *)
2016-12-29 12:00:00 +0100
2016-12-29 14:00:00 +0100
2016-12-29 16:00:00 +0100
2016-12-29 18:00:00 +0100
2016-12-29 20:00:00 +0100
2016-12-29 22:00:00 +0100
2016-12-30 00:00:00 +0100
2016-12-30 02:00:00 +0100
2016-12-30 04:00:00 +0100
2016-12-30 06:00:00 +0100
every_5h (0 */5 * * *)
2016-12-29 15:00:00 +0100
2016-12-29 20:00:00 +0100
2016-12-30 00:00:00 +0100
2016-12-30 05:00:00 +0100
2016-12-30 10:00:00 +0100
2016-12-30 15:00:00 +0100
2016-12-30 20:00:00 +0100
2016-12-31 00:00:00 +0100
2016-12-31 05:00:00 +0100
2016-12-31 10:00:00 +0100
every_12h (0 */12 * * *)
2016-12-29 12:00:00 +0100
2016-12-30 00:00:00 +0100
2016-12-30 12:00:00 +0100
2016-12-31 00:00:00 +0100
2016-12-31 12:00:00 +0100
2017-01-01 00:00:00 +0100
2017-01-01 12:00:00 +0100
2017-01-02 00:00:00 +0100
2017-01-02 12:00:00 +0100
2017-01-03 00:00:00 +0100
every_1d (0 0 */1 * *)
2016-12-30 00:00:00 +0100
2016-12-31 00:00:00 +0100
2017-01-01 00:00:00 +0100
2017-01-02 00:00:00 +0100
2017-01-03 00:00:00 +0100
2017-01-04 00:00:00 +0100
2017-01-05 00:00:00 +0100
2017-01-06 00:00:00 +0100
2017-01-07 00:00:00 +0100
2017-01-08 00:00:00 +0100
every_2d (0 0 */2 * *)
2016-12-31 00:00:00 +0100
2017-01-01 00:00:00 +0100
2017-01-03 00:00:00 +0100
2017-01-05 00:00:00 +0100
2017-01-07 00:00:00 +0100
2017-01-09 00:00:00 +0100
2017-01-11 00:00:00 +0100
2017-01-13 00:00:00 +0100
2017-01-15 00:00:00 +0100
2017-01-17 00:00:00 +0100
every_7d (0 0 * * 1)
2017-01-02 00:00:00 +0100
2017-01-09 00:00:00 +0100
2017-01-16 00:00:00 +0100
2017-01-23 00:00:00 +0100
2017-01-30 00:00:00 +0100
2017-02-06 00:00:00 +0100
2017-02-13 00:00:00 +0100
2017-02-20 00:00:00 +0100
2017-02-27 00:00:00 +0100
2017-03-06 00:00:00 +0100
```
2016-12-29 21:52:25 +01:00
Akinori MUSHA
3612ba7333
Add podcast tags to events emitted by RssAgent
...
The keys are only added when the feed is a podcast.
2016-11-29 23:59:32 +09:00
Akinori MUSHA
58fabb885c
Add a new Liquid filter rebase_hrefs
2016-10-29 20:40:52 +09:00
Akinori MUSHA
4d10132709
Fix a bug where an empty <link>
is wrongly parsed
...
Due to a problem in sax-machine's internals, an empty `<link/>` in RSS
would be parsed to JSON as `{ "href": "no_buffer" }`. Now empty
`<link/>` elements in RSS and ATOM are simply ignored just like other
collection elements like `<category>`.
2016-10-27 09:17:46 +09:00
Akinori MUSHA
5f5e246552
Use Struct#each_pair
2016-10-27 08:13:43 +09:00
Akinori MUSHA
e5c938aa85
Exclude empty entries from authors
2016-10-27 08:12:48 +09:00
Akinori MUSHA
852f39d480
Rescue error from Mail::Address#name and #address
...
`Mail::Address.new('')` does not raise any error but calling `name` on
the created instance does.
2016-10-27 08:11:10 +09:00
Akinori MUSHA
ba0d2ee0de
Retire ar_mysql_column_charset
...
Rails 5 has native support for charset/collate.
2016-10-11 22:25:54 +09:00
Akinori MUSHA
df6fb7e5fc
Fix super which is not defined yet here
2016-10-07 19:32:02 +09:00
Akinori MUSHA
d63f714ae9
RssAgent: Migrate from FeedNormalizer to Feedjira
...
FeedNormalizer is no longer maintained, and its Atom support has flaws
in that it throws away what RSS::Parser returns and falls back to using
SimpleRSS which is not capable of handling XML entities, resulting in
getting ususable URLs such as ones including `&`.
Feedjira is highly customizable as it implements parsers for various
feed formats on its own using sax-machine.
The `clean` option is reimplemented using the loofah gem.
`Feedjira::FeedUtilities#sanitize_entries!` is not used because it
tries to sanitize non-HTML properties too.
A new option `include_feed_info` is added, with which turned on feed
information is added to each event payload.
A new key `links` is added, which lists all `link` elements.
A new key `enclosure` is added.
2016-10-06 11:49:34 +09:00
Akinori MUSHA
7e79d576b5
Improve Utils.normalize_uri ( #1719 )
...
* Improve Utils.normalize_uri
Globally replacing generally unsafe characters in a URL would not fix
invalid authorities and paths, so use Addressable::URI to normalize them
when necessary.
This should fix #1701 .
* Remove an unused function
* Fix the test case to make sure an IPv6 address is supported
2016-10-05 14:17:04 +09:00
Karol Bucek
8ca39a5802
minor tweaks ( #1669 )
...
* avoid unnecesary method pollution
* production.rake typo fix
* remove un-used local variable assignment
2016-09-06 13:36:10 -04:00
Dominik Sander
811f9c27d1
Allow specifing Agent gem in the Gemfile
2016-06-20 10:37:33 +02:00
Judy Ngai
1abcd947f5
Uploadicons ( #1427 )
...
* checking in work for glyph upload
* checking in icon selections
* added more icon options
* work on upload icon
* fix issue
* switched to options for select
* removed wrapper and removed commment
* add migration to support icon
* remove right floating
* rm params session add it to model
* rebuilt the menu
* rm extra div
* converted js to coffee
* quick change in indentation
* using the view helper
* submitting icons and loading into form
* added icon to both scenario export and import
* added icon to spec
* low level caching the icons
* tentative work in making a new coffee class
* renamed js page
* switched to cache in memory and worked on coffee class
* made pathsmatching compatible for all scenarios
* switched to rails root
* fixed all rubocop stylings
* fixed line is too long styling
* rm private method definition
* patch to avoid having no icon
* blank or nil?
* reordering the lines
2016-06-15 23:52:50 -07:00
Pablo Carranza
74a3c88608
Add time tracking to http status agent ( #1517 )
...
* Add time tracking to http status agent
* Replace time tracking to monotonic
* Add tests for time tracker
2016-05-29 19:13:40 -04:00
Dominik Sander
a6a06c1cd1
Separate feature and non feature spec runs on travis
...
Add database_cleaner gem
Set up coveralls parralel builds in travisci
Require capybara_helper in rails_helper to make clear it affects all tests
2016-05-12 10:03:23 +02:00
Dominik Sander
73bcd927e3
Do not run/enqueue event propagation when a AgentPropagateJob is already enqueued
...
Fixes the creation of duplicated events caused by Agent.receive! running multiple times concurrently.
2016-04-17 17:15:50 +02:00
Brian Petro
b941f740e5
replace delayed_job queue methods with ActiveJob counterparts
2016-04-12 09:49:49 -04:00
Dominik Sander
38f1044078
Add LocalFileAgent and introduce concept to handle files
...
The idea is to have source agents (`LocalFile` and `S3Agent` for now), which just emit a "file_pointer" and consuming
agents (`ReadFile` and `CsvAgent`) which get access to an `IO` object to actually access the file contents. This way we
do not have to pass megabytes of data through the database and the consuming agents can work with relatively unlimited
file sizes.
2016-03-18 20:43:28 +01:00
Dominik Sander
ee1ebea830
Allow to configure various devise settings in .env
2016-03-17 20:47:45 +01:00
Andrew Cantino
67fa70553d
Allow a key with credential instead of a key_file
2016-02-01 20:29:40 -08:00
Akinori MUSHA
96b756ca11
Default ENV['RAILS_ENV'] to 'development'
2016-01-05 16:23:45 +09:00
Dominik Sander
daf45f7f0e
Do not try to load .env file on heroku
2015-12-10 20:54:11 +01:00
eric
d233946feb
Export Agents in GUID order so that the result is amenable to version control tracking
2015-12-07 10:30:14 -07:00
Dominik Sander
9c16b0fb51
Allow usage of PostgreSQL without exporting ON_HEROKU
...
Vendorizing the dotenv gem allows access to the `.env` configuration
in the `Gemfile`
Manual installation: Configure before installing gems
2015-12-04 21:31:45 +01:00
Andrew Cantino
9bdbdfa10d
Merge remote-tracking branch 'origin/master' into prevent_extra_threads_in_specs
2015-11-20 08:56:30 -08:00
Andrew Cantino
60d880e8be
remove p statements
2015-11-20 08:19:22 -08:00
Andrew Cantino
ff05700458
Prevent buildup of threads in specs
2015-11-19 21:39:03 -08:00
Akinori MUSHA
bde6a7269d
Merge pull request #1125 from cantino/website_agent_normalize_uri
...
Introduce Utils.normalize_uri and use it in WebsiteAgent
2015-11-15 01:53:18 +09:00
Akinori MUSHA
cdfdc7f3c5
Introduce Utils.normalize_uri and use it in WebsiteAgent
...
This fixes #938 , and the specs are from #958 . (Thanks @irfancharania!)
2015-11-15 01:41:19 +09:00
Dominik Sander
28983a7378
Merge pull request #1142 from dsander/fix-twitter-agent-restart-leak
...
Do check restarting LongRunnable::Worker for liveliness
2015-11-12 17:26:08 +01:00
Dominik Sander
e303239281
Do check restarting LongRunnable::Worker for liveliness
...
Fixes a memory leak caused by starting a new Worker when the old one is still restarting.
2015-11-12 00:07:30 +01:00
Akinori MUSHA
43a6a025a4
Remove SchedulerAgent#check! which name is confusing
2015-11-06 17:11:09 +09:00
oroce
4487a9270d
force-stopping services
2015-10-21 07:30:25 +02:00
Andrew Cantino
797e7e3b3c
When payloads get truncated in the DB due to encoding issues, at least do not blow up on reading them
2015-09-30 20:28:30 -07:00
Dominik Sander
06b022f161
Merge pull request #819 from dsander/agent-runner
...
Add AgentRunner and LongRunnable to support long running agents
2015-09-09 16:36:56 +02:00
Dominik Sander
e0f14698ba
Ensure database connection are checked back into the pool by wrapping calls in with_connection blocks
2015-09-07 15:52:54 +02:00
Dominik Sander
44ac1ed909
Restart TwitterStreamAgent::Worker every hour to prevent stalling
2015-09-07 15:52:54 +02:00
Dominik Sander
08028aa9ea
Revert TwitterStreamAgent back to the twitter-stream gem
2015-09-07 15:52:48 +02:00
Dominik Sander
991777ca29
Background worker load Dotenv if APP_SECRET_TOKEN is not set
...
Unified background worker initialization into pre_runner_boot
Set Rails.configuration.cache_classes = true for background workers
2015-08-23 16:58:44 +02:00
Dominik Sander
0f64e4e6b9
Add AgentRunner and LongRunnable to support long running agents
...
This changes HuginnScheduler and TwitterStream to use the new
AgentRunner
TwitterStream now uses the `twitter` gem to access the streaming API
which removes the `eventmachine` dependency (in production).
Make JabberAgent FormConfigurable and LongRunnable
Expect LongRunnable::Worker#run to be overridden
2015-08-23 16:58:44 +02:00
Ian Blenke
7b952fa8ac
Merged master to re-converge after merge of PR #983
2015-08-21 21:57:12 -04:00
Dominik Sander
1361c5693b
Moved deployment configuration files and changed the repository
...
Update ruby to 2.2.3
2015-08-19 22:12:50 +02:00
Dominik Sander
cec02c1bd6
Use runit instead of upstart as init system
2015-08-19 00:13:45 +02:00
Dominik Sander
0e4c9be723
First version of the manual deployment guide
2015-08-19 00:13:44 +02:00
Ian Blenke
16b4c6cad1
Fixed conflict in docker/scripts/init with bind to 0.0.0.0
2015-08-16 20:58:56 -04:00
Akinori MUSHA
51e8414257
Retire lib/prepend.rb
...
We do not support Ruby 1.9 any more.
2015-08-01 16:55:04 +09:00