$Id$ Feature Request: Bugs: 1. Change code to allow for tailing spaces when looking at the expression field 3/10/2004 - Removing any reference to company name 3/9/2004 - Changed out company logo - Added GPL notice to all libraries 1/21/2004 - Found a bug where a host would email off an alert with no text. I've put a stop gap fix in. 1/13/2004 - Fixed bug with msyslog.pgsql where SET STORAGE didn't have a space before it. - Fixed bug where hour in time of rules would default to 18:00 for no good reason - Fixed two problems with cloning rules: 1) order was not preserved 2) an imported version of the database couldn't clone because fields that were empty needed to have non-null defaults applied - Forgot to merge in latest VPN reports 1/6/2004 - Added 'left menu' support for ACID and MRTG groups in security framework should the ever be added 8) 1/6/2004 - Added 'left menu' support for ACID and MRTG groups in security framework should the ever be added 8) 1/1/2004 - Fixed spelling error in with the word 'threshold' in the rules schema - Fixed a bug in vacuumdb where it was 'ANALYZ' not 'ANALYZE' 12/19/2003 - Updated code to look into /opt/apache 12/10/2003 - Updated database to schema to not use compressed text fields(we'll see how this performs) - working on adding interfaces to more lock data and other new stats with PostgreSQL V7.4 - cleaned up some button descriptions on the maintenance page, also added a lock view as well as a settings view 12/9/2003 - Finished rule.php support for basic timer maintenance. Need to add another page to graft timers onto rules. - processlogs.php is now setup to support rule timers - need to write rule expiration process - started converting away from compressed text in hopes of providing faster data retrieval.... also pulled OIDs from DB definition 12/8/2003 - Adding support for date and time based rules with date ranges, day of week selections, as well as deleted rules, need to add interface to control rule timestamp properties 12/7/2003 - finished adding accumulation thresholds 12/3/2003 - adding support for both types of thresholds. Need to update web pages to reflect new radio buttons. - added web configuration support for supression thresholds and accumulating thresholds, now onto updating the log processor - basic supression works(kinda), need to verify functionality 12/2/2003 - customer profiles can now have multiple hosts added at once. - fixed a stupid bug where didn't behave right. next would stop working if you started at oct-26-2003 and it would stay on oct-26-2003 - Added web-based framework and database schema to support alert supression thresholds 12/1/2003 - Updated processlogs.php code to better deal with single entry/no rules vs mutliple rules 11/19/2003 - Updated mail table to enforce unique login ids(effectively stopping two processes from running at the same time, one will crash and die(safely)) - Updated openmail and closeopenmail to use transaction support since PostgreSQL no longer does server-side auto-commit(ie. convert everything to transactions) - Transaction support should now be officially added, will do some testing 11/18/2003 - Updating program for support with PostGreSQL V7.4 - Fixed host process table to reflect the fact that multiple hosts are in there by default 10/13/2003 - Added alert total to bottom of alert aggregation 10/6/2003 - finished support for alert aggregation - updated 1stview to pull the current time and date - fixed host.php bug where you could expire syslogs but not be forced to expire syslogs 10/5/2003 - still adding support for alert aggregation. Basic aggregation works plus alert zooming but need to add support for across the board for other alert queries 10/3/2003 - added ability to un/suspend log processors from the web interface - updated maintenance to rebuild all indexes in an better manner(ie. grab the index list from the DB rather then by hand) - added additional framework to do alert aggregation interface. Need to add 'aggregation code' for display 9/24/2003 - weeklyindexrebuild.php now pulls all indexes from the system and rebuilds them. The result is that the system will now rebuild any new indexes without manual reconfiguration. 9/15/2003 - launchid was not initiated correctly in the clonerule. 9/2/2003 - Fixed a bug where '\'s at the end of a line caused problems because we were not properly dealing with them in general. Fixed that. 8/23/2003 - Created another bug when fixing 5000 line paging. Timestamp was thrown off in view.php 8/18/2003 - Syncing changes from production smt environment: vacuumtsyslog.php - Updated weeklyindexrebuild.php to account for the correct indexes - Updated maint.php to account for the three new indexes for the launch program section 8/13/2003 - view.php has had several updates. Paging should now be fixed. Multiple searches appeared not to be working correctly. - Needed to add lastid as hidden var if the variable was set - Needed to use urlencode on top of htmlspecialcharacters, filters were broken because of it 8/11/2003 - 1stfilter.php doesn't list 'global' filters that you down own - modified view.php to not let the user save a filter with no description - added support to delete all of a user's filters(ie. do before delete) - another problem popped up with filter.php when I added the delete user filter option 8/1/2003 - processlogs.php now supports launching external programs! 7/31/2003 - Updated vacuumdb.php to do a full vacuum of the TSyslog table. Why? Because the system doesn't reclaim disk space or use old delete space for some reason - Almost finished adding launch program code, need to test. 7/27/2003 - Added weeklyindexrebuild.php which rebuilds all indexes at 5am Sunday morning 7/23/2003 - Continue the programming of the 'launch' ability into the system. Will need to touch code for clearing stale processors - Adding another maintenance option for viewing the log volume breakout of every host in the Syslog_TArchive table - Just shoot me: I have added reindexing support to the maintenance page. I have also updated vacuumdb to reindex before the vacuum - Updating maintenance displays to show what the object types are, views, tables, etc.. - Can now reindex the all of the SMT-related/created indexes from the maintenance page 7/22/2003 - Updating software to include a basic maintenance page - Create script to do 'vacuum analyze TSyslog', the system will attempt to vacuum every hour - Added maintenance section to allow for web-based manual db vacuum - Cleaned up maintance page to do 'analyze'. - Adjusted 'hourly' script to analyze, not vacuum 7/21/2003 - Updated processor.php to allow clearing of stale processors via the web browser - Updated processlogs.php to update processed ID's via the same delete transaction - processlogs.php no longer can clear stale processes, it now issues alerts in the event the system is taking longer than an hour between runs - 1stequiptype.php did not properly exit if user did not have permissions 8( - Found a bug in the BottomQuery portion of the distinction section for view.php. It was requesting entries from TSyslog, not tarchive. 8( - Basic launch administration is finished. Need to extend rules to support launching. - Updated rule.php to allow for the launch field. - Fixed a problem where using premade rules only pulled the description + expression. Updated to pull severity, facility, rule-or-level, and launchid 7/20/2003 - Found BIG BUG with how the system pulls syslogs. It turns out that some systems are able to force SMT to think it is learning data @ 1/1/2003. In any case, the system is inserting records but it is not accounting for them. It was alerting but not deleting them. 8( I fixed it. - I also fixed how the system calculates timeframes. - Added new index to TSyslog for host & TSyslog_ID to hopefully allow for faster searching 7/16/2003 - processlogs.php is more vocal about cleanup - changed page access so the system checks to see if the client connection is coming on a port < 443, if so then error - there was a bug with view.php asking for BottomTopQuery instead of BottomQuery. Fixed 7/9/2003 - processlogs.php wasn't queueing to 64K before migrating logs over. The system now dumps out debug output for every 64K block 7/8/2003 - alert.php now adjusted to join both tables - Found an issue with hosts.php where deleting a host deleted syslogs but not alerts related to those logs. Fixed that problem. 8) - Processlogs.php is alsmost finished. 8) - processlogs.php is done. Time to load another build onto dangermen.com! - Fixed expirelogs.php to expire off of the archive table, nightlyroguecheck also checks both tables 7/7/2003 - Will be working to have TSyslog archive logs to a different table after processing. The result should be a giant speed up! Starting after 0.212 - Created an archive table. view.php now pulls from the archive table & current table - alert.php needs to be adjusted. - processlogs.php needs to push from one table to another - Initial results are very positive 6/13/2003 - Found a but where host.php doesn't delete a processor association for a host that has been deleted. 8( 3/31/2003 - Fixed a bug in processlogs.php where it was submitting emails w/ subject using $host instead of $loghost 3/20/2003 - Finished adding 'per host' rate alerting - Cleaned up rate-warning emails include the hostname in the subject line of the email - Updated processor.php to only list those hosts where that have not been assigned. 8) - We don't just make the syslog product you buy, we make the syslog product you buy better! 3/19/2003 - Found a bug in view.php where saving filters was not saving 'facility & severity' rules - Update to pgsql.msyslog table to re-include premade hosts for SMT - Modifed customer.php to allow setting 'edit' attribute on a per-host basis - Added support for users to edit rules assuming they have 'permission' to do so. 8) - Broke user cloning, forgot to adjust for destination user as well as new attributes, all fixed - Added individual host log rate warnings, added per host rate warnings to host.php, need to do processlogs.php 2/21/2003 - Updated view.php as it was not having difficulties marking lines in red when multiple matches would be happening 2/3/2003 - Finished basic function comments in pix.php, should probably rename the library 1/27/2003 - Fixed a problem with numberofmonth where it was not going up to December. 1/14/2003 - Updated processlogs to be a little more carefull about 'divide by zero' errors when calculating speed numbers - Updated vacuumdb script to vacuumdb the securityframework instance as well as SMT. 1/13/2003 - Included default host 'localhost' with one rule that responds to root@localhost 1/12/2003 - Made sure smt will work with mod_auth_pgsql 12/4/2002 - addmail function was missing a appostrophe protection for SQL insertion - Took out a debug message in the clonedenial rules section 11/26/2002 - Removed dropdenials as I already had dropdenial. dropdenails was referenced in rule.php - Adding lots of comments, need to finish this task 10/23/2002 - vacuumdb now does the vacuum inside PHP as cleanpgsqlnightly isn't working quite right. 10/1/2002 - emails issued by processlogs now append the name of the box for which the alert belongs - the alert page now has a 'refresh' option - discovered another bug in alert.php where viewing alerts by host doesn't work anymore 8( - making alerts available to customers, that was the problem. - alerts should now be viewable by users 9/29/2002 - Pulled some debugging code - Made more premade rule adjustments 9/23/2002 - still working on the reporting engine 9/20/2002 - expire.php, archive.php, nightlyroguecheck.php, processlogs.php all use php-cli mode 8) - working on reports to breakdown data procesing into smaller chunks 9/2/2002 - Finished first report: cisco-pix-bandwidthbreakdown.php - Updated nightlyroguecheck.php to check logs from the last day to now 9/1/2002 - More work on the reporting framework 8/31/2002 - Begin adding support for pix utilization reports 8/30/2002 - Updated database indexes to have cencatenated index for TSyslog on host,date, & time - Updated the customer view so that the filter type wasn't a text box but hidden as it should be. 8/28/2002 - Fixed yet another bug with the customer view where hostdropdown where logincanseehost as we were passing it host instead of hostid 8/28/2002 - Fixed a bug with SMT w/ view.php and filters using facility & severity, the code even mentioned it was broken 8/27/2002 - Missing a bunch of indexes on alerts & syslogs, we want indexs for time and date 8/26/2002 - Updated processlogs to provide more details about time frames 8/24/2002 - Fixed a problem with the premade rules not correctly saving the rule type. 8/23/2002 - Fixed a problem with using facility & severity and not matching rules correctly in both view.php & processlogs.php - Added hostname as part of subject line in SMT report - Found more problems with facility & severity with view, appears processlogs.php is also flawed - Okay, so major fixes were made to processlogs.php and to view to finish up proper support for facility and severity 8/22/2002 - Took out an 'Expression:' debug statement - Did some adjustment to the time stamping of 'processlogs.php' 8/13/2002 - Added support for 's and \'s in the filtering code - Premade rules now supports 's and \'s. Also fixed new problems with rules page. Filters appears good as well - Started updating premade hosts for cloning - Pixes, LocalDirectors, CatOS Switches, and IOS Routers are now ready for cloning - IOS Switches and VPN devices remain 8/12/2002 - Took out all of the premade rules from the Syslog_TPremade as they were overkill and unnecessary 8/10/2002 - Updated rule.php & processlogs.php to correctly support \'s & "'"s 8/8/2002 - nightlyroguecheck had a few bugs, fixed - view.php was missing an AND for viewing syslogs for hosts assigned to a customer 8/5/2002 - Added pagma no-cache and 300 second refresh to alert.php - Displays time & date of last syslog message when query generates logs > 5000 alerts, provides info in relation to query timeframe 7/28/2002 - Added the ability to view the next 5000 lines should someone want to. 7/26/2002 - added the ability to administer equipment types - Fixed bug with emails where there wasn't an \r issues with each \n - Fixed bug where emails contained HTML color codes - ViewSaves would enable after anyone saved a syslog entry. Now it only enables after the logged in user saves something 7/24/2002 - There was an issue with filterid not being set correctly so filter.php could not properly tell between an add and a modfiy - Added navigation buttons to alerts page - Cloning of rules only appears if there is more than one host - Delete page slimmed down to only allow optional deletion of syslog messages - Saved results page displays error if there are no saved syslogs in the savedata table - Changed version number to V0.99.20B - SecurityFramework while a separate package has been sufficiently integrated into SMT 7/23/2002 - Filters are broken in that setting filters to facility & severity only 'includes' regardless of setting - Fixed problem with filters, they were 'half implemented' 7/21/2002 - All users of the appropriate security level will see the saved syslog option - Had to change Filter Type: Rule, etc... 7/15/2002 - Changed 'Rule Type: Rule, Log Level, and Both' to 'expression, facility + severity, and expression, facility & severity" - Added scripts directory w/ expire, processlogs, and a /tmp debug tool - Fixed renaming so that only syslogs may be renamed. 8) - Added nightlyroguecheck script to call the nightlyroguecheck.php script(checks for hosts who log but aren't defined) - I though "Multiple filter expressions appear to be broken when viewing syslogs", I was wrong. - Fixed 'color' problem with alert.php 7/10/2002 - Adjusted pgsql.msyslog so we do not use 'char' but 'varchar' 5/29/2002 - Still working on processlogs.php to update processed ids for those hosts w/ no rules - View.php line 321 appears to have issues 5/28/2002 - began work on processlogs.php to cover those hosts who are assigned to a processor but have no rules assigned. - customer.php and processor.php now check for duplicates/single assignments as appropriate 5/12/2002 - processlogs.php is finished(in terms of configuration) 5/11/2002 - Started working on processlogs.php 5/10/2002 - expire.php was only written to support a single expiration time and not a time per host. 5/9/2002 - view.php updated to start supporting thost_id 5/8/2002 - 1sthost.php and hosts.php should be converted to support thost_id - 1stcustomer.php and customer.php should be converted to support thost_id - 1stprocessor.php and processor.php should be converted to support thost_id 5/7/2002 - Started working on converting the system from using _host as a key to THost_ID 4/11/2002 - Fixed 1stcustomer.php as the form did not 'close' for either form - Fixed alert.php color coding - I had to install 'distinct on' in the SQL log selection as some log entries appeared more than once. 4/4/2002 - Added code to fix duplicate entries in emails - Changed version to V0.99.01B 3/18/2002 - Log data is color coded - A new version of processlogs.php is out w/ debug msgs in it. Working good on Harley. - Took debug out of 'saving syslogs'. - Save Syslogs now supports using "'" 8( Much work left to do w/ 's - Can now view data by 'user and host type' - Started work on deleting hosts from the system and accounting for host rules - adding a host no longer shows the 'renaming fields' 2/24/2002 - Hosts menu allows synchronizing other tables when renaming hosts - Fixed paging 2/22/2002 - Denial chains are complete. 8) 2/19/2002 - working on processlogsnew.php which cache's host rules & denial rules at the beginning to minimize DB access 2/18/2002 - Updated pages to announce how long they took to process - Clone rules broken, sequence not working - Fixed cloned rules as they were calling for the premade sequence number not the rule sequence number 2/17/2002 - Final support included for priority & severity - Created archive, supports dumping data to std out for bzip2 8) - Denial chain support added to system, processlogs.php all that remains 2/16/2002 - View, Alerts, and View Saves all use colors to convey severity - Filters support severity 2/15/2002 - Changed named to Syslog Management Tool(for now) - Viewer now supports filters using facility and severity 2/13/2002 - Updated view to look like a Berbee product. 8) 2/12/2002 - Made some progress on using filters w/ facilty & severity. Very buggy 2/11/2002 - View logs produces repeates... think unnecessary Syslog_TRules invovled. - Process logs was a bit messaged up, the old delvierymessage variable instead of deliverymessage - Per host/per person email now works - Added code to msyslog to support writing facility & severity to the log messages - Working on scheme where rules & filters can be filter/rules,filter/rules & log levels, or just log levels 2/10/2002 - Can now clone customer accounts - Added stale processor auto-cleaning code so the system will clean up 'old processors' after 30 minutes - System sends an alert email if the system recieves some 3000+ log entries in a given sample. - Nightly system issues emails notifying for hosts who are logging to the system but are not defined as hosts in the system 2/9/2002 - Updated code to use PGSQL V7.2 8) Can you say bigserial, no table lock vacuum, and much more? 8) - Looking into using the transaction interface. - host properties isn't properly keeping the alert log expiration time <= syslog expiration time - View host had a issue with 'view data from last five minutes' - View Saves had the group context wrong, denying access to the page if the group >= 2(ie noc or better) - Filter administration is should be finished