FFE: merge rsyslog 8.9.0-3 from debian

Bug #1488830 reported by Louis Bouchard
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rsyslog (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

Dear Release Team,

The request for a sponsor for the merge of rsyslog (LP: #1464201) has not received attention prior to feature freeze. Given the fact that Wily is the last release before LTS, it would be appropriate to have at least one release with rsyslog 8.9.0-3 before the next LTS. Another risk is to end up with a version of rsyslog in the next LTS that will no longer be supported upstream.

The new version builds and run on Wily.

The v8 release of rsyslog has been in use in Debian for a while now and Jessie is running 8.4.2. It incorporates numerous bugfixes for the v7 stream as well as for new v8 functionalities.

Here is a detailed list of changes applied to rsyslog between 7.4.4 and 8.9 :

------------------------------------------------------------------------------
Version 8.9.0 [v8-stable] 2015-04-07
- omprog: add option "hup.forward" to forwards HUP to external plugins
- imuxsock: added capability to use regular parser chain
- 0mq: improvements in input and output modules
- imtcp: add support for ip based bind for imtcp -> param "address"
- build system: added option --without-valgrind-testbench
------------------------------------------------------------------------------
Version 8.8.0 [v8-stable] 2015-02-24
- omkafka: add support for dynamic topics and auto partitioning
- imtcp/imptcp: support for broken Cisco ASA TCP syslog framing
- omfwd: more detailled error messages in case of UDP send error
- TLS syslog: enable capability to turn on GnuTLS debug logging
- improve rsyslogd -v output and error message with meta information
------------------------------------------------------------------------------
Version 8.7.0 [v8-stable] 2015-01-13
- add message metadata "system" to msg object
- imfile: add support for "filename" metadata
- imptcp: make stats counter names consistent with what imudp, imtcp uses
- added new module "omkafka" to support writing to Apache Kafka
- omfwd: add new "udp.senddelay" parameter
- mmnormalize enhancements
- RainerScript "foreach" iterator and array reading support
- now requires liblognorm >= 1.0.2
- add support for systemd >= 209 library names
- BSD "ntp" facility (value 12) is now also supported in filter
------------------------------------------------------------------------------
Version 8.6.0 [v8-stable] 2014-12-02
- configuration-setting rsyslogd command line options deprecated
- new and enhanced plugins for 0mq. These are currently experimantal.
- empty rulesets have been permitted. They no longer raise a syntax error.
- add parameter -N3 to enable config check of partial config file
- rsyslogd -e option has finally been removed
- testbench improvements
- testbench is now by default disabled
- add new RainerScript functions warp() and replace()
- mmnormalize can now also work on a variable
- new property date options for day ordinal and week number
- remove --enable-zlib configure option, we always require it
- slight source-tree restructuring: contributed modules are now in their
- imttcp was removed because it was an incompleted experimental module
- pmrfc3164sd because it was a custom module nobody used
- omoracle was removed because it was orphaned and did not build/work
---------------------------------------------------------------------------
Version 8.5.0 [v8-stable] 2014-10-24
- imfile greatly refactored and support for wildcards added
- PRI-handling code refactored for more clarity and robustness
- ommail: add support for RainerScript config system [action() object]
- refactored the auto-backgrounding method
- make gntls tcp syslog driver emit more error messages
- imgssapi: log remote peer address in some error messages
---------------------------------------------------------------------------
Version 8.4.2 [v8-stable] 2014-10-02
- behaviour change: "msg" of messages with invalid PRI set to "rawmsg"
---------------------------------------------------------------------------
Version 8.4.1 [v8-stable] 2014-09-30
- imudp: add for bracketing mode, which makes parsing stats easier
- permit at-sign in variable names
---------------------------------------------------------------------------
Version 8.4.0 [v8-stable] 2014-08-18
- this is the new stable branch, which incorporates all enhancements of
  rsyslog 8.3.
- mmjsonparse: support selectable cookie and target containers
- omprog: emit error message via syslog() if loading binary fails
- couple of patches imported from v7-stable (7.6.4)
- new pmciscoios parser supporting various Cisco IOS formats
- RFC3164 timestamp parser now accepts timezones and subsecond resolution
- new parser config object -- permits to define custom parser definitions
- new tzinfo config object -- permits to define time zone offsets
- unify input object naming
- made the missing (contributed) modules build under v8 [import from 8.2.2]
  Modules:
  * mmrfc5424addhmac
  * omrabbitmq
  * omgssapi
  * omhdfs
  * omzmq3
- added a cleanup process (janitor); permits to close omfile files after a
  timeout
- make omgssapi build under v8.3 [import vom v8.2]
- removed obsolete --disable-fsstnd configure option
---------------------------------------------------------------------------
Version 8.3.2 [v8-devel] 2014-05-02
- new template options for date extraction:
  - year
  - month
  - day
  - wday
  - hour
  - minute
  - second
  - tzoffshour
  - tzoffsmin
  - tzoffsdirection
  - wdayname
- imrelp: added support for per-listener ruleset and inputname
---------------------------------------------------------------------------
Version 8.3.1 [v8-devel] 2014-04-24
- external message modification interface now support modifying message PRI
- "jsonmesg" property will include uuid only if one was previously generated
- added plugin to rewrite message facility and/or severity
- permits to build against json-c 0.12
---------------------------------------------------------------------------
Version 8.3.0 [v8-devel] 2014-04-10
- new plugin for anonymizing credit card numbers
- external message modification modules are now supported
- new $jsonmesg property with JSON representation of whole message object
- improved error message for invalid field extraction in string template
- fix build problems on Solaris
---------------------------------------------------------------------------
Version 8.2.2 [v8-stable] 2014-06-02
- made the missing (contributed) modules build under v8
  * mmrfc5424addhmac
  * omrabbitmq
  * omgssapi
  * omhdfs
  * omzmq3
---------------------------------------------------------------------------
Version 8.2.1 [v8-stable] 2014-04-17
- permits to build against json-c 0.12
- doc is no longer shipped as part of the rsyslog tarball
---------------------------------------------------------------------------
Version 8.2.0 [v8-stable] 2014-04-02
This starts a new stable branch based on 8.1.6 plus the following changes:
- we now use doc from the rsyslog-doc project
- add support for specifying the liblogging-stdlog channel spec
- add "defaultnetstreamdrivercertfile" global variable to set a default
- omelasticsearch: add new "usehttps" parameter for secured connections
- "action resumed" message now also specifies module type
- add capability to override GnuTLS path in build process
- better and more consistent action naming, action queues now always
---------------------------------------------------------------------------
Version 8.1.6 [release candidate] 2014-02-20
- omfile: permit to set global defaults for action parameters
- add capability to escape control characters in the C way of doing it
- parser global parameters can now be set using RainerScript global()
- omprog: guard program-to-be-executed against CTL-C
- omprog bugfix: parameter "forceSingleInstance" is NOT mandatory
- add new jsonr property replacer option
- added external plugin interface
- ommongodb: add authentication support (untested)
---------------------------------------------------------------------------
Version 8.1.5 [devel] 2014-01-24
- omprog: ability to execute multiple program instances per action
- imfile now supports inotify (but must be explicitely turned on)
- imfile no longer has a limit on number of monitored files
- added ProcessInternalMessages global system parameter
- new dependency: liblogging-stdlog (for submitting to external logger)
---------------------------------------------------------------------------
Version 8.1.4 [devel] 2014-01-10
- add exec_template() RainerScript function
- imrelp: support for TCP KEEPALIVE added
- bumped librelp dependency to 1.2.2 to support new KEEPALIVE feature
- Add directives for numerically specifying GIDs/UIDs
---------------------------------------------------------------------------
Version 8.1.3 [devel] 2013-12-06
- changes to the strgen module interface
- new output module interface for transactional modules
- performance improvements
- failed action detection more precisely for a number of actions
---------------------------------------------------------------------------
Version 8.1.2 [experimental] 2013-11-28
- support for liblognorm1 added - results in performance improvements
- support for jemalloc added via --enable-jemalloc
- queue defaults have changed
---------------------------------------------------------------------------
Version 8.1.1 [experimental] 2013-11-19
- omhiredis: now supports v8 output module interface and works again
- mmaudit: now supports v8 output module interface and work again
---------------------------------------------------------------------------
Version 8.1.0 [experimental] 2013-11-15
- rewritten core engine for higher performance and new features
- module omruleset is no longer enabled by default.
---------------------------------------------------------------------------
Version 7.6.8 [v7.6-stable] 2014-10-??
- remove zpipe (a testing tool) from --enable-diagtools
---------------------------------------------------------------------------
Version 7.6.7 [v7.6-stable] 2014-10-02
- behaviour change: "msg" of messages with invalid PRI set to "rawmsg"
---------------------------------------------------------------------------
Version 7.6.4 [v7.6-stable] 2014-09-12
- add --enable-generate-man-pages configure switch (default: enabled)
- removed obsolete --disable-fsstnd configure option
- permits to build against json-c 0.12
- new omfile default module parameters
  * filecreatemode
  * fileowner
  * fileownernum
  * filegroup
  * filegroupnum
  * dirowner
  * dirownernum
  * dirgroup
  * dirgroupnum
---------------------------------------------------------------------------
Version 7.6.3 [v7.6-stable] 2014-03-27
- add capability to override GnuTLS path in build process
- support for librelp 1.2.5
---------------------------------------------------------------------------
Version 7.6.2 [v7.6-stable] 2014-03-17
- support for librelp 1.2.4
---------------------------------------------------------------------------
Version 7.6.1 [v7.6-stable] 2014-03-13
- added "action.reportSuspension" action parameter
- "action resumed" message now also specifies module type
- better and more consistent action naming, action queues now always
- add support for "tls-less" librelp
- build system improvements
  * autoconf subdir option
  * support for newer json-c packages
- imjournal enhancements:
  * log entries with empty message field are no longer ignored
  * invalid facility and severity values are replaced by defaults
  * new config parameters to set default facility and severity
---------------------------------------------------------------------------
Version 7.6.0 [v7.6-stable] 2014-02-12
- added ProcessInternalMessages global system parameter
- new dependency: liblogging-stdlog (for submitting to external logger)
---------------------------------------------------------------------------
Version 7.5.8 [v7-release candidate] 2014-01-09
- add exec_template() RainerScript function
- add debug.onShutdown and debug.logFile global parameters
- Add directives for numerically specifying GIDs/UIDs
- actions now report if they suspend and resume themselves
- add new impstats action counters:
  * suspended
  * suspended.duration
  * resumed
---------------------------------------------------------------------------
Version 7.5.7 [v7-devel] 2013-11-25
- queue defaults have changed
- worker thread pool handling has been improved
- now emit warning message if om with msg passing mode uses action queue
---------------------------------------------------------------------------
Version 7.5.6 [devel] 2013-10-29
- impstats: add capability to bind to a ruleset
- improved performance of RainerScript variable access
- RainerScript: make use of 64 bit for numbers where available
- slight performance optimization if GCC is used
- removed global variable support
- new module mmsequence, primarily used for action load balancing
- imudp: support for binding to ruleset added
---------------------------------------------------------------------------
Version 7.5.5 [devel] 2013-10-16
- imfile: permit to monitor an unlimited number of files
- imptcp: add "defaultTZ" input parameter
- imudp: support for multiple receiver threads added
- imudp: add "dfltTZ" input config parameter
- mmanon: removed the check for specific "terminator characters" after
  last octet.
- now requires libestr 0.1.7 as early versions had a nasty bug in
---------------------------------------------------------------------------
Version 7.5.4 [devel] 2013-10-07
- mmpstrucdata: new module to parse RFC5424 structured data into json
  message properties
- change main/ruleset queue defaults to be more enterprise-like
- omfwd: new action parameter "maxErrorMessages" added
- omfile: new module parameters to set action defaults added
  * dirCreateMode
  * fileCreateMode
- mmutf8fix: new module to fix invalid UTF-8 sequences
- imuxsock: handle unlimited number of additional listen sockets
- doc: improve usability by linking to relevant web ressources
- imtcp: add streamdriver.name module parameter
---------------------------------------------------------------------------
Version 7.5.3 [devel] 2013-09-11
- imfile: support for escaping LF characters added
- add support for global and local variables
- add main_queue() configuration object to configure main message queue
- imudp: add ability to specify SO_RCVBUF size (rcvbufSize parameter)
- imudp: use inputname for statistics, if configured
- impstats: add process resource usage counters [via getrusage()]
- impstats: add parameter "resetCounters" to report delta values
- librelp 1.2.0 is now required
- make use of new librelp generic error reporting facility
---------------------------------------------------------------------------
Version 7.5.2 [devel] 2013-07-04
- librelp 1.1.4 is now required
- omrelp: use transactional mode to make imrelp emit bulk sends
- omrelp: add "windowSize" parameter to set custom RELP window size
---------------------------------------------------------------------------
Version 7.5.1 [devel] 2013-06-26
- librelp 1.1.3 is required - older versions can lead to a segfault
- add mmfields, which among others supports easy parsing of CEF messages
- omrelp:
  * new parameter "compression.prioritystring" to control encryption
- imrelp:
  * new parameter "compression.dhbits" to control the number of
    bits being used for Diffie-Hellman key generation
  * new parameter "compression.prioritystring" to control encryption
    parameters used by GnuTLS
  * support for impstats added
  * support for setting permitted peers (client authentication) added
- imjournal: imported patches from 7.4.1
- omprog: add support for command line parameters
- added experimental TCP stream compression (imptcp only, currently)
- added BSD-specific syslog facilities
---------------------------------------------------------------------------
Version 7.5.0 [devel] 2013-06-11
- imrelp: implement "ruleset" module parameter
- imrelp/omrelp: add TLS & compression (zip) support
- omrelp: add "rebindInterval" parameter
- add -S command line option to specify IP address to use for RELP client
---------------------------------------------------------------------------
Version 7.4.11 [v7.4-stable] *never released*
- imjournal enhancements:
  * log entries with empty message field are no longer ignored
  * invalid facility and severity values are replaced by defaults
  * new config parameters to set default facility and severity
---------------------------------------------------------------------------
Version 7.4.9 [v7.4-stable] 2014-01-22
- added ProcessInternalMessages global system parameter
- new dependency: liblogging-stdlog (for submitting to external logger)
---------------------------------------------------------------------------
Version 7.4.8 [v7.4-stable] 2014-01-08
- rsgtutil provides better error messages on unfinished signature blocks
---------------------------------------------------------------------------
Version 7.4.7 [v7.4-stable] 2013-12-10
---------------------------------------------------------------------------
Version 7.4.6 [v7.4-stable] 2013-10-31
- now requires libestr 0.1.9 as earlier versions lead to problems with
---------------------------------------------------------------------------
Version 7.4.5 [v7.4-stable] 2013-10-22
- mmanon: removed the check for specific "terminator characters" after
  last octet.
- now requires libestr 0.1.8 as early versions had a nasty bug in
  string comparisons
- omelasticsearch: add failed.httprequests stats counter

Revision history for this message
Louis Bouchard (louis) wrote :

Wily's build log

Revision history for this message
Martin Pitt (pitti) wrote :

FWIW, this seems fine to me still at this stage in wily.

 - How did you test upgrades?
 - Does logging continue to the existing files, or is there any disruption users need to be aware of?
 - Does this change any permissions? (We did have bugs with new major rsyslog versions in the past, where it suddenly wasn't able any more to access existing files).

Thanks!

Changed in rsyslog (Ubuntu):
status: New → Incomplete
assignee: nobody → Louis Bouchard (louis-bouchard)
Revision history for this message
Louis Bouchard (louis) wrote :
Download full text (5.8 KiB)

Hi pitti,

The binary packages were built from the debdiff produced using sbuild. A VM with the latest Wily bits (as of 27/08) was used. The list of files touched by the rsyslog package prior to upgrade was gathered with :

root@WilyS-rsyslogmerge:~# ls -l $(dpkg -L rsyslog ) | awk '{print $1"\t"$3"\t"$4"\t"$9}' > before_upgrade

The the package was upgraded using the produced binary :

root@WilyS-rsyslogmerge:~# dpkg -i rsyslog_8.9.0-3ubuntu1_amd64.deb
(Reading database ... 95422 files and directories currently installed.)
Preparing to unpack rsyslog_8.9.0-3ubuntu1_amd64.deb ...
Unpacking rsyslog (8.9.0-3ubuntu1) over (7.4.4-1ubuntu14) ...
Setting up rsyslog (8.9.0-3ubuntu1) ...
Installing new version of config file /etc/rsyslog.conf ...
The user `syslog' is already a member of `adm'.
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Processing triggers for man-db (2.7.2-1) ...
Processing triggers for ureadahead (0.100.0-19) ...

Then a list of touched files was created again :

root@WilyS-rsyslogmerge:~# ls -l $(dpkg -L rsyslog ) | awk '{print $1"\t"$3"\t"$4"\t"$9}' > after_upgrade

Here is the diff of those two files :

--- before_upgrade 2015-08-27 12:25:47.928386611 +0200
+++ after_upgrade 2015-08-27 12:27:15.302598984 +0200
@@ -6,8 +6,10 @@
 -rw-r--r-- root root /etc/logcheck/ignore.d.server/rsyslog
 -rw-r--r-- root root /etc/logrotate.d/rsyslog
 -rw-r--r-- root root /etc/rsyslog.conf
+-rw-r--r-- root root /lib/systemd/system/rsyslog-dmesg.service
 -rw-r--r-- root root /lib/systemd/system/rsyslog.service
 -rw-r--r-- root root /usr/lib/rsyslog/imfile.so
+-rw-r--r-- root root /usr/lib/rsyslog/imjournal.so
 -rw-r--r-- root root /usr/lib/rsyslog/imklog.so
 -rw-r--r-- root root /usr/lib/rsyslog/imkmsg.so
 -rw-r--r-- root root /usr/lib/rsyslog/immark.so
@@ -25,15 +27,18 @@
 -rw-r--r-- root root /usr/lib/rsyslog/lmtcpsrv.so
 -rw-r--r-- root root /usr/lib/rsyslog/lmzlibw.so
 -rw-r--r-- root root /usr/lib/rsyslog/mmanon.so
+-rw-r--r-- root root /usr/lib/rsyslog/mmexternal.so
 -rw-r--r-- root root /usr/lib/rsyslog/mmjsonparse.so
+-rw-r--r-- root root /usr/lib/rsyslog/mmpstrucdata.so
+-rw-r--r-- root root /usr/lib/rsyslog/mmsequence.so
+-rw-r--r-- root root /usr/lib/rsyslog/mmutf8fix.so
+-rw-r--r-- root root /usr/lib/rsyslog/omjournal.so
 -rw-r--r-- root root /usr/lib/rsyslog/ommail.so
 -rw-r--r-- root root /usr/lib/rsyslog/omprog.so
--rw-r--r-- root root /usr/lib/rsyslog/omruleset.so
 -rw-r--r-- root root /usr/lib/rsyslog/omuxsock.so
 -rw-r--r-- root root /usr/lib/rsyslog/pmaixforwardedfrom.so
 -rw-r--r-- root root /usr/lib/rsyslog/pmcisconames.so
 -rw-r--r-- root root /usr/lib/rsyslog/pmlastmsg.so
--rw-r--r-- root root /usr/lib/rsyslog/pmrfc3164sd.so
 -rw-r--r-- root root /usr/lib/rsyslog/pmsnare.so
 -rw-r--r-- root root /usr/lib/tmpfiles.d/00rsyslog.conf
 -rw-r--r-- root root /usr/lib/tmpfiles.d/xconsole.conf
@@ -415,6 +420,7 @@
 lrwxrwxrwx root root rmnologin.service
 -rw-r--r-- root root rpcbind.target
 -rw-r--r-- root root rsync.service
+-rw-r--r-- root root rsyslog-dmesg.service
 -rw-r--r-- root root rsyslog.service
 lrwxrwxrwx root root runlevel0.target
 lrwxrwxrwx root root runlevel1.target
@@ -633,6 +639,7 @@
 /usr/lib/r...

Read more...

Changed in rsyslog (Ubuntu):
status: Incomplete → New
Revision history for this message
Martin Pitt (pitti) wrote :

Thanks for the detailled testing! This LGTM, +1.

Wrt. the addition to "imjournal.so": It seems this version now supports pulling data from the journal instead of journal pushing into rsyslog? Is that enabled by default? This would be more robust as right now messages often get lost because rsyslog sometimes can't read them fast enough and thus drops some. (This is unrelated to this FFE, but interesting for me as the systemd maintainer).

Changed in rsyslog (Ubuntu):
status: New → Confirmed
Revision history for this message
Louis Bouchard (louis) wrote :

New debdiff for the merge with pitti's requests implemented.

Revision history for this message
Louis Bouchard (louis) wrote :

Sorry for the noise, wrong bug :-/

Louis Bouchard (louis)
Changed in rsyslog (Ubuntu):
assignee: Louis Bouchard (louis-bouchard) → nobody
Changed in rsyslog (Ubuntu):
status: Confirmed → Fix Released
status: Fix Released → Fix Committed
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.