Bug 1125818 - bash 5.0: regression on sed invocation with character class and variable substitution
bash 5.0: regression on sed invocation with character class and variable subs...
Classification: openSUSE
Product: openSUSE Tumbleweed
Classification: openSUSE
Component: Basesystem
Other Other
: P5 - None : Normal (vote)
: ---
Assigned To: Dr. Werner Fink
E-mail List
Depends on:
  Show dependency treegraph
Reported: 2019-02-18 16:28 UTC by Martin Wilck
Modified: 2019-02-19 15:05 UTC (History)
2 users (show)

See Also:
Found By: ---
Services Priority:
Business Priority:
Blocker: ---
Marketing QA Status: ---
IT Deployment: ---


Note You need to log in before you can comment on or make changes to this bug.
Description Martin Wilck 2019-02-18 16:28:26 UTC
OK with bash 4.4:

mwilck@zeus:~> echo $BASH_VERSION
mwilck@zeus:~> var="with\\.backslash"
mwilck@zeus:~> sed -E 's/[[:space:]]'$var'/hi/' /dev/null

Problem with 5.0:

$ var="with\\.backslash"
$ sed -E 's/[[:space:]]'$var'/hi/' /dev/null
sed: -e expression #1, char 36: recursive escaping after \c not allowed

This seems pretty mysterious at first. Why '\c'?
It becomes clearer when using the "-x" option:

$ set -x; sed -E 's/[[:space:]]'$var'/hi/' /dev/null; set +x
+ sed '\s/\[\[\:\s\p\a\c\e\:\]\]with\.backslash/\h\i/' /dev/null
sed: -e expression #1, char 36: recursive escaping after \c not allowed

bash has inserted extra quotes in the '[[:space:]]' expression, converting it to the invalid expression '\s\p\a\c\e'. (see https://lists.gnu.org/archive/html/bug-sed/2017-02/msg00017.html why it's invalid).

This extra quoting doesn't happen with bash 4.4:

mwilck@zeus:~> set -x; sed -E 's/[[:space:]]'$var'/hi/' /dev/null; set +x
+ sed -E 's/[[:space:]]with\.backslash/hi/' /dev/null

The extra quoting also doesn't happen with bash 5.0 if the variable is between double quotes, or if the backslashed string is inserted into the sed invocation directly:

$ sed -E 's/[[:space:]]'"$var"'/hi/' /dev/null
( no error)
$ sed -E 's/[[:space:]]'with\\.backslash'/hi/' /dev/null
( no error)

So a simple workaround would be to use double quotes in shell scripts wherever variable substitution . However, this is a distinct change in behavior wrt previous bash versions, and the behavior of bash 5.0 is inconsistent. Quoting single characters in a (already single-quoted) character class seems wrong anyway.

I encountered this using bpoirier's modified quilt scripts (https://build.opensuse.org/package/show/home:benjamin_poirier:series_sort/quilt-ks)

Ben seems to have noted the same problem, as he inserted quotes in the latest version he committed, a week ago (https://build.opensuse.org/package/rdiff/home:benjamin_poirier:series_sort/quilt-ks?linkrev=base&rev=5).
Comment 1 Dr. Werner Fink 2019-02-19 07:20:52 UTC
Seems to be fixed in 5.0.2(1)-release

 werner@boole:~> var="with\\.backslash"
 werner@boole:~> set -x; sed -E 's/[[:space:]]'$var'/hi/' /dev/null; set +x
 + sed -E 's/[[:space:]]with\.backslash/hi/' /dev/null
 + set +x

Please retest with latest bash 5.0.2

Patch-ID:       bash50-001

Bug-Reported-by:        axel@freakout.de
Bug-Reference-ID:       <201901082050.x08KoShS006731@bongo.freakout.de>
Bug-Reference-URL:      http://lists.gnu.org/archive/html/bug-bash/2019-01/msg00079.html


Under certain circumstances, the glob expansion code did not remove
backslashes escaping characters in directory names (or portions of a
pattern preceding a slash).

Patch-ID:       bash50-002

Bug-Reported-by:        Ante Peric <synthmeat@gmail.com>
Bug-Reference-ID:       <B7E3B567-2467-4F7B-B6B9-CA4E75A9C93F@gmail.com>
Bug-Reference-URL:      http://lists.gnu.org/archive/html/bug-bash/2019-01/msg00095.html


When an alias value ends with an unquoted literal tab (not part of a quoted
string or comment), alias expansion cannot correctly detect the end of the
alias value after expanding it.
Comment 2 Martin Wilck 2019-02-19 14:53:24 UTC
Confirmed - fixed in bash-5.0-2.1 (Factory).

Sorry for not having tested the latest update.
Comment 3 Dr. Werner Fink 2019-02-19 15:05:35 UTC
No problem, I'm happy if tested at all :)