How do you escape a user-provided search term that you don't want evaluated for sed?

I'm trying to escape a user-provided search string that can contain any arbitrary character and give it to sed, but can't figure out how to make it safe for sed to use. In sed, we do s/search/replace/, and I want to search for exactly the characters in the search string without sed interpreting them (e.g., the '/' in 'my/path' would not close the sed expression).

I read this related question concerning how to escape the replace term. I would have thought you'd do the same thing to the search, but apparently not because sed complains.

Here's a sample program that creates a file called "my_searches". Then it reads each line of that file and performs a search and replace using sed.

#!/bin/bash

# The contents of this heredoc will be the lines of our file.
read -d '' SAMPLES << 'EOF'
/usr/include
P@$$W0RD$?
"I didn't", said Jane O'Brien.
`ls -l`
~!@#$%^&*()_+-=:'}{[]/.,`"\|
EOF
echo "$SAMPLES" > my_searches

# Now for each line in the file, do some search and replace
while read line
do
        echo "------===[ BEGIN $line ]===------"

        # Escape every character in $line (e.g., ab/c becomes \a\b\/\c).  I got
        # this solution from the accepted answer in the linked SO question.
        ES=$(echo "$line" | awk '{gsub(".", "\\\\&");print}')

        # Search for the line we read from the file and replace it with
        # the text "replaced"
        sed 's/'"$ES"'/replaced/' < my_searches     # Does not work

        # Search for the text "Jane" and replace it with the line we read.
        sed 's/Jane/'"$ES"'/' < my_searches         # Works

        # Search for the line we read and replace it with itself.
        sed 's/'"$ES"'/'"$ES"'/' < my_searches      # Does not work

        echo "------===[ END ]===------"
        echo
done < my_searches

When you run the program, you get sed: xregcomp: Invalid content of \{\} for the last line of the file when it's used as the 'search' term, but not the 'replace' term. I've marked the lines that give this error with # Does not work above.

------===[ BEGIN ~!@#$%^&*()_+-=:'}{[]/.,`"| ]===------
sed: xregcomp: Invalid content of \{\}
------===[ END ]===------

If you don't escape the characters in $line (i.e., sed 's/'"$line"'/replaced/' < my_searches), you get this error instead because sed tries to interpret various characters:

------===[ BEGIN ~!@#$%^&*()_+-=:'}{[]/.,`"| ]===------
sed: bad format in substitution expression
sed: No previous regexp.
------===[ END ]===------

So how do I escape the search term for sed so that the user can provide any arbitrary text to search for? Or more precisely, what can I replace the ES= line in my code with so that the sed command works for arbitrary text from a file?

I'm using sed because I'm limited to a subset of utilities included in busybox. Although I can use another method (like a C program), it'd be nice to know for sure whether or not there's a solution to th开发者_如何转开发is problem.

---

This is a relatively famous problem—given a string, produce a pattern that matches only that string. It is easier in some languages than others, and sed is one of the annoying ones. My advice would be to avoid sed and to write a custom program in some other language.

• You could write a custom C program, using the standard library function strstr. If this is not fast enough, you could use any of the Boyer-Moore string matchers you can find with Google—they will make search extremely fast (sublinear time).

• You could write this easily enough in Lua:

  local function quote(s) return (s:gsub('%W', '%%%1')) end
  local function replace(first, second, s)
    return (s:gsub(quote(first), second))
  end
  for l in io.lines() do io.write(replace(arg[1], arg[2], l), '\n') end
  

  If not fast enough, speed things up by applying quote to arg[1] only once, and inline frunciton replace.

---

As ghostdog mentioned, awk '{gsub(".", "\\\\&");print}' is incorrect because it escapes out non-special characters. What you really want to do is perhaps something like:

awk 'gsub(/[^[:alpha:]]/, "\\\\&")'

This will escape out non-alpha characters. For some reason I have yet to determine, I still cant replace "I didn't", said Jane O'Brien. even though my code above correctly escapes it to

\"I\ didn\'t\"\,\ said\ Jane\ O\'Brien\.

It's quite odd because this works perfectly fine

$ echo "\"I didn't\", said Jane O'Brien." | sed s/\"I\ didn\'t\"\,\ said\ Jane\ O\'Brien\./replaced/
replaced`

---

this : echo "$line" | awk '{gsub(".", "\\\\&");print}' escapes every character in $line, which is wrong!. do an echo $ES after that and $ES appears to be \/\u\s\r\/\i\n\c\l\u\d\e. Then when you pass to the next sed, (below)

sed 's/'"$ES"'/replaced/' my_searches

, it will not work because there is no line that has pattern \/\u\s\r\/\i\n\c\l\u\d\e. The correct way is something like:

$ sed 's|\([@$#^&*!~+-={}/]\)|\\\1|g' file
\/usr\/include
P\@\$\$W0RD\$?
"I didn't", said Jane O'Brien.
\`ls -l\`
\~\!\@\#\$%\^\&\*()_\+-\=:'\}\{[]\/.,\`"\|

you put all the characters you want escaped inside [], and choose a suitable delimiter for sed that is not in your character class, eg i chose "|". Then use the "g" (global) flag.

tell us what you are actually trying to do, ie an actual problem you are trying to solve.

---

This seems to work for FreeBSD sed:

# using FreeBSD & Mac OS X sed
ES="$(printf "%q" "${line}")"
ES="${ES//+/\\+}"
sed -E s$'\777'"${ES}"$'\777'replaced$'\777' < my_searches
sed -E s$'\777'Jane$'\777'"${line}"$'\777' < my_searches
sed -E s$'\777'"${ES}"$'\777'"${line}"$'\777' < my_searches

---

The -E option of FreeBSD sed is used to turn on extended regular expressions.

The same is available for GNU sed via the -r or --regexp-extended options respectively.

For the differences between basic and extended regular expressions see, for example:

http://www.gnu.org/software/sed/manual/sed.html#Extended-regexps

Maybe you can use FreeBSD-compatible minised instead of GNU sed?

# example using FreeBSD-compatible minised, 
# http://www.exactcode.de/site/open_source/minised/

# escape some punctuation characters with printf
help printf
printf "%s\n" '!"#$%&'"'"'()*+,-./:;<=>?@[\]^_`{|}~'
printf "%q\n" '!"#$%&'"'"'()*+,-./:;<=>?@[\]^_`{|}~'

# example line
line='!"#$%&'"'"'()*+,-./:;<=>?@[\]^_`{|}~  ...  and Jane ...'

# escapes in regular expression
ES="$(printf "%q" "${line}")"        # escape some punctuation characters
ES="${ES//./\\.}"                    # . -> \.
ES="${ES//\\\\(/(}"                  # \( -> (
ES="${ES//\\\\)/)}"                  # \) -> )

# escapes in replacement string
lineEscaped="${line//&/\&}"          # & -> \&   

minised s$'\777'"${ES}"$'\777'REPLACED$'\777' <<< "${line}"
minised s$'\777'Jane$'\777'"${lineEscaped}"$'\777' <<< "${line}"
minised s$'\777'"${ES}"$'\777'"${lineEscaped}"$'\777' <<< "${line}"

---

To avoid potential backslash confusion, we could (or rather should) use a backslash variable like so:

backSlash='\\'
ES="${ES//${backSlash}(/(}"    # \( -> (              
ES="${ES//${backSlash})/)}"    # \) -> )

(By the way using variables in such a way seems like a good approach for tackling parameter expansion issues ...)

---

... or to complete the backslash confusion ...

backSlash='\\'
lineEscaped="${line//${backSlash}/${backSlash}}"   # double backslashes
lineEscaped="${lineEscaped//&/\&}"                 # & -> \&

---

If you have bash, and you're just doing a pattern replacement, just do it natively in bash. The ${parameter/pattern/string} expansion in Bash will work very well for you, since you can just use a variable in place of the "pattern" and replacement "string" and the variable's contents will be safe from word expansion. And it's that word expansion which makes piping to sed such a hassle. :)

It'll be faster than forking a child process and piping to sed anyway. You already know how to do the whole while read line thing, so creatively applying the capabilities in Bash's existing parameter expansion documentation can help you reproduce pretty much anything you can do with sed. Check out the bash man page to start...