By Ash


2009-02-08 11:57:39 8 Comments

What's the simplest way to do a find and replace for a given input string, say abc, and replace with another string, say XYZ in file /tmp/file.txt?

I am writting an app and using IronPython to execute commands through SSH — but I don't know Unix that well and don't know what to look for.

I have heard that Bash, apart from being a command line interface, can be a very powerful scripting language. So, if this is true, I assume you can perform actions like these.

Can I do it with bash, and what's the simplest (one line) script to achieve my goal?

14 comments

@MMParvin 2017-01-14 09:45:22

You can use sed:

sed -i 's/abc/XYZ/gi' /tmp/file.txt

You can use find and sed if you don't know your filename:

find ./ -type f -exec sed -i 's/abc/XYZ/gi' {} \;

Find and replace in all Python files:

find ./ -iname "*.py" -type f -exec sed -i 's/abc/XYZ/gi' {} \;

@gvee 2020-07-08 11:00:51

-i isn't "ignore case", it's -i[SUFFIX], --in-place[=SUFFIX] (edit files in place (makes backup if SUFFIX supplied))

@chrio 2009-06-14 16:37:52

You may also use the ed command to do in-file search and replace:

# delete all lines matching foobar 
ed -s test.txt <<< $'g/foobar/d\nw' 

See more in "Editing files via scripts with ed".

@Peterino 2014-09-12 09:59:57

This solution is independent from GNU/FreeBSD (Mac OSX) incompatibilities (unlike sed -i <pattern> <filename>). Very nice!

@centurian 2013-02-07 14:48:02

I found this thread among others and I agree it contains the most complete answers so I'm adding mine too:

  1. sed and ed are so useful...by hand. Look at this code from @Johnny:

    sed -i -e 's/abc/XYZ/g' /tmp/file.txt
    
  2. When my restriction is to use it in a shell script, no variable can be used inside in place of "abc" or "XYZ". The BashFAQ seems to agree with what I understand at least. So, I can't use:

    x='abc'
    y='XYZ'
    sed -i -e 's/$x/$y/g' /tmp/file.txt
    #or,
    sed -i -e "s/$x/$y/g" /tmp/file.txt
    

    but, what can we do? As, @Johnny said use a while read... but, unfortunately that's not the end of the story. The following worked well with me:

    #edit user's virtual domain
    result=
    #if nullglob is set then, unset it temporarily
    is_nullglob=$( shopt -s | egrep -i '*nullglob' )
    if [[ is_nullglob ]]; then
       shopt -u nullglob
    fi
    while IFS= read -r line; do
       line="${line//'<servername>'/$server}"
       line="${line//'<serveralias>'/$alias}"
       line="${line//'<user>'/$user}"
       line="${line//'<group>'/$group}"
       result="$result""$line"'\n'
    done < $tmp
    echo -e $result > $tmp
    #if nullglob was set then, re-enable it
    if [[ is_nullglob ]]; then
       shopt -s nullglob
    fi
    #move user's virtual domain to Apache 2 domain directory
    ......
    
  3. As one can see if nullglob is set then, it behaves strangely when there is a string containing a * as in:

    <VirtualHost *:80>
     ServerName www.example.com
    

    which becomes

    <VirtualHost ServerName www.example.com
    

    there is no ending angle bracket and Apache2 can't even load.

  4. This kind of parsing should be slower than one-hit search and replace but, as you already saw, there are four variables for four different search patterns working out of one parse cycle.

The most suitable solution I can think of with the given assumptions of the problem.

@slim 2013-03-06 10:51:09

In your (2) -- you can do sed -e "s/$x/$y/", and it will work. Not the double quotes. It can get seriously confusing if the strings in the variables themselves contain characters with special meaning. For example if x="/" or x="\". When you hit these issues, it probably means you should stop trying to use the shell for this job.

@rayro 2013-04-09 15:35:27

I was surprised when I stumbled over this...

There is a replace command which ships with the "mysql-server" package, so if you have installed it try it out:

# replace string abc to XYZ in files
replace "abc" "XYZ" -- file.txt file2.txt file3.txt

# or pipe an echo to replace
echo "abcdef" |replace "abc" "XYZ"

See man replace for more on this.

@Philip Whitehouse 2014-08-28 15:31:04

Two things are possible here: a) replace is a useful independent tool and the MySQL folks should release it separately and depend on it b) replace requires some bit of MySQL o_O Either way, installing mysql-server to get replace would be the wrong thing to do :)

@paul 2016-08-18 10:26:29

only works for mac? in my ubuntu I centos that command does not exist

@Phius 2017-01-11 15:53:30

That's because you don't have mysql-server package installed. As pointed by @rayro, replace is part of it.

@Steven Vachon 2018-01-04 16:52:33

"Warning: replace is deprecated and will be removed in a future version."

@JLB 2018-03-15 18:01:09

As of 2018, it is included in the MariaDB package for Ubuntu.

@neuro_sys 2018-04-11 21:29:02

Not as powerful, but I have a simple "script" here that does the same in case downloading a MySQL or MariaDB is overkill: gist.github.com/neuro-sys/3bf00b6cf28a93e07e44

@Maor 2018-06-03 10:00:40

Be careful not to run the REPLACE command on Windows! On Windows the REPLACE command is for a fast replication of files. Not relevant to this discussion.

@zcourts 2015-12-07 12:48:56

This is an old post but for anyone wanting to use variables as @centurian said the single quotes mean nothing will be expanded.

A simple way to get variables in is to do string concatenation since this is done by juxtaposition in bash the following should work:

sed -i -e "s/$var1/$var2/g" /tmp/file.txt

@Alnitak 2009-02-08 12:10:32

File manipulation isn't normally done by Bash, but by programs invoked by Bash, e.g.:

perl -pi -e 's/abc/XYZ/g' /tmp/file.txt

The -i flag tells it to do an in-place replacement.

See man perlrun for more details, including how to take a backup of the original file.

@slim 2009-02-08 12:44:08

The purist in me says you can't be sure Perl will be available on the system. But that's very seldom the case nowadays. Perhaps I'm showing my age.

@CMCDragonkai 2013-11-05 18:31:06

Can you show a more complex example. Something like replacing "chdir /blah" with "chdir /blah2". I tried perl -pi -e 's/chdir (?:\\/[\\w\\.\\-]+)+/chdir blah/g' text, but I keep getting an error with Having no space between pattern and following word is deprecated at -e line 1. Unmatched ( in regex; marked by <-- HERE in m/(chdir)( )( <-- HERE ?:\\/ at -e line 1.

@Alfonso Santiago 2019-12-04 13:12:14

@CMCDragonkai Check this answer: stackoverflow.com/a/12061491/2730528

@johnny 2009-02-08 12:20:29

The easiest way is to use sed (or perl):

sed -i -e 's/abc/XYZ/g' /tmp/file.txt

Which will invoke sed to do an in-place edit due to the -i option. This can be called from bash.

If you really really want to use just bash, then the following can work:

while read a; do
    echo ${a//abc/XYZ}
done < /tmp/file.txt > /tmp/file.txt.t
mv /tmp/file.txt{.t,}

This loops over each line, doing a substitution, and writing to a temporary file (don't want to clobber the input). The move at the end just moves temporary to the original name.

@slim 2009-02-08 12:39:20

Except that invoking mv is pretty much as 'non Bash' as using sed. I nearly said the same of echo, but it's a shell builtin.

@Panky 2011-11-15 23:45:00

The -i argument for sed doesn't exist for Solaris (and I would think some other implementations) however, so keep that in mind. Just spent several minutes figuring that out...

@checksum 2013-02-20 14:55:56

Note to self: about regular expression of sed: s/..../..../ - Substitute and /g - Global

@Martin L. 2013-02-21 21:26:48

Ps: For me, this variant is 1/3 faster: for a in `cat hosts.txt` ; do echo ${a//abc/XYZ} ; done > hosts.txt.t)

@Austin 2013-02-21 21:36:35

Note for Mac users who get an invalid command code C error... For in-place replacements, BSD sed requires a file extension after the -i flag because it saves a backup file with the given extension. For example: sed -i '.bak' 's/find/replace/' /file.txt You can skip the backup by using an empty string like so: sed -i '' 's/find/replace/' /file.txt

@Boris D. Teoharov 2014-08-06 00:03:43

Tip: If you want case insensitive repalce use s/abc/XYZ/gi

@Royi 2020-05-01 13:38:06

Anyway to disable the RegEx functionality and use them as only strings (Replace only the whole word)?

@kenorb 2015-05-11 20:21:39

To edit text in the file non-interactively, you need in-place text editor such as vim.

Here is simple example how to use it from the command line:

vim -esnc '%s/foo/bar/g|:wq' file.txt

This is equivalent to @slim answer of ex editor which is basically the same thing.

Here are few ex practical examples.

Replacing text foo with bar in the file:

ex -s +%s/foo/bar/ge -cwq file.txt

Removing trailing whitespaces for multiple files:

ex +'bufdo!%s/\s\+$//e' -cxa *.txt

Troubleshooting (when terminal is stuck):

  • Add -V1 param to show verbose messages.
  • Force quit by: -cwq!.

See also:

@vineeshvs 2019-04-20 13:46:31

Wanted to do the replacements interactively. Hence tried "vim -esnc '%s/foo/bar/gc|:wq' file.txt". But the terminal is stuck now. How shall we make the replacements interactively without the bash shell behaving weirdly.

@kenorb 2019-04-26 22:39:30

To debug, add -V1, to force quit, use wq!.

@J Ajay 2016-05-31 23:55:34

Try the following shell command:

find ./  -type f -name "file*.txt" | xargs sed -i -e 's/abc/xyz/g'

@tripleee 2016-10-11 14:08:56

This is an excellent answer to "how do I accidentally all the files in all subdirectories, too" but that does not seem to be what is asked here.

@kenorb 2019-09-26 15:20:16

This syntax won't work for BSD version of sed, use sed -i'' instead.

@johnraff 2017-04-15 02:09:16

If the file you are working on is not so big, and temporarily storing it in a variable is no problem, then you can use Bash string substitution on the whole file at once - there's no need to go over it line by line:

file_contents=$(</tmp/file.txt)
echo "${file_contents//abc/XYZ}" > /tmp/file.txt

The whole file contents will be treated as one long string, including linebreaks.

XYZ can be a variable eg $replacement, and one advantage of not using sed here is that you need not be concerned that the search or replace string might contain the sed pattern delimiter character (usually, but not necessarily, /). A disadvantage is not being able to use regular expressions or any of sed's more sophisticated operations.

@Brian Hannay 2017-09-25 17:37:03

Any tips for using this with tab characters? For some reason my script doesn't find anything with tabs after changing from sed with lots of escaping to this method.

@johnraff 2017-10-27 07:07:16

If you want to put a tab in the string you're replacing, you can do so with Bash's "dollared single quotes" syntax, so a tab is represented by $'\t', and you can do $ echo 'tab'$'\t''separated' > testfile; $ file_contents=$(<testfile); $ echo "${file_contents//$'\t'/TAB}"; tabTABseparated `

@micalith 2016-05-03 18:44:19

You can use python within the bash script too. I didn't have much success with some of the top answers here, and found this to work without the need for loops:

#!/bin/bash
python
filetosearch = '/home/ubuntu/ip_table.txt'
texttoreplace = 'tcp443'
texttoinsert = 'udp1194'

s = open(filetosearch).read()
s = s.replace(texttoreplace, texttoinsert)
f = open(filetosearch, 'w')
f.write(s)
f.close()
quit()

@Linux4you 2015-07-27 19:08:59

Be careful if you replace URLs with "/" character.

An example of how to do it:

sed -i "s%http://domain.com%http://www.domain.com/folder/%g" "test.txt"

Extracted from: http://www.sysadmit.com/2015/07/linux-reemplazar-texto-en-archivos-con-sed.html

@zalex 2015-02-12 08:34:49

You can use rpl command. For example you want to change domain name in whole php project.

rpl -ivRpd -x'.php' 'old.domain.name' 'new.domain.name' ./path_to_your_project_folder/  

This is not clear bash of cause, but it's a very quick and usefull. :)

@slim 2009-02-08 12:13:13

Bash, like other shells, is just a tool for coordinating other commands. Typically you would try to use standard UNIX commands, but you can of course use Bash to invoke anything, including your own compiled programs, other shell scripts, Python and Perl scripts etc.

In this case, there are a couple of ways to do it.

If you want to read a file, and write it to another file, doing search/replace as you go, use sed:

sed 's/abc/XYZ/g' <infile >outfile

If you want to edit the file in place (as if opening the file in an editor, editing it, then saving it) supply instructions to the line editor 'ex'

echo "%s/abc/XYZ/g
w
q
" | ex file

Ex is like vi without the fullscreen mode. You can give it the same commands you would at vi's ':' prompt.

@awattar 2020-07-28 14:02:47

Can edit many files in place using 'ex' editor?

@slim 2020-07-30 09:05:27

@awattar Just do them one at a time in a for loop.

@awattar 2020-07-30 09:07:29

Is there an option to use it together with globbing as a one liner?

@slim 2020-07-30 09:15:59

Not as far as I know. for f in report*.txt; do echo "%s/abc/XYZ/g \n w \n q \n" | ex file; done is clean and simple. Why put functionality into ex that the shell already has?

@slim 2020-07-30 09:16:32

(Or, if your problem outgrows the shell, use Python/Perl/whatever)

@awattar 2020-07-30 09:17:39

Thank you for the comprehensive comments.

Related Questions

Sponsored Content

38 Answered Questions

[SOLVED] How can I check if a program exists from a Bash script?

  • 2009-02-26 21:52:49
  • gregh
  • 735711 View
  • 2319 Score
  • 38 Answer
  • Tags:   bash

29 Answered Questions

[SOLVED] Pipe to/from the clipboard in Bash script

14 Answered Questions

[SOLVED] How do I set a variable to the output of a command in Bash?

7 Answered Questions

37 Answered Questions

[SOLVED] How do I parse command line arguments in Bash?

67 Answered Questions

[SOLVED] How to get the source directory of a Bash script from within the script itself?

  • 2008-09-12 20:39:56
  • Jiaaro
  • 1807243 View
  • 5129 Score
  • 67 Answer
  • Tags:   bash directory

12 Answered Questions

[SOLVED] What is the cleanest way to ssh and run multiple commands in Bash?

  • 2010-12-10 18:50:48
  • Eli
  • 386904 View
  • 359 Score
  • 12 Answer
  • Tags:   bash unix ssh

13 Answered Questions

[SOLVED] Looping through the content of a file in Bash

  • 2009-10-05 17:52:54
  • Peter Mortensen
  • 1731292 View
  • 1458 Score
  • 13 Answer
  • Tags:   linux bash loops unix io

20 Answered Questions

[SOLVED] How do I tell if a regular file does not exist in Bash?

  • 2009-03-12 14:48:43
  • Bill the Lizard
  • 2584622 View
  • 3351 Score
  • 20 Answer
  • Tags:   bash file-io scripting

10 Answered Questions

[SOLVED] How to get a password from a shell script without echoing

  • 2010-10-20 17:33:28
  • BD at Rivenhill
  • 375714 View
  • 446 Score
  • 10 Answer
  • Tags:   bash shell scripting sh

Sponsored Content