By euforia

2012-07-05 18:57:34 8 Comments

I have a DataFrame with 4 columns of which 2 contain string values. I was wondering if there was a way to select rows based on a partial string match against a particular column?

In other words, a function or lambda function that would do something like, cell_in_question) 

returning a boolean. I am familiar with the syntax of df[df['A'] == "hello world"] but can't seem to find a way to do the same with a partial string match say 'hello'.

Would someone be able to point me in the right direction?


@cardamom 2020-04-29 17:31:18

Should you need to do a case insensitive search for a string in a pandas dataframe column:

df[df['A'].str.contains("hello", case=False)]

@sharon 2014-11-10 17:05:17

I tried the proposed solution above:


and got an error:

ValueError: cannot mask with array containing NA / NaN values

you can transform NA values into False, like this:

df[df["A"].str.contains("Hello|Britain", na=False)]

@joshlk 2015-07-02 14:00:41

Or you can do: df[df['A'].str.contains("Hello|Britain", na=False)]

@Nagabhushan S N 2020-02-05 15:05:04

df[df['A'].astype(str).str.contains("Hello|Britain")] worked as well

@Serhii Kushchenko 2020-02-20 13:06:07

Maybe you want to search for some text in all columns of the Pandas dataframe, and not just in the subset of them. In this case, the following code will help.

df[df.apply(lambda row: row.astype(str).str.contains('String To Find').any(), axis=1)]

Warning. This method is relatively slow, albeit convenient.

@Katu 2019-11-20 13:22:49

Using contains didn't work well for my string with special characters. Find worked though.

df[df['A'].str.find("hello") != -1]

@cs95 2019-03-25 09:58:20

How do I select by partial string from a pandas DataFrame?

This post is meant for readers who want to

  • search for a substring in a string column (the simplest case)
  • search for multiple substrings (similar to isin)
  • match a whole word from text (e.g., "blue" should match "the sky is blue" but not "bluejay")
  • match multiple whole words
  • Understand the reason behind "ValueError: cannot index with vector containing NA / NaN values"

...and would like to know more about what methods should be preferred over others.

(P.S.: I've seen a lot of questions on similar topics, I thought it would be good to leave this here.)

Basic Substring Search

# setup
df1 = pd.DataFrame({'col': ['foo', 'foobar', 'bar', 'baz']})

0     foo
1  foobar
2     bar
3     baz

str.contains can be used to perform either substring searches or regex based search. The search defaults to regex-based unless you explicitly disable it.

Here is an example of regex-based search,

# find rows in `df1` which contain "foo" followed by something

1  foobar

Sometimes regex search is not required, so specify regex=False to disable it.

#select all rows containing "foo"
df1[df1['col'].str.contains('foo', regex=False)]
# same as df1[df1['col'].str.contains('foo')] but faster.

0     foo
1  foobar

Performance wise, regex search is slower than substring search:

df2 = pd.concat([df1] * 1000, ignore_index=True)

%timeit df2[df2['col'].str.contains('foo')]
%timeit df2[df2['col'].str.contains('foo', regex=False)]

6.31 ms ± 126 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.8 ms ± 241 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Avoid using regex-based search if you don't need it.

Addressing ValueErrors
Sometimes, performing a substring search and filtering on the result will result in

ValueError: cannot index with vector containing NA / NaN values

This is usually because of mixed data or NaNs in your object column,

s = pd.Series(['foo', 'foobar', np.nan, 'bar', 'baz', 123])

0     True
1     True
2      NaN
3     True
4    False
5      NaN
dtype: object

# ---------------------------------------------------------------------------
# ValueError                                Traceback (most recent call last)

Anything that is not a string cannot have string methods applied on it, so the result is NaN (naturally). In this case, specify na=False to ignore non-string data,

s.str.contains('foo|bar', na=False)

0     True
1     True
2    False
3     True
4    False
5    False
dtype: bool

Multiple Substring Search

This is most easily achieved through a regex search using the regex OR pipe.

# Slightly modified example.
df4 = pd.DataFrame({'col': ['foo abc', 'foobar xyz', 'bar32', 'baz 45']})

0     foo abc
1  foobar xyz
2       bar32
3      baz 45


0     foo abc
1  foobar xyz
3      baz 45

You can also create a list of terms, then join them:

terms = ['foo', 'baz']

0     foo abc
1  foobar xyz
3      baz 45

Sometimes, it is wise to escape your terms in case they have characters that can be interpreted as regex metacharacters. If your terms contain any of the following characters...

. ^ $ * + ? { } [ ] \ | ( )

Then, you'll need to use re.escape to escape them:

import re
df4[df4['col'].str.contains('|'.join(map(re.escape, terms)))]

0     foo abc
1  foobar xyz
3      baz 45

re.escape has the effect of escaping the special characters so they're treated literally.

# '\\.foo\\^'

Matching Entire Word(s)

By default, the substring search searches for the specified substring/pattern regardless of whether it is full word or not. To only match full words, we will need to make use of regular expressions here—in particular, our pattern will need to specify word boundaries (\b).

For example,

df3 = pd.DataFrame({'col': ['the sky is blue', 'bluejay by the window']})

0        the sky is blue
1  bluejay by the window

Now consider,


0        the sky is blue
1  bluejay by the window



0  the sky is blue

Multiple Whole Word Search

Similar to the above, except we add a word boundary (\b) to the joined pattern.

p = r'\b(?:{})\b'.format('|'.join(map(re.escape, terms)))

0  foo abc
3   baz 45

Where p looks like this,

# '\\b(?:foo|baz)\\b'

A Great Alternative: Use List Comprehensions!

Because you can! And you should! They are usually a little bit faster than string methods, because string methods are hard to vectorise and usually have loopy implementations.

Instead of,

df1[df1['col'].str.contains('foo', regex=False)]

Use the in operator inside a list comp,

df1[['foo' in x for x in df1['col']]]

0  foo abc
1   foobar

Instead of,

regex_pattern = r'foo(?!$)'

Use re.compile (to cache your regex) + inside a list comp,

p = re.compile(regex_pattern, flags=re.IGNORECASE)
df1[[bool( for x in df1['col']]]

1  foobar

If "col" has NaNs, then instead of

df1[df1['col'].str.contains(regex_pattern, na=False)]


def try_search(p, x):
        return bool(
    except TypeError:
        return False

p = re.compile(regex_pattern)
df1[[try_search(p, x) for x in df1['col']]]

1  foobar

More Options for Partial String Matching: np.char.find, np.vectorize, DataFrame.query.

In addition to str.contains and list comprehensions, you can also use the following alternatives.

Supports substring searches (read: no regex) only.

df4[np.char.find(df4['col'].values.astype(str), 'foo') > -1]

0     foo abc
1  foobar xyz

This is a wrapper around a loop, but with lesser overhead than most pandas str methods.

f = np.vectorize(lambda haystack, needle: needle in haystack)
f(df1['col'], 'foo')
# array([ True,  True, False, False])

df1[f(df1['col'], 'foo')]

0  foo abc
1   foobar

Regex solutions possible:

regex_pattern = r'foo(?!$)'
p = re.compile(regex_pattern)
f = np.vectorize(lambda x: pd.notna(x) and bool(

1  foobar

Supports string methods through the python engine. This offers no visible performance benefits, but is nonetheless useful to know if you need to dynamically generate your queries.

df1.query('col.str.contains("foo")', engine='python')

0     foo
1  foobar

More information on query and eval family of methods can be found at Dynamic Expression Evaluation in pandas using pd.eval().

Recommended Usage Precedence

  1. (First) str.contains, for its simplicity and ease handling NaNs and mixed data
  2. List comprehensions, for its performance (especially if your data is purely strings)
  3. np.vectorize
  4. (Last) df.query

@Denis de Bernardy 2019-07-16 11:37:03

Could you edit in the correct method to use when searching for a string in two or more columns? Basically: any(needle in haystack for needling in ['foo', 'bar'] and haystack in (df['col'], df['col2'])) and variations I tried all choke (it complains about any() and rightly so... But the doc is blissfully unclear as to how to do such a query.

@cs95 2019-07-28 06:30:17

@DenisdeBernardy df[['col1', 'col2']].apply(lambda x: x.str.contains('foo|bar')).any(axis=1)

@ankii 2019-07-28 07:41:06

@cs95 Extracting rows with substring containing whitespace after + in pandas df It was answered soon, but you might want to have a look at it.

@cs95 2019-07-28 07:53:52

@ankiiiiiii Looks like you missed the part of my answer where I mentioned regex metacharacters: "Sometimes, it is wise to escape your terms in case they have characters that can be interpreted as regex metacharacters".

@ankii 2019-07-28 07:55:51

@cs95 at that time, I thought only of space as the problem and searched this page and documentation page for the same. While writing the question, special char came to my mind and I got quick answers before I could search for it. Great answer of yours! +1

@00schneider 2019-08-13 10:54:51

Very helpful! Why do you need to insert r if regex = True by default? eg here: df1[df1['col'].str.contains(r'foo(?!$)')]

@cs95 2019-08-13 13:11:47

@00schneider r in this case is used to indicate a raw string literal. These make it easier to write regular expression strings.

@Murtaza Haji 2020-04-16 04:09:16

How to use this for finding values in multiple columns?

@xpeiro 2019-06-01 14:56:35

There are answers before this which accomplish the asked feature, anyway I would like to show the most generally way:


This way let's you get the column you look for whatever the way is wrote.

( Obviusly, you have to write the proper regex expression for each case )

@cs95 2019-06-23 05:18:51

This filters on the column headers. It isn't general, it's incorrect.

@cs95 2019-12-30 18:35:40

@MicheldeRuiter that's still incorrect, that'd filter on index labels instead!

@Michel de Ruiter 2019-12-31 10:38:02

Does not answer the question. But I learnt something. :)

@Philipp Schwarz 2016-10-12 21:04:32

If anyone wonders how to perform a related problem: "Select column by partial string"


df.filter(like='hello')  # select columns which contain the word hello

And to select rows by partial string matching, pass axis=0 to filter:

# selects rows which contain the word hello in their index label
df.filter(like='hello', axis=0)  

@elPastor 2017-06-17 21:53:57

This can be distilled to: df.loc[:, df.columns.str.contains('a')]

@Ted Petrou 2017-10-25 02:57:07

which can be further distilled to df.filter(like='a')

@PV8 2020-01-09 09:35:13

this should be an own question + answer, already 50 people searched for it...

@Philipp Schwarz 2020-01-09 09:37:22

@PV8 question already exists:…. But when I search on google for "pandas Select column by partial string", this thread appears first

@Garrett 2012-07-17 21:52:18

Based on github issue #620, it looks like you'll soon be able to do the following:


Update: vectorized string methods (i.e., Series.str) are available in pandas 0.8.1 and up.

@LonelySoul 2013-06-27 16:41:13

How do we go about "Hello" and "Britain" if I want to find them with "OR" condition.

@Garrett 2013-06-27 19:20:45

Since str.* methods treat the input pattern as a regular expression, you can use df[df['A'].str.contains("Hello|Britain")]

@zyxue 2017-03-01 17:25:43

Is it possible to convert .str.contains to use .query() api?

@Franck Dernoncourt 2017-07-05 18:01:43

@François Leblanc 2018-02-13 20:22:43

df[df['value'].astype(str).str.contains('1234.+')] for filtering out non-string-type columns.

@NeStack 2018-11-30 17:32:40

What about "AND" condition - how do you look for multiple strings on the same time, so not just "hello", but ["hello", "hey", "hi"]?

@Garrett 2018-11-30 19:35:34

to "AND" substrings when the order of substrings is important/known, you could use df[df.A.str.contains("STR1.*STR2")]. if order is unimportant/unknown, df[df.A.str.contains("STR1") & df.A.str.contains("STR2")]

@cs95 2019-03-25 20:20:10

@NeStack I've added more information about multiple substring searches here.

@Mike 2014-11-10 19:26:27

Say you have the following DataFrame:

>>> df = pd.DataFrame([['hello', 'hello world'], ['abcd', 'defg']], columns=['a','b'])
>>> df
       a            b
0  hello  hello world
1   abcd         defg

You can always use the in operator in a lambda expression to create your filter.

>>> df.apply(lambda x: x['a'] in x['b'], axis=1)
0     True
1    False
dtype: bool

The trick here is to use the axis=1 option in the apply to pass elements to the lambda function row by row, as opposed to column by column.

@ComplexData 2016-10-18 20:23:28

How do I modify above to say that x['a'] exists only in beginning of x['b']?

@cs95 2019-03-25 10:27:59

apply is a bad idea here in terms of performance and memory. See this answer.

@Christian 2014-04-10 15:36:14

Quick note: if you want to do selection based on a partial string contained in the index, try the following:


@Yury Bayda 2015-05-08 21:27:16

You can just df[df.index.to_series().str.contains('LLChit')]

@euforia 2012-07-06 17:08:46

Here's what I ended up doing for partial string matches. If anyone has a more efficient way of doing this please let me know.

def stringSearchColumn_DataFrame(df, colName, regex):
    newdf = DataFrame()
    for idx, record in df[colName].iteritems():

        if, record):
            newdf = concat([df[df[colName] == record], newdf], ignore_index=True)

    return newdf

@MarkokraM 2014-04-10 13:56:53

Should be 2x to 3x faster if you compile regex before loop: regex = re.compile(regex) and then if

@Teepeemm 2018-06-20 19:36:04

@MarkokraM says that the most recent regexs are cached for you, so you don't need to compile yourself.

@cs95 2019-03-25 10:26:10

Do not use iteritems to iterate over a DataFrame. It ranks last in terms of pandorability and performance

Related Questions

Sponsored Content

15 Answered Questions

[SOLVED] Delete column from pandas DataFrame

64 Answered Questions

[SOLVED] How to replace all occurrences of a string?

21 Answered Questions

[SOLVED] How to iterate over rows in a DataFrame in Pandas?

17 Answered Questions

[SOLVED] Selecting multiple columns in a pandas dataframe

26 Answered Questions

[SOLVED] Renaming columns in pandas

64 Answered Questions

[SOLVED] What is the difference between String and string in C#?

10 Answered Questions

[SOLVED] Does Python have a string 'contains' substring method?

59 Answered Questions

[SOLVED] How do I read / convert an InputStream into a String in Java?

3 Answered Questions

10 Answered Questions

[SOLVED] How to select rows from a DataFrame based on column values?

Sponsored Content