By Huey


2016-08-07 00:18:40 8 Comments

At first, I tried writing some code that looked like this:

import numpy as np
import pandas as pd
np.random.seed(2016)
train = pd.DataFrame(np.random.choice([np.nan, 1, 2], size=(10, 3)), 
                     columns=['Age', 'SibSp', 'Parch'])

complete = train.dropna()    
complete['AgeGt15'] = complete['Age'] > 15

After getting SettingWithCopyWarning, I tried using.loc:

complete.loc[:, 'AgeGt15'] = complete['Age'] > 15
complete.loc[:, 'WithFamily'] = complete['SibSp'] + complete['Parch'] > 0

However, I still get the same warning. What gives?

3 comments

@Jamie Edgecombe 2019-11-08 12:29:14

I think your .loc solution would work, if it wasn't for the np.nans in the original data frame. You could either complete = train.dropna().reset_index() or Pandas .assign() will avoid SettingWithCopyWarning and is the recommended way of creating new columns, returning a new data frame object. Your example:

complete = complete.assign(**{'AgeGt15': np.where(complete['Age'] > 15, True, False)})

@unutbu 2016-08-07 01:08:52

Note: As of pandas version 0.24, is_copy is deprecated and will be removed in a future version. While the private attribute _is_copy exists, the underscore indicates this attribute is not part of the public API and therefore should not be depended upon. Therefore, going forward, it seems the only proper way to silence SettingWithCopyWarning will be to do so globally:

pd.options.mode.chained_assignment = None

When complete = train.dropna() is executed, dropna might return a copy, so out of an abundance of caution, Pandas sets complete.is_copy to a Truthy value:

In [220]: complete.is_copy
Out[220]: <weakref at 0x7f7f0b295b38; to 'DataFrame' at 0x7f7eee6fe668>

This allows Pandas to warn you later, when complete['AgeGt15'] = complete['Age'] > 15 is executed that you may be modifying a copy which will have no effect on train. For beginners this may be a useful warning. In your case, it appears you have no intention of modifying train indirectly by modifying complete. Therefore the warning is just a meaningless annoyance in your case.

You can silence the warning by setting,

complete.is_copy = False       # deprecated as of version 0.24

This is quicker than making an actual copy, and nips the SettingWithCopyWarning in the bud (at the point where _check_setitem_copy is called):

def _check_setitem_copy(self, stacklevel=4, t='setting', force=False):
    if force or self.is_copy:
        ...

If you are really confident you know what you are doing, you can shut off the SettingWithCopyWarning globally with

pd.options.mode.chained_assignment = None # None|'warn'|'raise'

An alternative way to silence the warning is to make a new copy:

complete = complete.copy()

However, you may not want to do this if the DataFrame is large, since copying can take a significant amount of time and memory, and it is completely pointless (except for the sake of silencing a warning) if you know complete is already a copy.

@ayhan 2016-08-07 01:22:11

Do you think this is consistent? It raises the same warning with drop_duplicates but not with drop.

@unutbu 2016-08-07 01:39:30

@ayhan: There is also no warning if complete = complete.assign(AgeGt15=(complete['Age'] > 15)) is used. The mechanism Pandas uses to infer SettingWithCopyWarning is not fool-proof. It catches the most common cases, but not all.

@MaxU 2016-08-07 11:21:39

thank you for the is_copy trick - i didn't know it exists

@nnaqa 2018-10-02 05:35:12

I resolve it by creating copy of dataframe:

complete = train.copy()

@Mike 2019-03-10 18:11:39

Although the marked answer recommends using is_copy, it is not the best way to handle as is_copy will also throw an error message saying it is deprecated -- see panda docs. In light of this, copy() is the best way to ensure no warning messages are thrown.

Related Questions

Sponsored Content

18 Answered Questions

[SOLVED] Get list from pandas DataFrame column headers

14 Answered Questions

[SOLVED] "Large data" work flows using pandas

13 Answered Questions

[SOLVED] How to deal with SettingWithCopyWarning in Pandas?

10 Answered Questions

[SOLVED] How do I get the row count of a pandas DataFrame?

0 Answered Questions

unexpected SettingWithCopyWarning with pandas

0 Answered Questions

1 Answered Questions

1 Answered Questions

SettingWithCopyWarning when modifying a single column in pandas

  • 2017-04-28 15:30:21
  • Aufban
  • 574 View
  • 0 Score
  • 1 Answer
  • Tags:   python pandas

1 Answered Questions

[SOLVED] Dealing with SettingWithCopyWarning when assigning columns in Pandas

  • 2017-02-24 12:44:09
  • Maximilian Köstler
  • 2313 View
  • 2 Score
  • 1 Answer
  • Tags:   python pandas

0 Answered Questions

Is this Pandas 'SettingWithCopyWarning' a False Positive?

  • 2016-10-08 19:04:02
  • traggatmot
  • 173 View
  • 1 Score
  • 0 Answer
  • Tags:   python pandas

Sponsored Content