By NC maize breeding Jim


2014-01-27 19:37:56 8 Comments

My first SO question: I am confused about this behavior of apply method of groupby in pandas (0.12.0-4), it appears to apply the function TWICE to the first row of a data frame. For example:

>>> from pandas import Series, DataFrame
>>> import pandas as pd
>>> df = pd.DataFrame({'class': ['A', 'B', 'C'], 'count':[1,0,2]})
>>> print(df)
   class  count  
0     A      1  
1     B      0    
2     C      2

I first check that the groupby function works ok, and it seems to be fine:

>>> for group in df.groupby('class', group_keys = True):
>>>     print(group)
('A',   class  count
0     A      1)
('B',   class  count
1     B      0)
('C',   class  count
2     C      2)

Then I try to do something similar using apply on the groupby object and I get the first row output twice:

>>> def checkit(group):
>>>     print(group)
>>> df.groupby('class', group_keys = True).apply(checkit)
  class  count
0     A      1
  class  count
0     A      1
  class  count
1     B      0
  class  count
2     C      2

Any help would be appreciated! Thanks.

Edit: @Jeff provides the answer below. I am dense and did not understand it immediately, so here is a simple example to show that despite the double printout of the first group in the example above, the apply method operates only once on the first group and does not mutate the original data frame:

>>> def addone(group):
>>>     group['count'] += 1
>>>     return group

>>> df.groupby('class', group_keys = True).apply(addone)
>>> print(df)

      class  count
0     A      1
1     B      0
2     C      2

But by assigning the return of the method to a new object, we see that it works as expected:

df2 = df.groupby('class', group_keys = True).apply(addone) print(df2)

      class  count
0     A      2
1     B      1
2     C      3

3 comments

@cs95 2019-05-20 06:32:41

This "issue" has now been fixed: Upgrade to 0.25+

Starting from v0.25, GroupBy.apply() will only evaluate the first group once. See GH24748.

Relevant example from documentation:

pd.__version__                                                                                                          
# '0.25.0.dev0+590.g44d5498d8'

df = pd.DataFrame({"a": ["x", "y"], "b": [1, 2]})                                                                      

def func(group): 
    print(group.name) 
    return group                                                                                                                     

New behaviour (>=v0.25):

df.groupby('a').apply(func)                                                                                            
x
y

   a  b
0  x  1
1  y  2

Old behaviour (<=v0.24.x):

df.groupby('a').apply(func)
x
x
y

   a  b
0  x  1
1  y  2

Pandas still uses the first group to determine whether apply can take a fast path or not. But at least it no longer has to evaluate the first group twice. Nice work, devs!

@CMCDragonkai 2019-09-13 07:59:40

Oh so basically Pandas will still run apply twice on the first row. This fix only applies to the group in groupby.apply. Damn.

@geosmart 2018-04-04 03:17:33

you can use for loop to avoid the groupby.apply duplicate first row,

log_sample.csv

guestid,keyword
1,null
2,null
2,null
3,null
3,null
3,null
4,null
4,null
4,null
4,null

my code snippit

df=pd.read_csv("log_sample.csv") 
grouped = df.groupby("guestid")

for guestid, df_group in grouped:
    print(list(df_group['guestid'])) 

df.head(100)

output

[1]
[2, 2]
[3, 3, 3]
[4, 4, 4, 4]

@Zero 2014-09-08 01:39:33

This is by design, as described here and here

The apply function needs to know the shape of the returned data to intelligently figure out how it will be combined. To do this it calls the function (checkit in your case) twice to achieve this.

Depending on your actual use case, you can replace the call to apply with aggregate, transform or filter, as described in detail here. These functions require the return value to be a particular shape, and so don't call the function twice.

However - if the function you are calling does not have side-effects, it most likely does not matter that the function is being called twice on the first value.

Related Questions

Sponsored Content

22 Answered Questions

[SOLVED] How to iterate over rows in a DataFrame in Pandas?

15 Answered Questions

[SOLVED] Delete column from pandas DataFrame

17 Answered Questions

[SOLVED] Selecting multiple columns in a pandas dataframe

26 Answered Questions

[SOLVED] Renaming columns in pandas

20 Answered Questions

10 Answered Questions

[SOLVED] Group by in LINQ

  • 2011-09-06 19:44:20
  • test123
  • 1303135 View
  • 1061 Score
  • 10 Answer
  • Tags:   c# linq group-by

2 Answered Questions

[SOLVED] Using group by on multiple columns

10 Answered Questions

[SOLVED] Does Python have a string 'contains' substring method?

16 Answered Questions

[SOLVED] Select first row in each GROUP BY group?

Sponsored Content