By medriscoll


2009-07-28 18:21:47 8 Comments

I have a data frame containing a factor. When I create a subset of this data frame using subset() or another indexing function, a new data frame is created. However, the factor variable retains all of its original levels -- even when they do not exist in the new data frame.

This creates headaches when doing faceted plotting or using functions that rely on factor levels.

What is the most succinct way to remove levels from a factor in my new data frame?

Here's my example:

df <- data.frame(letters=letters[1:5],
                    numbers=seq(1:5))

levels(df$letters)
## [1] "a" "b" "c" "d" "e"

subdf <- subset(df, numbers <= 3)
##   letters numbers
## 1       a       1
## 2       b       2
## 3       c       3    

## but the levels are still there!
levels(subdf$letters)
## [1] "a" "b" "c" "d" "e"

13 comments

@Jerome Smith 2019-01-28 21:10:38

Unfortunately factor() doesn't seem to work when using rxDataStep of RevoScaleR. I do it in two steps: 1) Convert to character and store in temporary external data frame (.xdf). 2) Convert back to factor and store in definitive external data frame. This eliminates any unused factor levels, without loading all the data into memory.

# Step 1) Converts to character, in temporary xdf file:
rxDataStep(inData = "input.xdf", outFile = "temp.xdf", transforms = list(VAR_X = as.character(VAR_X)), overwrite = T)
# Step 2) Converts back to factor:
rxDataStep(inData = "temp.xdf", outFile = "output.xdf", transforms = list(VAR_X = as.factor(VAR_X)), overwrite = T)

@Aurèle 2017-06-12 09:44:29

For the sake of completeness, now there is also fct_drop in the forcats package http://forcats.tidyverse.org/reference/fct_drop.html.

It differs from droplevels in the way it deals with NA:

f <- factor(c("a", "b", NA), exclude = NULL)

droplevels(f)
# [1] a    b    <NA>
# Levels: a b <NA>

forcats::fct_drop(f)
# [1] a    b    <NA>
# Levels: a b

@hatmatrix 2009-07-28 22:41:31

All you should have to do is to apply factor() to your variable again after subsetting:

> subdf$letters
[1] a b c
Levels: a b c d e
subdf$letters <- factor(subdf$letters)
> subdf$letters
[1] a b c
Levels: a b c

EDIT

From the factor page example:

factor(ff)      # drops the levels that do not occur

For dropping levels from all factor columns in a dataframe, you can use:

subdf <- subset(df, numbers <= 3)
subdf[] <- lapply(subdf, function(x) if(is.factor(x)) factor(x) else x)

@Dirk Eddelbuettel 2009-07-29 14:16:12

That's fine for a one-off, but in a data.frame with a large number of columns, you get to do that on every column that is a factor ... leading to the need for a function such as drop.levels() from gdata.

@hatmatrix 2009-07-29 17:09:15

I see... but from a user-perspective it's quick to write something like subdf[] <- lapply(subdf,function(x) if(is.factor(x)) factor(x) else x) ...Is drop.levels() much more efficient computationally or better with large data sets? (One would have to rewrite the line above in a for-loop for a huge data frame, I suppose.)

@medriscoll 2009-07-30 04:18:31

Thanks Stephen & Dirk - I'm giving this one the thumbs up for the caes of one factor, but hopefully folks will read these comments for your suggestions on cleaning up an entire data frame of factors.

@Johan 2014-05-09 10:41:33

As a side-effect the function converts the data frame to a list, so the mydf <- droplevels(mydf) solution suggested by Roman Luštrik and Tommy O'Dell below is preferable.

@Matt Bannert 2014-06-19 14:31:44

What also might be noteworthy: rlm really goes wrong when your data.frame contains factors which contain levels with no data. You'll get an error: singular fits are not implemented in 'rlm' . Most of the time your matrix is not singular, it's just exactly this problem.

@webelo 2016-07-01 00:36:30

Also: this method does preserve the ordering of the variable.

@jangorecki 2015-12-09 14:56:58

Looking at the droplevels methods code in the R source you can see it wraps to factor function. That means you can basically recreate the column with factor function.
Below the data.table way to drop levels from all the factor columns.

library(data.table)
dt = data.table(letters=factor(letters[1:5]), numbers=seq(1:5))
levels(dt$letters)
#[1] "a" "b" "c" "d" "e"
subdt = dt[numbers <= 3]
levels(subdt$letters)
#[1] "a" "b" "c" "d" "e"

upd.cols = sapply(subdt, is.factor)
subdt[, names(subdt)[upd.cols] := lapply(.SD, factor), .SDcols = upd.cols]
levels(subdt$letters)
#[1] "a" "b" "c"

@David Arenburg 2016-01-24 13:24:08

I think the data.table way would be something like for (j in names(DT)[sapply(DT, is.factor)]) set(DT, j = j, value = factor(DT[[j]]))

@jangorecki 2016-11-30 13:25:27

@DavidArenburg it doesn't change much here as we call [.data.table only once

@Prradep 2015-07-15 11:14:54

Another way of doing the same but with dplyr

library(dplyr)
subdf <- df %>% filter(numbers <= 3) %>% droplevels()
str(subdf)

Edit:

Also Works ! Thanks to agenis

subdf <- df %>% filter(numbers <= 3) %>% droplevels
levels(subdf$letters)

@DfAC 2015-05-25 12:08:08

Very interesting thread, I especially liked idea to just factor subselection again. I had the similar problem before and I just converted to character and then back to factor.

   df <- data.frame(letters=letters[1:5],numbers=seq(1:5))
   levels(df$letters)
   ## [1] "a" "b" "c" "d" "e"
   subdf <- df[df$numbers <= 3]
   subdf$letters<-factor(as.character(subdf$letters))

@Gregor 2019-02-13 15:47:22

I mean, factor(as.chracter(...)) works, but just less efficiently and succinctly than factor(...). Seems strictly worse than the other answers.

@Diogo 2014-01-31 03:25:56

here is a way of doing that

varFactor <- factor(letters[1:15])
varFactor <- varFactor[1:5]
varFactor <- varFactor[drop=T]

@David Arenburg 2019-02-13 15:43:57

This is a dupe of this answer that was posted 5 years earlier.

@Roman Luštrik 2010-11-26 11:37:26

Since R version 2.12, there's a droplevels() function.

levels(droplevels(subdf$letters))

@Señor O 2014-01-30 17:28:23

Alternatively, you can just scroll down a little bit...

@tim 2015-06-28 19:02:59

@RomanLuštrik Sadly sort by votes still makes the accepted answer #1, even though it (now) has fewer votes than yours :-(

@Mars 2015-11-21 05:44:51

An advantage of this method over using factor() is that it's not necessary to modify the original dataframe or create a new persistent dataframe. I can wrap droplevels around a subsetted dataframe and use it as the data argument to a lattice function, and groups will be handled correctly.

@Meep 2016-07-05 00:48:10

I've noticed that if I have an NA level in my factor (a genuine NA level), it is dropped by dropped levels, even if the NAs are present.

@Brendan OConnor 2009-09-01 20:37:36

I wrote utility functions to do this. Now that I know about gdata's drop.levels, it looks pretty similar. Here they are (from here):

present_levels <- function(x) intersect(levels(x), x)

trim_levels <- function(...) UseMethod("trim_levels")

trim_levels.factor <- function(x)  factor(x, levels=present_levels(x))

trim_levels.data.frame <- function(x) {
  for (n in names(x))
    if (is.factor(x[,n]))
      x[,n] = trim_levels(x[,n])
  x
}

@ars 2009-07-29 03:40:37

Here's another way, which I believe is equivalent to the factor(..) approach:

> df <- data.frame(let=letters[1:5], num=1:5)
> subdf <- df[df$num <= 3, ]

> subdf$let <- subdf$let[ , drop=TRUE]

> levels(subdf$let)
[1] "a" "b" "c"

@David Arenburg 2019-02-13 15:42:05

Ha, after all these years I didn't know there is a `[.factor` method that has a drop argument and you've posted this in 2009...

@hadley 2009-07-28 23:53:43

If you don't want this behaviour, don't use factors, use character vectors instead. I think this makes more sense than patching things up afterwards. Try the following before loading your data with read.table or read.csv:

options(stringsAsFactors = FALSE)

The disadvantage is that you're restricted to alphabetical ordering. (reorder is your friend for plots)

@andrewj 2009-07-29 01:37:27

You can also do read.csv(file='foo.csv', as.is=T).

@Matt Parker 2009-07-28 18:44:32

This is obnoxious. This is how I usually do it, to avoid loading other packages:

levels(subdf$letters)<-c("a","b","c",NA,NA)

which gets you:

> subdf$letters
[1] a b c
Levels: a b c

Note that the new levels will replace whatever occupies their index in the old levels(subdf$letters), so something like:

levels(subdf$letters)<-c(NA,"a","c",NA,"b")

won't work.

This is obviously not ideal when you have lots of levels, but for a few, it's quick and easy.

@Dirk Eddelbuettel 2009-07-28 18:37:13

It is a known issue, and one possible remedy is provided by drop.levels() in the gdata package where your example becomes

> drop.levels(subdf)
  letters numbers
1       a       1
2       b       2
3       c       3
> levels(drop.levels(subdf)$letters)
[1] "a" "b" "c"

There is also the dropUnusedLevels function in the Hmisc package. However, it only works by altering the subset operator [ and is not applicable here.

As a corollary, a direct approach on a per-column basis is a simple as.factor(as.character(data)):

> levels(subdf$letters)
[1] "a" "b" "c" "d" "e"
> subdf$letters <- as.factor(as.character(subdf$letters))
> levels(subdf$letters)
[1] "a" "b" "c"

@daroczig 2011-01-17 11:31:13

The reorder parameter of the drop.levels function is worth mentioning: if you have to preserve the original order of your factors, use it with FALSE value.

@Vrokipal 2018-06-20 19:12:13

Using gdata for just drop.levels yields "gdata: read.xls support for 'XLS' (Excel 97-2004) files ENABLED." "gdata: Unable to load perl libaries needed by read.xls()" "gdata: to support 'XLSX' (Excel 2007+) files." "gdata: Run the function 'installXLSXsupport()'" "gdata: to automatically download and install the perl". Use droplevels from baseR (stackoverflow.com/a/17218028/9295807)

@Dirk Eddelbuettel 2018-06-20 19:21:33

Stuff happens over time. You are commenting on an answer I wrote nine years ago. So let's take this as a hint to generally prefer base R solutions as those are the ones using functionality that is still going to be around N years from now.

Related Questions

Sponsored Content

13 Answered Questions

[SOLVED] How to join (merge) data frames (inner, outer, left, right)?

22 Answered Questions

[SOLVED] Adding new column to existing DataFrame in Python pandas

19 Answered Questions

[SOLVED] Drop data frame columns by name

  • 2011-01-05 14:34:29
  • Btibert3
  • 1168942 View
  • 744 Score
  • 19 Answer
  • Tags:   r dataframe r-faq

3 Answered Questions

[SOLVED] Override [.data.frame to drop unused factor levels by default

1 Answered Questions

R empty data frame after subsetting by factor

6 Answered Questions

[SOLVED] Elegant way to drop rare factor levels from data frame

  • 2014-06-17 08:34:36
  • BiXiC
  • 3890 View
  • 8 Score
  • 6 Answer
  • Tags:   r subset

7 Answered Questions

[SOLVED] Reorder levels of a factor without changing order of values

  • 2010-03-03 22:44:57
  • crangos
  • 74999 View
  • 107 Score
  • 7 Answer
  • Tags:   r levels

1 Answered Questions

[SOLVED] Subset a data frame for each factor level

  • 2016-10-27 21:08:53
  • Bobby
  • 325 View
  • 1 Score
  • 1 Answer
  • Tags:   r subset

2 Answered Questions

[SOLVED] R - show only levels used in a subset of data frame

  • 2015-02-17 21:35:26
  • Alium Britt
  • 1034 View
  • 2 Score
  • 2 Answer
  • Tags:   r dataframe r-factor

2 Answered Questions

[SOLVED] Subset all levels of a single factor

  • 2013-07-28 05:59:05
  • whistler
  • 6434 View
  • 7 Score
  • 2 Answer
  • Tags:   r

Sponsored Content