By Chris


2019-01-11 04:14:18 8 Comments

I need to run Batch API on a huge dataset. Specifically I have a custom table with a lot of rows which I need to iterate over. The batch process works fine using small numbers. When using the full set I run out of memory before it even starts because I am loading a very large array of IDs to pass in to the batch function. Think of a 5 million item array. What is a good way to handle this? Break it into smaller chunks somehow to then pass in to Batch API? Increase the memory limit to an (ungodly) amount?

1 comments

@EagleUK 2019-01-21 15:22:36

Add a column for 'processed'. Each time you run the batch process select rows with a limit of 100 unprocessed rows for you batch process and then mark each row as processed.

You could run this on cron or something or run a nested batch.

Related Questions

Sponsored Content

1 Answered Questions

[SOLVED] Batch API: What am I doing wrong?

  • 2018-08-29 10:28:43
  • Jimmyb_1991
  • 40 View
  • 1 Score
  • 1 Answer
  • Tags:   8 batch-api

1 Answered Questions

[SOLVED] Passing form value into batch process

  • 2018-04-03 19:35:28
  • Shawn Matthews
  • 57 View
  • 1 Score
  • 1 Answer
  • Tags:   forms batch-api

1 Answered Questions

Batch API timeout during initialization

  • 2017-07-15 18:07:00
  • NoRandom
  • 224 View
  • 0 Score
  • 1 Answer
  • Tags:   batch-api

1 Answered Questions

[SOLVED] Start next batch process automatically

  • 2015-11-25 07:10:00
  • RDP
  • 56 View
  • 0 Score
  • 1 Answer
  • Tags:   7 batch-api

1 Answered Questions

[SOLVED] How to solve claimitem() in batch process?

  • 2015-08-03 13:10:31
  • Ajay Reddy
  • 125 View
  • 0 Score
  • 1 Answer
  • Tags:   8 batch-api

1 Answered Questions

[SOLVED] How do I process a huge list of files?

  • 2015-06-08 03:34:15
  • Free Radical
  • 123 View
  • 0 Score
  • 1 Answer
  • Tags:   7 cron batch-api

1 Answered Questions

[SOLVED] Batch Process in D8

  • 2015-06-03 10:24:03
  • Ajay Reddy
  • 2275 View
  • -1 Score
  • 1 Answer
  • Tags:   8 batch-api

2 Answered Questions

[SOLVED] Feeds - Background process vs Batch API

1 Answered Questions

[SOLVED] Batch API callback method is not call

  • 2014-12-16 16:59:49
  • Drupal Dev
  • 61 View
  • 0 Score
  • 1 Answer
  • Tags:   batch-api

2 Answered Questions

[SOLVED] Can a batch process be stopped

  • 2013-11-03 07:15:17
  • Robbert
  • 4133 View
  • 7 Score
  • 2 Answer
  • Tags:   7 batch-api

Sponsored Content