Handling Large Scale Deletions

Nextworld Platform
1

Guidance: Best Approach for Handling Large-Scale Deletions in Nextworld

1. Why Data Purge is the Recommended Solution

For deleting hundreds of thousands of records, we strongly recommend using the Data Purge framework over Logic Block implementations. Here’s why:

Data Purge Benefits: • Chunked deletion with automatic commits - Deletes in 10,000 record batches with commits between chunks • Built-in dead tuple management - Prevents table bloat by releasing locks between batches • Automatic cascade handling - Manages header-detail relationships correctly • Progress tracking and preview mode - See impact before execution • No code required - Configure through metadata

2. How to Configure Data Purge

To set up a Data Purge:

Step 1: Create a Purge Configuration record with: • Table: (Your specific table) • Filter criteria: (Your business rules) • Purge frequency: How often to run (can be triggered on-demand) • Age threshold: Optional minimum age of records before deletion

Step 2: Use the Data Purge App to: • Run in preview mode first to validate impact • Review estimated deletion counts • Execute the actual purge during off-peak hours

Step 3: For dynamic filters, you can: • Update the purge configuration programmatically before execution • Use the DataPurgeAppService API to trigger with updated criteria

3. If Data Purge Cannot Be Used

If business requirements absolutely prevent using Data Purge (e.g., real-time user-triggered deletes), here’s the least problematic Logic Block approach:

Critical Implementation Requirements:

  1. Create a BACK (background) type Logic Block

  2. Fetch records in small batches (500-1000 max)

  3. Delete each batch in a separate transaction

  4. Add delay between batches (100-500ms)

  5. Track progress for user visibility

:warning: Why This Will Still Be Problematic: • No bulk delete capability - Each record deleted individually • Long execution times - 100k records could take hours • Database impact - Continuous load on the database • No automatic cascade handling - Must manually handle related records • Transaction log growth - Each delete is a separate transaction • Difficult rollback - No built-in recovery mechanism

Sample Pattern (conceptual):

Logic Block: DeleteRecordsFromTable
Type: BACK (Background)
Actions:
1. Query with limit 500
2. For each record: Delete
3. Commit transaction
4. If more records exist: Schedule next batch

:pushpin: Recommendation

Given the volume (hundreds of thousands of records) and the performance implications, I strongly recommend working with your Solution Architect to implement Data Purge rather than Logic Blocks. The infrastructure impact of row-by-row deletion at this scale could affect system performance for all users.

2

Relates to Best Logic Approach for Mass Deletes. Jillene’s guidance here is the best answer.

4

If I want to run a scheduled purge - for example, delete all records in a table that are more than 7 days old - what is the best way to do that?

My initial thought involves of configuring a Job Scheduler that calls the DataPurgeExecutePurgeWrapper and passes it one of the Data Purge Definition records as a driving record. Does that seem reasonable?

5

A reason why I ask is, when defining a table, there are these Purge Active and Data Purge Group options. The help for those references the Purge Configuration Setup app.

image
image3244×991 142 KB

However the help for Purge Configuration Setup says it’s for internal use only.

image
image2941×1104 293 KB

So I’m confused why that Purge Active field is available for citizen developers if the feature that leverages it is for internal use, and the correct feature to be used isn’t mentioned at all in the Table Definitions app.

6

Yes, this is the preferred approach. The path you mention in your second comment was our original idea for handling these use cases, but was overly convoluted and limited. Passing a Data Purge Definition to a Job Scheduler is much simpler, more flexible, and configurable for everyone’s needs.