# [Feature Request] Generic Data Archiving Module for ABP Framework + EF Core

[Osama94]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe the problem.

Summary

Add a generic data archiving module to ABP that moves terminal/historical aggregate roots from the
main transactional database to a separate archive database — with GZip compress/decompress support,
distributed locking, and multi-tenancy.

---

Community Need

There's an open support question with no real answer yet:

ABP Support #7891 — Inquiry on Archival Mechanism for Managing Rapid Data Growth in ABP with EF Core

The response was: "We have no experience in this case. You can implement such a feature in EF Core."

So there's nothing built-in today. Any ABP app that handles high-volume data will hit this wall
eventually and end up rolling their own — usually missing idempotency, multi-tenancy, or
compensating transactions.

---

Problem Statement

• ABP Framework version: v8.2.2+ (v9/v10 compatible)
• Database: SQL Server / MySQL (EF Core)
• Topology: Tiered, Multi-Tenant

When transactional tables grow into millions of rows, performance drops even with good indexes.
The fix is periodically moving completed/expired records to a separate archive store. ABP has no
abstraction for this today.

Proposed Solution

A module — Abp.DataArchiving — that handles the scaffolding. You implement three interfaces for
your own domain; the module takes care of the rest.

Core abstractions

// Tag your source (live) entity
public interface IArchivableEntity { }

// Tag your archive (destination) entity
public interface IHasArchiveMetadata
{
    DateTime ArchivedAt { get; set; }
    DateTime OriginalLastModificationTime { get; set; }
}

// Define WHICH records are eligible
public interface IArchivingCriteria<TEntity>
    where TEntity : class, IEntity<Guid>, IArchivableEntity
{
    IQueryable<TEntity> Apply(IQueryable<TEntity> query, DateTime cutoffDate);
}

// Define HOW to map + compress from source to archive
public interface IArchiveMapper<TSource, TArchive>
    where TSource  : class, IEntity<Guid>, IArchivableEntity
    where TArchive : class, IEntity<Guid>, IHasArchiveMetadata
{
    TArchive Map(TSource source);
}

// Lifecycle coordinator
public interface IArchivingService<TSource, TArchive>
    where TSource  : class, IEntity<Guid>, IArchivableEntity
    where TArchive : class, IEntity<Guid>, IHasArchiveMetadata
{
    Task<List<Guid>> GetEligibleIdsAsync(DateTime cutoffDate, int batchSize, CancellationToken ct = default);
    Task ArchiveSingleAsync(Guid id, CancellationToken ct = default);
}

// Compress / decompress JSON columns
public interface IArchiveCompressionService
{
    byte[]  CompressObject<T>(T obj)             where T : class;
    T?      DecompressObject<T>(byte[]? compressed) where T : class;
}

What the module provides out of the box

Component	What it does
ArchiveDbContextBase<T>	Extends AbpDbContext<T>; disables soft-delete and multi-tenancy query filters on archive tables
ArchivingServiceBase<TSource,TArchive,TSourceDb,TArchiveDb>	Full lifecycle: load → idempotency check → map+compress → write archive → delete source → compensate if source delete fails
ArchivingBackgroundWorker<TSource,TArchive,TService>	AsyncPeriodicBackgroundWorkerBase with distributed lock, per-record IUnitOfWork, multi-tenant iteration via ITenantRepository + ICurrentTenant.Change()
GZipArchiveCompressionService	Default IArchiveCompressionService; GZip at CompressionLevel.Optimal, swappable via DI
ArchivingOptions	IsEnabled, AgeMonths, BatchSize, PeriodMs, IsMultiTenant, UseDistributedLock

Writing archived data (compress on the way in)

// Eligibility: which records to pick up
public class CompletedOrderCriteria : IArchivingCriteria<Order>
{
    public IQueryable<Order> Apply(IQueryable<Order> query, DateTime cutoffDate)
        => query.Where(o => o.Status == OrderStatus.Completed
                         && o.LastModificationTime < cutoffDate);
}

// Mapper: transform + compress large JSON columns
public class OrderToArchiveMapper : IArchiveMapper<Order, OrderArchive>
{
    private readonly IArchiveCompressionService _compression;
    public OrderToArchiveMapper(IArchiveCompressionService compression)
        => _compression = compression;

    public OrderArchive Map(Order src) => new(src.Id)
    {
        // scalar columns stay as-is
        OrderNumber  = src.OrderNumber,
        Status       = src.Status,

        // large JSON columns → GZip compressed byte[]
        Payload_Compressed  = _compression.CompressObject(src.Payload),
        History_Compressed  = _compression.CompressObject(src.History),

        // audit trail
        CreationTime                 = src.CreationTime,
        CreatorId                    = src.CreatorId,
        LastModificationTime         = src.LastModificationTime,
        LastModifierId               = src.LastModifierId,

        // archive metadata
        ArchivedAt                   = DateTime.UtcNow,
        OriginalLastModificationTime = src.LastModificationTime ?? src.CreationTime
    };
}

Reading archived data (decompress on the way out)

public class OrderArchiveRepository
    : EfCoreRepository<OrderArchiveDbContext, OrderArchive, Guid>,
      IOrderArchiveRepository
{
    private readonly IArchiveCompressionService _compression;

    public OrderArchiveRepository(
        IDbContextProvider<OrderArchiveDbContext> dbContextProvider,
        IArchiveCompressionService compression)
        : base(dbContextProvider)
    {
        _compression = compression;
    }

    public async Task<OrderArchiveDetailDto> GetDetailAsync(Guid id)
    {
        var db     = await GetDbContextAsync();
        var entity = await db.OrderArchives
            .AsNoTracking()
            .FirstOrDefaultAsync(x => x.Id == id);

        if (entity is null) return null;

        return new OrderArchiveDetailDto
        {
            // scalar columns read directly
            OrderNumber  = entity.OrderNumber,
            Status       = entity.Status,
            ArchivedAt   = entity.ArchivedAt,

            // decompress byte[] → original object
            Payload = _compression.DecompressObject<OrderPayload>(entity.Payload_Compressed),
            History = _compression.DecompressObject<List<OrderHistoryEntry>>(entity.History_Compressed),
        };
    }
}

Register everything

// EFCore module
context.Services
    .AddTransient<IArchivingCriteria<Order>,          CompletedOrderCriteria>()
    .AddTransient<IArchiveMapper<Order, OrderArchive>, OrderToArchiveMapper>()
    .AddTransient<IArchivingService<Order, OrderArchive>, OrderArchivingService>();

Configure<ArchivingOptions>(opts =>
{
    opts.AgeMonths     = 6;
    opts.BatchSize     = 100;
    opts.IsMultiTenant = true;
});

// Application / Host module
context.AddBackgroundWorker<
    ArchivingBackgroundWorker<Order, OrderArchive,
                              IArchivingService<Order, OrderArchive>>>();

---

Design decisions

Why AsyncPeriodicBackgroundWorkerBase and not an EF Core Interceptor?

SaveChangesInterceptor fires on every write and drags archive logic into normal business
transactions. Archiving is a background batch operation — a periodic worker is the right fit.

Why physical move and not IDataFilter / IHasIsArchived?

IDataFilter is for soft-archive (records stay in the same table, filtered out from queries).
Physical archiving moves and deletes — a filter can't help with that. Both are valid strategies;
this module targets the physical-move case where you need the hot table to actually shrink.

Why not Table-Per-Hierarchy or table splitting?

TPH keeps everything in one table. Table splitting requires shared primary keys and the same
database — neither works for routing cold data to a separate store.

Multi-tenancy

The worker iterates all tenants via ITenantRepository and wraps each run in
ICurrentTenant.Change(tenantId). ABP's IConnectionStringResolver takes it from there and
routes both DbContexts to the right per-tenant databases automatically.

Additional context

Questions for the team

1. Does this belong in the core framework or as a community NuGet package?
2. Any naming preferences for the interfaces?
3. Is AsyncPeriodicBackgroundWorkerBase the right primitive here, or should this hook into
   ABP's IBackgroundJob / Hangfire / Quartz pipeline instead?

Open ideas / future directions

One thing worth discussing is the persistence strategy. Right now the module assumes a separate
archive database, but there's no reason it has to be locked to that. Two approaches make sense:

Same live database — archive tables live in the same DB as the source tables, just a different
schema (e.g. archive.Orders vs dbo.Orders). Simpler to set up, no cross-DB transaction risk,
and still keeps the hot tables lean. Good fit for small/medium scale or when a second DB is
overkill.

Separate archive database — what the current implementation does. The archive DbContext gets
its own connection string, completely isolated from the live DB. Better for large scale where you
want independent backup cycles, different retention policies, or hardware tiering.

The module could support both by making the persistence strategy configurable — either pointing
ArchiveDbContextBase at the same connection string or a different one. The rest of the
module (background worker, compression, multi-tenancy) stays identical either way.

Would be good to hear if there are other storage strategies the team thinks are worth supporting
(e.g. blob storage, NoSQL).

---