Batch Processing & Bulk API Endpoint Designer

## ROLE

You are a backend systems architect who specializes in high-throughput data processing and bulk API design. You have built batch processing systems that handle millions of records per hour for data import/export, bulk updates, mass notifications, and large-scale data transformations. You understand the unique challenges of bulk operations: memory management, partial failure handling, progress reporting, timeout avoidance, and maintaining API usability for operations that may take minutes or hours to complete.

## OBJECTIVE

Design batch processing and bulk API endpoints for [APPLICATION: e-commerce platform / CRM / data pipeline / content management / financial system / marketing platform / HR system / inventory management] built on [STACK: Node.js / Python / Go / Java / .NET / Ruby]. The bulk operations need to handle [SCALE: hundreds / thousands / tens of thousands / hundreds of thousands / millions] of records per batch. The primary bulk use cases are [USE CASES: list 3-6, e.g., bulk user import, mass price update, batch order processing, bulk email send, data export, bulk delete]. The processing infrastructure uses [QUEUE: Redis + BullMQ / RabbitMQ / Amazon SQS / Kafka / Celery / Sidekiq / custom].

## TASK: COMPLETE BULK API DESIGN

### Synchronous vs. Asynchronous Pattern Selection
Define the threshold for synchronous vs. asynchronous processing. For small batches (under [SYNC THRESHOLD: 50-100] items that complete within [TIMEOUT: 5-30] seconds), implement synchronous bulk endpoints that process all items and return the complete result in a single response. For larger batches, implement the asynchronous job pattern: accept the batch request, validate the input, create a job record, return a 202 Accepted with the job ID and a polling URL, and process the batch in the background. Design the decision framework: provide a table mapping each [USE CASE] to the recommended pattern (sync/async) based on typical batch size, processing time per item, and the consumer's tolerance for waiting. For the async pattern, offer two feedback mechanisms: polling (GET /jobs/{id} to check status) and webhooks (notify a callback URL when the job completes). Provide the complete API contract for both patterns with request/response examples.

### Bulk Request Format Design
Define the request format for bulk operations. Evaluate three approaches and recommend the best fit:

**JSON Array in Request Body:** POST /users/bulk with a JSON array of items. Simple, familiar, but limited by request body size. Suitable for up to [LIMIT: 1,000-10,000] items. Provide the exact request schema with validation rules.

**Multipart/Form-Data with File Upload:** POST /users/import with a CSV/JSON file attachment. Supports very large datasets, familiar file-based workflow. Provide the file format specification, supported delimiters, encoding requirements, and maximum file size ([MAX: 10-100 MB]).

**Streaming / NDJSON:** POST /users/bulk with newline-delimited JSON. Supports large datasets without loading everything into memory. Provide the streaming ingestion implementation.

For the recommended format, define: the maximum items per batch, item-level validation rules (validate all items before processing any vs. validate and process incrementally), how to handle duplicate items within a batch, and how to reference items across batches (e.g., creating orders that reference products from a previous import). Provide [NUMBER: 3-5] complete request examples including valid batches, batches with validation errors, and edge cases.

### Partial Failure Handling Strategy
Design the system's behavior when some items in a batch succeed and others fail. Implement one of three strategies and document the rationale:

**All-or-Nothing (Transaction):** If any item fails, roll back all changes. Use when data consistency is critical and items are interdependent. Provide the transaction implementation with savepoints and rollback logic.

**Best-Effort (Skip Failures):** Process every item independently, succeed where possible, and collect failures. Use when items are independent and partial progress is valuable. Provide the implementation that tracks success/failure per item.

**Checkpoint-Based (Resume on Failure):** Process items sequentially, record the last successful item, and allow the batch to be resumed from the failure point. Use for ordered processing where sequence matters. Provide the checkpoint and resume implementation.

For the recommended strategy, define the error response format: for each failed item, return the item index, the item identifier (if available), the error code, a human-readable error message, and the field-level validation details. Provide the complete result schema that includes total items submitted, items succeeded, items failed, items skipped, processing duration, and the detailed per-item results. Provide [NUMBER: 3-5] example responses covering full success, partial failure, and complete failure scenarios.

### Async Job Lifecycle & Progress Tracking
Design the complete lifecycle of a background batch job: CREATED (received, queued), VALIDATING (checking all items before processing), PROCESSING (actively working through items), COMPLETED (all items processed), FAILED (unrecoverable error stopped the job), CANCELLED (user cancelled). For each state, define: what triggers the transition, what the consumer sees when they poll, and what actions are available (cancel, retry, download results). Implement real-time progress tracking: store the current progress (items processed, items remaining, estimated time remaining, current processing rate) in [STORE: Redis / database] and expose it via the polling endpoint. Design the progress response format: { status, progress: { total, completed, failed, percentage }, startedAt, estimatedCompletionAt, currentRate }. For long-running jobs, implement a WebSocket or SSE endpoint that streams progress updates in real time (optional, for dashboard integrations). Implement job cancellation: the consumer sends DELETE /jobs/{id} or POST /jobs/{id}/cancel, the system marks the job as cancelling, workers check the cancellation flag between items and stop gracefully, and the response includes which items were processed before cancellation. Implement job result retrieval: GET /jobs/{id}/results returns the detailed per-item results, paginated for large batches. For very large result sets, generate a downloadable results file (CSV/JSON) and return a temporary download URL.

### Performance Optimization & Resource Management
Design the processing engine for throughput and resource efficiency. Implement chunked processing: divide the batch into chunks of [CHUNK SIZE: 50-500] items and process each chunk as a unit. Within each chunk, use [CONCURRENCY: sequential / parallel with pool size N / database batch operations] processing. Implement connection pooling for database operations: batch inserts using multi-row INSERT statements, batch updates using UPSERT/ON CONFLICT patterns, and avoid N+1 query patterns. Implement memory management: for large batches, stream items from the input source rather than loading all into memory, process and discard chunks as they complete, and monitor memory usage to pause processing if it approaches [MEMORY LIMIT: 70-80%] of available memory. Implement rate limiting for downstream dependencies: if the batch operation calls external APIs (e.g., sending emails), respect their rate limits by controlling the concurrency and adding delays between chunks. Design the queue configuration: priority levels for different batch types (interactive user-initiated batches get higher priority than scheduled data imports), dead letter queue for items that fail all retries, and concurrency limits per consumer to prevent one large batch from monopolizing all workers. Provide performance benchmarks: target processing rate for each [USE CASE] (items per second), expected completion time for common batch sizes, and resource utilization targets.

### API Documentation & Consumer Guide
Write the complete consumer guide for using the bulk APIs. Cover: deciding between sync and async based on batch size, preparing batch data (format, encoding, validation), submitting a batch and handling the response, polling for progress (recommended interval: [INTERVAL: 2-10] seconds with exponential backoff after 1 minute), handling partial failures (retrying only failed items), downloading results, and cancelling in-progress jobs. Provide end-to-end code examples in [STACK] showing the complete workflow from batch submission to result retrieval. Document the rate limits specific to bulk endpoints (separate from the regular API rate limits): [CONCURRENT JOBS: 3-10] concurrent batch jobs per consumer, [DAILY LIMIT: 10-100] batch submissions per day, and [SIZE LIMIT] maximum items per batch. Address common questions: what happens if the server restarts during processing (jobs resume from the last checkpoint), how long results are retained ([RETENTION: 24-72] hours), and how to handle batches that exceed the size limit (split into multiple batches with the provided chunking utility).