Container memory allocation modeling

container-memory-allocation-modeling

License: restricted — no clear open-source license detected for the underlying tool; verify licensing before commercial use or redistribution. <!-- asb-license-banner -->

Summary

Model and validate the maximum worker count for parallel task execution in a containerized application by applying a documented memory-sizing formula and verifying it against known test cases. This skill is essential for tuning online (Redis/RQ-based) deployments to avoid memory exhaustion while maximizing throughput.

When to use

When configuring a multi-worker online deployment of a containerized Streamlit application (e.g., OpenMS workflows using RQ workers), you need to determine how many RQ worker processes can safely run in a single container given available RAM. Apply this skill before spinning up production containers to prevent out-of-memory crashes and ensure predictable scaling.

When NOT to use

• Offline/local deployments using multiprocessing.Process instead of RQ — the formula is specific to online Docker mode with Redis queue and should not be applied to offline deployments, which do not use the same memory model.
• Single-worker or development environments — if you are running a single Streamlit instance without parallel workers, memory allocation is not determined by this formula.
• Non-OpenMS or non-RQ-based task queues — the formula is calibrated for RQ worker memory footprint (1.5GB per worker) and may not apply to Celery or other queue systems.

Inputs

• available_memory (GB): float — total RAM available in the target Docker container
• reference_test_case (optional): known memory-to-worker mapping (e.g., 8GB → 4 workers) for validation

Outputs

• max_workers: integer — maximum safe RQ worker process count for the container
• verification_report: document or log — test results and formula validation summary

How to apply

Extract the worker-count memory formula from deployment documentation: max_workers = (available_memory - 2GB) / 1.5GB. Implement this as a Python function that accepts available_memory in GB as a floating-point input and returns the maximum safe worker count. Validate the formula against the reference example: 8GB container should yield 4 workers (verified by computing (8 - 2) / 1.5 = 4.0). Test boundary conditions: memory ≤ 2GB should clamp to 0 workers (no workers can run), 2GB exactly yields 0, and values like 5.5GB yield 2.4 (round or floor as per deployment policy). Generate a verification report documenting all test cases and confirming the 8GB example produces exactly 4 workers before applying the formula to your target container size.

Related tools

• RQ (Redis Queue) (Task queue in which workers execute enqueued OpenMS workflow jobs; memory footprint (1.5GB per worker) is the basis of the sizing formula) — https://github.com/OpenMS/streamlit-template
• Redis (In-process message broker and job store for RQ; runs locally within the container; memory overhead (~2GB reserved) is subtracted in the formula) — https://github.com/OpenMS/streamlit-template
• Docker (Container runtime; available_memory is the total RAM allocated to the container via docker run -m or docker-compose memory limit) — https://github.com/OpenMS/streamlit-template
• Python (Language for implementing the formula calculation function and validation test suite)

Examples

python -c "available_memory = 8.0; max_workers = int((available_memory - 2) / 1.5); print(f'Max workers for {available_memory}GB: {max_workers}')"

Evaluation signals

• Formula implementation correctly computes (available_memory - 2GB) / 1.5GB with floating-point arithmetic and produces 4.0 for the 8GB reference example.
• Edge-case test passes: memory values ≤ 2GB return 0 workers (no negative workers); memory = 2GB exactly returns 0; memory = 5.5GB returns 2.4 or floor(2.4) = 2 depending on rounding policy.
• Verification report documents formula source, implementation, test cases, and confirmation that 8GB example produces exactly 4 workers.
• Formula result, when applied to actual container memory limit and multiplied by 1.5GB per worker, does not exceed available_memory minus 2GB reserved headroom.
• Deployed container with computed max_workers value does not experience out-of-memory errors or worker crashes during sustained parallel workflow execution.

Limitations

• Formula assumes each RQ worker process consumes approximately 1.5GB of RAM under typical OpenMS workflow loads; jobs with exceptionally large in-memory data structures may require manual downward adjustment.
• The 2GB reserved overhead includes Redis, Streamlit, system processes, and margin; this is a static offset and does not scale with container size, so the formula may over-allocate workers on very large containers (e.g., 64GB+).
• The formula is calibrated for single-container online deployments as documented; distributed multi-container deployments or heterogeneous worker pools with different memory profiles require separate sizing analysis.
• No dynamic recomputation occurs if container memory limit is changed at runtime; the formula must be re-evaluated and workers restarted to adapt to new memory constraints.

Evidence

• [other] max_workers formula and 8GB reference example: "max_workers = (available_memory - 2GB) / 1.5GB with stated example of 8GB container → max 4 workers"
• [readme] Online deployment mode and RQ worker memory footprint context: "All components run within the same Docker container, ensuring identical environments for the web app and worker processes."
• [readme] Redis memory overhead in single-container architecture: "Why Single Container? - No networking issues: All communication via localhost"
• [other] RQ as task queue technology choice for online mode: "Task Queue | RQ (Redis Queue) | Lightweight, Python-native, simpler than Celery"
• [other] Multiprocessing used in offline mode (not applicable to formula): "The existing multiprocessing system remains completely unchanged for offline/local deployments"