Introduction
This guide provides step-by-step instructions to deploy the W&B Platform in air-gapped, fully disconnected, or restricted network customer-managed environments. Air-gapped deployments are common in:- Secure government facilities
- Financial institutions with strict network isolation
- Healthcare organizations with compliance requirements
- Industrial control systems (ICS) environments
- Research facilities with classified networks
Prerequisites
Before starting, ensure your air-gapped environment meets the following requirements.Version requirements
| Software | Minimum version |
|---|---|
| Kubernetes | v1.32 or newer (Supported Kubernetes versions) |
| Helm | v3.x |
| MySQL | v8.0.x is required, v8.0.32 or newer; v8.0.44 or newer is recommended. Aurora MySQL 3.x releases, must be v3.05.2 or newer |
| Redis | v7.x |
SSL/TLS requirements
W&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress/load balancer. The W&B Server application does not terminate SSL or TLS connections. Important: W&B does not support self-signed certificates and custom CAs. Using self-signed certificates will cause challenges for users and is not supported. If possible, using a service like Let’s Encrypt is a great way to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you. If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and side car containers.Hardware requirements
CPU Architecture: W&B runs on Intel (x86) CPU architecture only. ARM is not supported. Sizing: For CPU, memory, and disk sizing recommendations for Kubernetes nodes and MySQL, see the Sizing section in the reference architecture. Requirements vary based on whether you’re running Models, Weave, or both.MySQL database
W&B requires an external MySQL database. For production, W&B strongly recommends using managed database services: Managed database services provide automated backups, monitoring, high availability, patching, and reduce operational overhead. See the reference architecture for complete MySQL requirements, including sizing recommendations and configuration parameters. For database creation SQL, see the bare-metal guide. For questions about your deployment’s database configuration, contact support or your AISE. For MySQL configuration parameters for self-managed instances, see the reference architecture MySQL configuration section.Redis
W&B depends on a single-node Redis 7.x deployment used by W&B’s components for job queuing and data caching. For convenience during testing and development of proofs of concept, W&B Self-Managed includes a local Redis deployment that is not appropriate for production deployments. For production deployments, W&B can connect to a Redis instance in the following environments:- AWS Elasticache
- Google Cloud Memory Store
- Azure Cache for Redis
- Redis deployment hosted in your cloud or on-premise infrastructure
Object storage
W&B requires object storage with pre-signed URL and CORS support. Recommended storage providers:- Amazon S3: Object storage service offering industry-leading scalability, data availability, security, and performance.
- Google Cloud Storage: Managed service for storing unstructured data at scale.
- Azure Blob Storage: Cloud-based object storage solution for storing massive amounts of unstructured data.
- CoreWeave AI Object Storage: High-performance, S3-compatible object storage service optimized for AI workloads.
- Enterprise S3-compatible storage: MinIO Enterprise (AIStor), NetApp StorageGRID, or other enterprise-grade solutions
MinIO Open Source is in maintenance mode with no active development or pre-compiled binaries. For production deployments, W&B recommends using managed object storage services or enterprise S3-compatible solutions such as MinIO Enterprise (AIStor).
Air-gapped specific requirements
In addition to the standard requirements above, air-gapped deployments require:- Internal container registry: Access to a private container registry (Harbor, JFrog Artifactory, Nexus, etc.) with all required W&B images
- Internal Helm repository: Access to a private Helm chart repository with W&B Helm charts
- Image transfer capability: A method to transfer container images from an internet-connected system to your air-gapped registry
- License file: A valid W&B Enterprise license (contact your W&B account team)
Prepare your air-gapped environment
Step 1: Set up internal container registry
For a successful air-gapped deployment, all required container images must be available in your air-gapped container registry.You are responsible for tracking the W&B Operator’s requirements and maintaining your container registry with updated images regularly. For the most current list of required container images and versions, refer to the Helm chart, or contact W&B Support or your assigned W&B support engineer.
Core W&B component containers
The following core images are required:docker.io/wandb/controller- W&B Kubernetes Operatordocker.io/wandb/local- W&B application serverdocker.io/wandb/console- W&B management consoledocker.io/wandb/megabinary- W&B microservices (API, executor, glue, parquet)
Dependency containers
The following third-party dependency images are required:docker.io/bitnamilegacy/redis- Required for local Redis deployment during testing and development. For production Redis requirements, see the Redis section in Prerequisites.docker.io/otel/opentelemetry-collector-contrib- OpenTelemetry agent for collecting metrics and logsquay.io/prometheus/prometheus- Prometheus for metrics collectionquay.io/prometheus-operator/prometheus-config-reloader- Prometheus dependency
Get the complete image list
To extract the complete list of required images and versions from the Helm chart:-
On an internet-connected system, download the W&B Helm charts from the W&B Helm charts repository:
-
Inspect the
values.yamlfiles to identify all container images and their versions:Alternatively, use this command to extract just the repository names (without version tags):The list of repositories will look similar to the following:To get the specific version tags for each image, use the first command above (grep -E "repository:|tag:"), which will show both repository names and their corresponding version tags.
Transfer images to air-gapped registry
-
On an internet-connected system, pull and save all required images.
Use shell variables to manage versions consistently:Replace version numbers in the examples below with the actual versions from your Helm chart inspection in step 2 above. The versions shown here are examples and will become outdated.
-
Transfer the
.tarfiles to your air-gapped environment using your approved method (USB drive, secure file transfer, etc.). -
In your air-gapped environment, load and push images to your internal registry:
Step 2: Set up internal Helm chart repository
Along with the container images, ensure the following Helm charts are available in your internal Helm repository:-
On an internet-connected system, download the charts:
-
Transfer the
.tgzchart files to your air-gapped environment and upload them to your internal Helm repository according to your repository’s procedures. Theoperatorchart deploys the W&B Kubernetes Operator (Controller Manager). Theoperator-wandbchart deploys the W&B Platform using the values configured in the Custom Resource (CR).
Step 3: Configure Helm repository access
-
In your air-gapped environment, configure Helm to use your internal repository:
-
Verify the charts are available:
Deploy W&B in air-gapped environment
Step 4: Install the Kubernetes Operator
The W&B Kubernetes Operator (controller manager) manages the W&B platform components. To install it in an air-gapped environment, configure it to use your internal container registry.-
Create a
values.yamlfile with the following content:Replace the repository and tag with the actual versions you transferred to your internal registry in Step 1. The version shown here (1.13.3) is an example and will become outdated. -
Install the operator and Custom Resource Definition (CRD):
-
Verify the operator is running:
You should see the operator pod in a
Runningstate.
Step 5: Set up MySQL database
Before configuring the W&B Custom Resource, set up an external MySQL database. For production deployments, W&B strongly recommends using managed database services where available. However, if you are running your own MySQL instance, create the database and user: Create a database and a user with the following SQL commands. ReplaceSOME_PASSWORD with a secure password of your choice:
Step 6: Configure W&B Custom Resource
After installing the W&B Kubernetes Operator, configure the Custom Resource (CR) to point to your internal Helm repository and container registry. This configuration ensures the Kubernetes operator uses your internal registry and repository when deploying the required components of the W&B platform.The example configuration below includes image version tags that will become outdated. Replace all
tag: values with the actual versions you transferred to your internal registry in Step 1.wandb.yaml with the following content:
Replace all placeholder values (hostnames, passwords, tags, etc.) with your actual configuration values. The example above shows the most commonly used components.
settingsMigrationJobweave-tracefilestreamflat-runs-table
Step 7: Deploy the W&B platform
-
Apply the W&B Custom Resource to deploy the platform:
-
Monitor the deployment progress:
The deployment may take several minutes as the operator creates all necessary components.
OpenShift configuration
W&B fully supports deployment on air-gapped OpenShift Kubernetes clusters. OpenShift deployments require additional security context configurations due to OpenShift’s stricter security policies.OpenShift security context constraints
OpenShift uses Security Context Constraints (SCCs) to control pod permissions. By default, OpenShift assigns therestricted SCC to pods, which prevents running as root and requires specific user IDs.
Option 1: Use restricted SCC (recommended)
Configure W&B components to run with the restricted SCC by setting appropriate security contexts in your Custom Resource:Option 2: Create custom SCC (if required)
If your deployment requires capabilities not available in therestricted SCC, create a custom SCC:
-
Apply the SCC:
-
Bind the SCC to the W&B service accounts:
OpenShift routes
OpenShift uses Routes instead of standard Kubernetes Ingress. Configure W&B to use OpenShift Routes:OpenShift image pull configuration
If your OpenShift cluster uses an internal image registry with authentication:-
Create an image pull secret:
-
Reference the secret in your Custom Resource:
OpenShift complete example
Here’s a complete example CR for OpenShift air-gapped deployment:Replace all
tag: values in this example with the actual versions you transferred to your internal registry in Step 1. The versions shown are examples and will become outdated.Contact W&B Support or your assigned W&B support engineer for comprehensive OpenShift configuration examples tailored to your security requirements.
Verify your installation
After deploying W&B, verify the installation is working correctly: To verify the installation, W&B recommends using the W&B CLI. The verify command executes several tests that verify all components and configurations.This step assumes that the first admin user account is created with the browser.
- Install the W&B CLI:
- Log in to W&B:
- Verify the installation:
Additional air-gapped verification
For air-gapped deployments, also verify:-
Image pull: Confirm all pods successfully pulled images from your internal registry:
All images should point to your internal registry and all pods should be in
Runningstate. -
External connectivity: Verify W&B is not attempting external connections (it shouldn’t in air-gapped mode):
- License validation: Access the W&B console and verify your license is active.
Troubleshooting
Image pull errors
If pods fail to pull images:- Verify images exist in your internal registry
- Check image pull secret is correctly configured
- Verify network connectivity from Kubernetes nodes to registry
-
Check registry authentication credentials
OpenShift SCC errors
If pods fail with permission errors on OpenShift:Helm chart not found
If the operator cannot find the platform chart:- Verify the chart repository URL in the Custom Resource
- Check that the operator pod can reach your internal Helm repository
-
Verify the chart exists in your repository:
Frequently asked questions
Can I use a different ingress class?
Yes, configure your ingress class by modifying the ingress settings in your Custom Resource:How do I handle certificate bundles with multiple certificates?
Split the certificates into multiple entries in thecustomCACerts section:
How do I prevent automatic updates?
Configure the operator to not automatically update W&B:- Set
airgapped: truein the operator installation (this disables automatic update checks) - Control version updates by manually updating the
spec.chart.versionin your Custom Resource - Optionally, disable automatic updates from the W&B System Console
W&B strongly recommends customers with Self-Managed instances update their deployments with the latest release at minimum once per quarter to maintain support and receive the latest features, performance improvements, and fixes. W&B supports a major release for 12 months from its initial release date. Refer to Release policies and processes.
Does the deployment work with no connection to public repositories?
Yes. Whenairgapped: true is set in the operator configuration, the Kubernetes operator uses only your internal resources and does not attempt to connect to public repositories.
How do I update W&B in an air-gapped environment?
To update W&B:- Pull new container images on an internet-connected system
- Transfer images to your air-gapped registry
- Upload new Helm charts to your internal repository
-
Update the
spec.chart.versionand image tags in your Custom Resource - Apply the updated Custom Resource The operator will perform a rolling update of the W&B components.
Next steps
After successful deployment:- Configure user authentication: Set up SSO or other authentication methods
- Set up monitoring: Configure monitoring for your W&B instance and infrastructure
- Plan for updates: Review the Server upgrade process and establish an update cadence
- Configure backups: Establish backup procedures for your MySQL database
- Document your process: Create runbooks for your specific air-gapped update procedures
Getting help
If you encounter issues during deployment:- Review the Reference Architecture for infrastructure guidance
- Check the Operator guide for configuration details
- Contact W&B Support or your assigned W&B support engineer
- For OpenShift-specific issues, reference Red Hat OpenShift documentation