Deploy Red Hat Quay on ROSA HCP with AWS S3, RDS, and ElastiCache (CLI)

This guide deploys Red Hat Quay on Red Hat OpenShift Service on AWS (ROSA) with Hosted Control Planes (HCP) using only the oc and aws CLIs. It uses:

• Amazon S3 for registry storage with IRSA and the Quay STSS3Storage backend (no long-lived S3 access keys).
• Amazon RDS for PostgreSQL for the Quay metadata database (password authentication via DB_URI).
• Amazon ElastiCache for Redis for Quay’s Redis workloads (in-VPC connectivity; optional AUTH token).

The RDS networking and bootstrap steps follow the same pattern as Connect to RDS database with STS from ROSA . ElastiCache is created in the same database VPC with a matching security-group pattern.

References

• Deploying the Red Hat Quay registry | Red Hat Quay 3.16
• Using an external PostgreSQL database
• Using an external Redis database
• Configure AWS S3 cloud storage with STS
• S3 IAM bucket policy for Quay Enterprise (Solution 3680151) — use with the bucket policy in §5.4 to avoid 403 Forbidden on S3
• Troubleshooting the QuayRegistry CR

Prerequisites

• A ROSA HCP cluster with cluster-admin (or equivalent) access
• OpenShift CLI (oc) logged in
• AWS CLI configured with permissions to create VPC, RDS, ElastiCache, S3, IAM roles/policies
• jq installed (required for §1.2–§1.3 tag merge and helpers)
• Optional: ROSA CLI (rosa) if you prefer rosa describe cluster for region/OIDC (alternatives below use only oc)

1. Names and environment variables

Set names used throughout this guide:

This CLUSTER_NAME segment is used in AWS resource names (RDS, subnet groups, IAM policy name, database VPC Name tag quay-${CLUSTER_NAME}-Storage, etc.). Override with export CLUSTER_NAME=... if your API server hostname does not match the name you want for those resources.

Quay app service account (used in the IAM trust policy) is typically:

Confirm after install with oc get sa -n "${QUAY_NAMESPACE}" and adjust the trust policy if your operator version uses a different name.

1.1 Region, account, OIDC (ROSA HCP)

1.2 AWS resource tags (JSON)

Set QUAY_AWS_TAGS_JSON to a JSON array of {"Key":"…","Value":"…"} objects (same information you might keep in a Terraform map). Customize keys/values for your organization.

1.3 Tag helper functions

Define helpers once in the same shell session you use for §2 through §5. They read QUAY_AWS_TAGS_JSON from §1.2.

Usage pattern

• EC2 create-tags: load pairs into an array, then pass --tags "${ARRAY[@]}". For example, after §3.1 creates the database VPC, VPC_DB is set and you can run:

• RDS / ElastiCache / IAM: append --tags "${_QUAY_AWS_TAG_PAIRS[@]}" to create-db-subnet-group, create-db-instance, create-cache-subnet-group, create-cache-cluster, create-policy, and create-role (see §2, §3, §5).

• S3: use ${QUAY_AWS_S3_TAGGING_JSON} with aws s3api put-bucket-tagging --tagging (§2).

1.4 ROSA VPC and CIDR (for peering and security groups)

Quay on ROSA must reach private RDS and ElastiCache in VPC_DB. You need private IP connectivity between the ROSA cluster VPC and VPC_DB—typically VPC peering (§3.2) or AWS Transit Gateway ( What is Transit Gateway? ). A NAT gateway public IP is not used for these rules.

1.5 Passwords

Amazon RDS MasterUserPassword accepts only printable ASCII and must not contain /, @, ", or space ( CreateDBInstance ). Passwords from openssl rand -base64 often include / (and sometimes @), which triggers InvalidParameterValue.

Use hex secrets for RDS and for the Quay DB user (also keeps DB_URI free of @, :, #, and % in the password):

2. Create S3 bucket for Quay

Complete §1.2 and §1.3 first so QUAY_AWS_S3_TAGGING_JSON is set.

Requires QUAY_AWS_S3_TAGGING_JSON from §1.3. Optional (recommended for production): enable default encryption and block public access using your organization’s standards.

3. Database VPC, RDS, and ElastiCache

Create a dedicated VPC_DB for RDS and ElastiCache, peer it to the ROSA VPC (VPC_ROSA from §1.4), then allow private traffic from ${ROSA_VPC_CIDR} on the RDS and Redis security groups.

Run §1.2 and §1.3 before §3 so quay_aws_tags_to_cli_pairs and load_quay_aws_tag_pairs are defined. Each subsection below runs load_quay_aws_tag_pairs to refresh _QUAY_AWS_TAG_PAIRS for that shell.

3.1 VPC and subnets

The database VPC (VPC_DB) is tagged with Name=quay-${CLUSTER_NAME}-Storage (in addition to §1.2 tags).

3.2 VPC peering (ROSA VPC ↔ database VPC)

Run after §1.4 and §3.1 so VPC_ROSA, ROSA_VPC_CIDR, VPC_DB, and VPC_DB_CIDR are set.

1. Create and accept the peering connection (same AWS account):

2. Optional — DNS across the peering (helps resolve RDS/ElastiCache endpoints from the cluster):

3. Routes — add a route to VPC_DB in every route table used by the ROSA VPC (private subnets, worker subnets), and a route to ROSA_VPC_CIDR in every route table in VPC_DB. Use replace with your route table IDs if you manage them explicitly; this loop targets all route tables in each VPC:

   If a route already exists, create-route may fail—ignore or use delete-route + create-route to update.

4. Wait until the peering status is active (poll until Status.Code is active):

3.3 RDS subnet group

3.4 Create RDS PostgreSQL

Private RDS without a public IP (reachable from ROSA over the peering):

Wait until the instance is available:

3.5 Allow ROSA to reach RDS

Allow TCP 5432 from the ROSA VPC CIDR (private traffic over the peering / TGW):

For tighter security, use a worker subnet CIDR or security group reference instead of the whole VPC CIDR.

3.6 ElastiCache subnet group and security group

3.7 Create Redis cluster

Single-node example (adjust node type for production). Leave REDIS_AUTH_TOKEN unset for the simplest path; enabling AUTH often requires transit encryption on the replication group—see ElastiCache in-transit encryption .

3.8 Endpoints

4. Prepare PostgreSQL for Quay

Create a dedicated database user and database for Quay, and enable extensions required by Red Hat Quay ( external PostgreSQL ).

1. Generate a password for the Quay DB user and keep it for §7 (DB_URI). Use hex (same rule as §1.5—avoid /, @, ", and space for consistency and safe DB_URI):

2. Run a short-lived client pod on the cluster:

3. Inside the pod, connect as the RDS master user (postgres):

4. In psql, run (replace quay if you changed QUAY_DB_USER / QUAY_DB_NAME). Use the same password as QUAY_DB_PASSWORD on your workstation—you can echo it in another terminal and paste it into the SQL.

   Create the user, then create the database without OWNER (owned by postgres initially), then ALTER DATABASE … OWNER TO:

   Why not CREATE DATABASE quay OWNER quay? PostgreSQL only allows that if the session user is a superuser or can SET ROLE to the owner role ( CREATE DATABASE ). On Amazon RDS, the master user is not always equivalent to a true superuser for this check, so you may see ERROR: must be able to SET ROLE "quay". Creating the database first (default owner postgres), then ALTER DATABASE quay OWNER TO quay, avoids that.

5. exit the pod shell.

5. IAM role for Quay (S3 via IRSA / STSS3Storage)

The Quay application pod uses AWS credentials from the service account (IRSA). You need:

1. An identity-based IAM policy attached to the Quay IRSA role (§5.1–5.3).
2. A resource-based S3 bucket policy on the registry bucket that allows that same IAM role as Principal (§5.4).

Red Hat documents this combination in S3 IAM bucket policy for Quay Enterprise (Solution 3680151) , which addresses errors such as Invalid storage configuration, S3ResponseError: 403 Forbidden, and related S3 access failures when Quay uses IAM roles for storage.

5.1 S3 policy document

5.2 Trust policy (OIDC → Quay service account)

5.3 Create role and attach S3 policy

5.4 S3 bucket policy (resource-based)

Apply a bucket policy that allows the Quay IRSA role ARN to access the bucket. This follows the pattern in S3 IAM bucket policy for Quay Enterprise (Solution 3680151) : set Principal.AWS to the role Quay assumes (the same QUAY_IRSA_ROLE_ARN from §5.3).

6. Install the Red Hat Quay Operator (CLI)

This guide installs the operator in openshift-operators with an AllNamespaces OperatorGroup (the default global pattern on OpenShift). That mode is required for operator-managed monitoring on the QuayRegistry ( operator reconcile ). Use channel: stable-3.16 as below; confirm the channel exists with oc get packagemanifest quay-operator -n openshift-marketplace -o jsonpath='{.status.channels[*].name}{"\n"}'.

6.1 Ensure a global OperatorGroup exists

Default OpenShift clusters already have an AllNamespaces group in openshift-operators:

You should see a group whose spec is empty or has no targetNamespaces (AllNamespaces). If your cluster has no such group, create one (adjust the name if it conflicts):

6.2 Create the Quay registry namespace

For QuayRegistry, secrets, and workloads (same as the rest of this guide):

6.3 Subscribe in openshift-operators

Install the Red Hat Quay Operator from openshift-operators (not from ${QUAY_NAMESPACE}):

6.4 Wait for the CSV

Stop when the Red Hat Quay CSV shows Succeeded.

6.5 Confirm the operator deployment

Name/version may differ:

7. Build config.yaml and secrets

Set DB_URI (URL-encode special characters in the password if needed). Example:

If you enabled Redis AUTH, add password: ... under both BUILDLOGS_REDIS and USER_EVENTS_REDIS (use the same token as REDIS_AUTH_TOKEN).

Create the config bundle secret:

Remove config.yaml from shared machines after applying, or create the secret without writing a world-readable file (e.g. process substitution) per your security policy.

8. Create the QuayRegistry

After §6 (AllNamespaces operator install), set monitoring: managed: true so the operator can create ServiceMonitor resources and related monitoring integration (requires User Workload Monitoring or equivalent, per Red Hat Quay documentation).

9. Annotate the Quay app service account (IRSA)

After the operator creates the Quay deployment, confirm the service account name:

Annotate the Quay application service account (usually ${QUAY_REGISTRY_NAME}-quay-app):

Restart workloads that use the Quay application service account so they pick up the annotation (deployment names follow ${QUAY_REGISTRY_NAME}-quay-*, e.g. example-registry-quay-app and example-registry-quay-mirror):

With mirror: managed: true (§8), ${QUAY_REGISTRY_NAME}-quay-mirror is present (e.g. example-registry-quay-mirror). If you disabled mirror, skip the second command.

If the trust policy sub does not match the actual SA, edit quay-trust-policy.json, run aws iam update-assume-role-policy, then verify again.

10. Verification

Look for ComponentsCreationSuccess in Events.

Retrieve the registry route:

Optional: push an image to confirm S3 and the full stack.

11. Create the first user (API)

With FEATURE_USER_INITIALIZE: true and SUPER_USERS set in config.yaml, use the Red Hat procedure:

Using the API to create the first user

Example (replace host and payload per the documentation):

Follow the official doc for the exact JSON and token fields required by your Quay version.

12. Cleanup (optional)

Remove OpenShift resources:

Do not delete the default global OperatorGroup in openshift-operators unless your cluster policy allows it.

12.1 AWS cleanup (CLI)

Use the same AWS_REGION, CLUSTER_NAME, QUAY_S3_BUCKET, and VPC_DB values as when you created resources (re-export from §1–§3 if needed). Set AWS_ACCOUNT_ID if unset: export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text). Do not delete the ROSA VPC (VPC_ROSA); only tear down VPC_DB and resources this guide created.

1. S3 — remove objects and the bucket (bucket policy is removed with the bucket):

2. IAM — detach the policy, delete the role, delete the customer managed policy

Policy and role names match §5.1 and §5.3 (${CLUSTER_NAME}-quay-s3-policy, ${CLUSTER_NAME}-quay-irsa):

If delete-policy fails because another principal still references the policy, list attachments with aws iam list-entities-for-policy --policy-arn "${S3_POLICY_ARN}" and detach them first.

3. ElastiCache — delete the cluster, then the subnet group

4. RDS — delete the instance, then the DB subnet group

To retain a snapshot instead of --skip-final-snapshot, use --final-db-snapshot-identifier per delete-db-instance .

5. VPC peering — delete the connection (saves VPC_PEERING_ID from §3.2, or discover it)

If VPC_PEERING_ID is None, swap requester / accepter filters (or list all peerings for ${VPC_DB} and pick the correct ID).

6. Security groups — delete the Redis SG (optional: revoke RDS ingress you added)

The Redis security group name is quay-redis-${CLUSTER_NAME}. If RDS used the default security group for VPC_DB, revoke the rule you added in §3.5 instead of deleting that group:

7. Subnets and VPC — delete subnets, then VPC_DB

Use the subnet IDs from §3.1 (SUBNET_A, SUBNET_B, SUBNET_C), or list them:

If subnet delete fails, dependencies remain (ENIs, load balancers, etc.); resolve those in the VPC console or with describe-network-interfaces --filters Name=subnet-id,Values=....

Suggested order summary

1. OpenShift resources (above).
2. S3 bucket (12.1 step 1).
3. IAM role and policy (step 2).
4. ElastiCache (step 3).
5. RDS (step 4).
6. VPC peering (step 5).
7. Security groups (step 6).
8. Subnets and VPC_DB (step 7).

13. Troubleshooting

• Troubleshooting the QuayRegistry CR
• IRSA / S3 403 Forbidden or invalid storage configuration: Confirm the IAM role policy (§5.1) and the S3 bucket policy (§5.4) both allow the Quay IRSA role for arn:aws:s3:::${QUAY_S3_BUCKET} and arn:aws:s3:::${QUAY_S3_BUCKET}/*. See S3 IAM bucket policy for Quay Enterprise (Solution 3680151) . Verify eks.amazonaws.com/role-arn on the Quay app SA, trust policy sub and aud, and that Principal.AWS in the bucket policy matches QUAY_IRSA_ROLE_ARN exactly.
• Database connection errors: Verify VPC peering (§3.2) routes and security group allows 5432 from ${ROSA_VPC_CIDR} (§3.5), DB_URI credentials, and sslmode (often require for RDS).
• example-registry-quay-app-upgrade-* pod CrashLoopBackOff / psycopg2.OperationalError: The upgrade (migration) Job uses the same configBundleSecret as the registry. Two common RDS issues show up together in logs:
  • FATAL: no pg_hba.conf entry for host "…", user "quay", database "quay", no encryption — The client connected without TLS. Amazon RDS for PostgreSQL expects SSL; keep DB_CONNECTION_ARGS with sslmode: require (see §7). If that block is missing or the secret was created before you added it, update config.yaml, re-create the config bundle secret, and let the operator reconcile (or delete the failed upgrade Job so it is recreated). With force_ssl enabled on the instance, non-SSL attempts are rejected and pg_hba can report no encryption.
  • FATAL: password authentication failed for user "quay" — The password in DB_URI does not match the quay role in PostgreSQL (typo when pasting in §4, different QUAY_DB_PASSWORD than used in CREATE USER, or shell/heredoc altered the password). Align the database role with DB_URI: connect to RDS with psql as the master user (same host/port/SSL as §4), then set the role password to exactly the string used in DB_URI (between : and @ in postgresql://quay:PASSWORD@…—URL-decode if you encoded special characters): Use your real password in place of the placeholder; if the password contains a single quote, double it ('' inside the string). Update config.yaml / configBundleSecret if you change DB_URI instead, then let the operator reconcile or delete the failed quay-app-upgrade Job so it runs again.
  • Ensure private connectivity from the ROSA VPC to VPC_DB (§3.2) and that the RDS security group allows ${ROSA_VPC_CIDR} on 5432 (§3.5).
• CREATE DATABASE quay OWNER quay → ERROR: must be able to SET ROLE "quay": Use CREATE DATABASE quay; then ALTER DATABASE quay OWNER TO quay; as in §4 (RDS master user is not always treated like a full superuser for the single-step OWNER clause).
• Redis / BUILDLOGS_REDIS … context deadline exceeded: The client timed out connecting to Redis (wrong host/port, missing §3.2 routes, or SG). Confirm VPC peering (or TGW) and ${ROSA_VPC_CIDR} on TCP 6379 (§3.6). Confirm host in BUILDLOGS_REDIS / USER_EVENTS_REDIS is the primary endpoint from aws elasticache describe-cache-clusters. If you enabled Redis AUTH, set password in both Redis blocks. If you enabled in-transit encryption, set ssl: true in config and follow ElastiCache in-transit encryption (port is still typically 6379 for the cluster endpoint).
• RolloutBlocked / MonitoringComponentDependencyError (Monitoring is only supported in AllNamespaces mode): Confirm §6—the quay-operator Subscription must be in openshift-operators with a Succeeded CSV, and monitoring: managed: true in §8 must match that install mode. Reapply the QuayRegistry after fixing the operator install.