Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions administration/kosli_capture/getting_started.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
---
title: "Getting started with Kosli Capture"
sidebarTitle: "Getting started"
description: "Learn how to configure Kosli Capture for your organization"
tag: "BETA"
---

<Warning>
Kosli Capture is still in active development. Its capabilities and configuration format may change, and onboarding is done together with Kosli's Customer Success team.
Comment thread
gsavage marked this conversation as resolved.
</Warning>

Kosli Capture is a managed service that runs on Kosli's infrastructure and connects to your cloud platform to observe the resources deployed there. To get set up with Kosli Capture you need to grant permissions to Kosli's cloud account and enable Kosli Capture within your Kosli org.

## Overview

Getting started with Kosli Capture involves two steps.

<Steps>
<Step title="Prepare your environment">
Create an IAM role in your cloud accounts specifically for Kosli Capture.
</Step>
<Step title="Enable Kosli Capture">
Enable Kosli Capture for your Kosli org, and the regular snapshots will appear in Kosli.
</Step>
</Steps>

<Tabs>
<Tab title="AWS">

## Prepare your environment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement## Prepare your environment, ## Enable Kosli Capture and ## Excluding resources each appear twice on this page, once per tab. Headings inside <Tabs> still register as page headings, so the right-hand table of contents lists all three twice with nothing to say which entry is AWS and which is GCP, and the duplicated slugs get suffixed (#excluding-resources, #excluding-resources-1).

That second point has a caller: overview.md:44 links to #excluding-resources, which resolves to the AWS heading only — so a GCP reader following that link lands on ECS tag instructions.

Suggest either demoting the in-tab headings to ### with the cloud in the text (### Prepare your AWS environment), or keeping one set of ## sections and putting only the cloud-specific prose in tabs beneath each.

Fix this →


In order for Kosli Capture to reach into your AWS cloud, to discover your ECS clusters and Lambdas, you need to grant permission to Kosli to do so. This requires the creation of an IAM role that Kosli can assume; the role will exist within your AWS account.

To simplify this process, Kosli has created a CloudFormation template that contains a role with the minimum set of permissions needed by Kosli Capture. The role can be assumed by Kosli and is protected by an external Id; each organization within Kosli has its own external Id. The CloudFormation template can be downloaded from the Settings page for your organization within the Kosli UI.
Comment thread
gsavage marked this conversation as resolved.
Comment thread
gsavage marked this conversation as resolved.

If you would rather create the role yourself, see [Kosli Capture Security](/administration/kosli_capture/security) for the trust policy and the full set of permissions the role needs.

The CloudFormation template can be deployed within an AWS account, or can be attached to an AWS Organizational Unit (OU) as a StackSet; this latter option ensures the correct IAM permissions are rolled out to all AWS accounts within the OU.

When used, the CloudFormation template will send your AWS account id to Kosli, so that we are automatically notified that your account is ready to be included in Kosli Capture. Similarly, if you delete the CloudFormation stack we will be notified and know that the account is no longer to be included.

## Enable Kosli Capture

When you have created the IAM role, using the CloudFormation template, you can activate Kosli Capture within the Kosli user-interface. Kosli Capture runs on a five-minute schedule, and once you have enabled it, Kosli Capture will pick up your environment the next time it runs - you should see environments and snapshots appearing within a few minutes.

## Excluding resources

If there are resources you do not wish to include within a Kosli Capture Managed snapshot, for example an ECS cluster that you consider to be out of scope, you can add a tag to it indicating that the item should be skipped. Adding a tag with the name `kosli.capture` and the value `false` will ensure that Kosli Capture skips over that resource.
</Tab>
<Tab title="GCP">
## Prepare your environment
Comment on lines +50 to +51

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvementsecurity.md:110-112 carries a <Warning>GCP support is coming soon</Warning> inside its GCP tab; this tab has no such marker. A reader who opens the GCP tab here gets a two-step procedure written in the present tense ("The Terraform can be downloaded from the Settings page") for something that cannot be done yet, and will go hunting for a Settings download that isn't there.

Suggested change
<Tab title="GCP">
## Prepare your environment
<Tab>
<Warning>
GCP support is coming soon
</Warning>
## Prepare your environment

Fix this →


In order for Kosli Capture to reach into your GCP cloud, to discover your Kubernetes clusters, you need to grant permission to Kosli to do so. This requires the creation of an IAM role that Kosli can assume; the role will exist within your Google Cloud account.

To simplify this process, Kosli has created a Terraform configuration that contains a role with the minimum set of permissions needed by Kosli Capture. The role can be used by Kosli and is protected by ensuring the `principalSet` supplied contains the specific Kosli Capture role that was created for your organization. The Terraform can be downloaded from the Settings page for your organization within the Kosli UI.
Comment on lines +53 to +55

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement — the GCP tab describes the AWS mechanism with GCP words swapped in: there is no "IAM role that Kosli can assume" in a GCP project, and no "Google Cloud account" that holds it. security.md:121-133 gets this right — Kosli Capture exchanges its AWS credentials for a Google token and impersonates a service account admitted by a workload identity pool.

The mismatch matters because a GCP admin reading this page will go looking for a role to grant and a trust policy to write, neither of which exists, and principalSet is then dropped on them with no explanation of where it appears.

Suggested change
In order for Kosli Capture to reach into your GCP cloud, to discover your Kubernetes clusters, you need to grant permission to Kosli to do so. This requires the creation of an IAM role that Kosli can assume; the role will exist within your Google Cloud account.
To simplify this process, Kosli has created a Terraform configuration that contains a role with the minimum set of permissions needed by Kosli Capture. The role can be used by Kosli and is protected by ensuring the `principalSet` supplied contains the specific Kosli Capture role that was created for your organization. The Terraform can be downloaded from the Settings page for your organization within the Kosli UI.
In order for Kosli Capture to reach into your GCP project, to discover your Kubernetes clusters, you need to grant permission to Kosli to do so. This requires a service account in your project that Kosli Capture can impersonate through workload identity federation; the service account, and the workload identity pool that guards it, exist within your own project.
To simplify this process, Kosli has created a Terraform configuration that creates that service account, a custom role holding the minimum set of permissions needed by Kosli Capture, and a workload identity pool that accepts only the Kosli-side role created for your organization. The Terraform can be downloaded from the Settings page for your organization within the Kosli UI.

Fix this →


If you would rather create the role yourself, see [Kosli Capture Security](/administration/kosli_capture/security) for the trust policy and the full set of permissions the role needs.
When used, you need the CloudFormation template will send your AWS account id to Kosli, so that we are automatically notified that your account is ready to be included in Kosli Capture. Similarly, if you delete the CloudFormation stack we will be notified and know that the account is no longer to be included.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical — this sentence is an AWS leftover pasted into the GCP tab. It is ungrammatical ("you need the CloudFormation template will send"), and it contradicts security.md:135, which states plainly that "GCP has no equivalent of the CloudFormation phone-home feature, so the configuration emits the values Kosli needs as Terraform outputs." There is no CloudFormation stack in a GCP onboarding, so both the notification claim and the "delete the stack" claim are wrong here.

This is also the step that closes the GCP loop — telling the reader to hand kosli_capture_provider and kosli_capture_service_account to Kosli. As written they are told nothing has to be done.

Suggested change
When used, you need the CloudFormation template will send your AWS account id to Kosli, so that we are automatically notified that your account is ready to be included in Kosli Capture. Similarly, if you delete the CloudFormation stack we will be notified and know that the account is no longer to be included.
GCP has no equivalent of the CloudFormation "phone-home" feature, so the Terraform configuration emits the two values Kosli needs — the workload identity provider and the service account email — as outputs. Share them with Kosli once the deployment completes; you can read them at any time with `gcloud infra-manager deployments describe`. See [Kosli Capture Security](/administration/kosli_capture/security) for the deployment command.

Fix this →


## Enable Kosli Capture

When you have created the role and added the account details to Kosli, you can activate Kosli Capture within the Kosli user-interface. Kosli Capture runs on a five-minute schedule, and once you have enabled it, Kosli Capture will pick up your environment the next time it runs - you should see environments and snapshots appearing within a few minutes.

## Excluding resources

If there are resources you do not wish to include within a Kosli Capture Managed snapshot, for example a specific cluster that you consider to be out of scope, you can add a tag to it indicating that the item should be skipped. Adding a tag with the name `kosli.capture` and the value `false` will ensure that Kosli Capture skips over that resource.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improvement — "add a tag … with the name kosli.capture" carries over unchanged from the AWS tab, but on GCP "tag" is not one thing, and the key as written may not be applicable:

  • GCP labels — key syntax is lowercase letters, digits, underscores and hyphens; a . is not permitted, so kosli.capture cannot be a label key.
  • Resource Manager tags — a different feature, created at org/project level and bound to resources, where the key is <namespace>/<short-name>.
  • Kubernetes labels on the cluster's own objects — plausible, since the GCP path snapshots GKE namespaces and pods, and kosli.capture is a valid Kubernetes label key.

Which of the three it is changes what the reader has to go and do, and whether the exclusion applies to a cluster, a namespace or a pod. Worth naming explicitly — or saying that exclusion isn't supported for GCP yet, which is also a useful answer while the tab is marked coming soon.

Related: overview.md:44 links here as "filter out resources based on AWS tags", so the overview currently promises this only for AWS.

Fix this →

</Tab>
</Tabs>

59 changes: 59 additions & 0 deletions administration/kosli_capture/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
---
title: "Kosli Capture Managed Service"
sidebarTitle: "Kosli Capture"
description: "Learn how the Kosli Capture Managed Service snapshots your cloud environments from Kosli's infrastructure, with no software to install."
tag: "BETA"
---

<Warning>
Kosli Capture is still in active development. Its capabilities and configuration format may change, and onboarding is done together with Kosli's Customer Success team.
</Warning>

Kosli Capture is a managed service that runs on Kosli's infrastructure and connects to your cloud platform to observe the resources deployed there. You grant Kosli Capture a set of permissions, and it uses them to run a `kosli snapshot` every few minutes against the infrastructure you have allowed it to scan.

Kosli also supports reporting from your own cloud accounts by running the Kosli CLI on a schedule. Kosli Capture inverts this, with Kosli running the regular [snapshots](/getting_started/environments) so there is no software for you to install.

To set it up for your organization, see [Getting started with Kosli Capture](/administration/kosli_capture/getting_started).

## Overview

Kosli Capture connects to your cloud accounts using permissions that you manage. You configure Kosli Capture by activating it for different services, and Kosli Capture uses the permissions to regularly reach into your estate and record snapshots, sending the data into your Kosli organization. Kosli Capture uses details about your infrastructure, such as the name of an ECS cluster, to build environments within Kosli.

<Frame>
<img src="/images/administration/kosli-capture-overview.png" alt="Diagram showing Kosli Capture, inside Kosli, sending queries to and receiving snapshots from three customer cloud accounts, then passing the data to the Kosli API and database" />
</Frame>

## Security

The security of your cloud infrastructure is the primary driver behind the internal architecture of
Kosli Capture. You grant a read-only IAM role in your account, protected by an external ID that acts
as a shared secret between Kosli and you. On Kosli's side, each Kosli Capture job runs under a role
scoped to your organization alone, so a worker running for another customer cannot reach your cloud
account. Kosli Capture holds no customer data; snapshots go straight to Kosli through the same ingest
path as your existing pipelines. See [Kosli Capture Security](/administration/kosli_capture/security)
for the isolation model and the full list of permissions.

## Hands-off operation

Comment thread
gsavage marked this conversation as resolved.
Kosli Capture has been designed to operate with no on-going support from you. Once the initial security permissions have been created, Kosli Capture will continue to operate in a headless mode. As your cloud infrastructure changes over time, Kosli Capture will continue to find resources without you needing to do anything; your application teams do not need to take any action in order to onboard their products and services into Kosli.

## Finding resources

Kosli Capture finds all supported resources within your AWS accounts, and determines which Kosli environment should hold the snapshots. Kosli Capture will create physical environments for you inside Kosli.

Kosli Capture can [filter out resources based on AWS tags](/administration/kosli_capture/getting_started#excluding-resources).

As your cloud environment evolves, such as the addition of new AWS ECS clusters or the retirement of existing Lambdas, Kosli Capture automatically detects the changes. Because Kosli Capture creates physical environments as needed, when your infrastructure changes, Kosli will keep up. No changes to the configuration created during the initial setup are required.

## Multiple AWS accounts

Kosli Capture can operate across multiple AWS regions and accounts, allowing you to snapshot development, QA, pre-production, and production workloads with the same process.
Comment thread
gsavage marked this conversation as resolved.

## Operation

When Kosli Capture runs against one of your cloud accounts, it starts by gaining temporary credentials for the role you have created. It then uses these credentials to find resources to snapshot, such as finding all of your AWS ECS clusters. For each resource it identifies, Kosli Capture generates a snapshot within Kosli.

<Frame>
<img src="/images/administration/kosli-capture-4-steps.png" alt="Diagram of the four steps Kosli Capture follows in a customer AWS account: assume the IAM role using the external ID, receive temporary STS credentials, find the ECS clusters, then snapshot the clusters" />
</Frame>

Loading
Loading