From 10199664b9532e1318e6485d9de149a825efde8f Mon Sep 17 00:00:00 2001 From: April M <36110273+aimurphy@users.noreply.github.com> Date: Thu, 17 Sep 2026 10:48:32 -0700 Subject: [PATCH 1/3] property support exceptions --- modules/connectors/pages/sinks/elastic-search.adoc | 8 ++++++-- modules/connectors/pages/sinks/kafka.adoc | 8 ++++++-- modules/connectors/pages/sinks/kinesis.adoc | 8 ++++++-- modules/connectors/pages/sources/debezium-mysql.adoc | 8 ++++++-- modules/connectors/pages/sources/debezium-oracle.adoc | 8 ++++++-- .../connectors/pages/sources/debezium-postgres.adoc | 8 ++++++-- .../connectors/pages/sources/debezium-sqlserver.adoc | 8 ++++++-- modules/connectors/pages/sources/kafka.adoc | 8 ++++++-- modules/connectors/pages/sources/kinesis.adoc | 10 ++++++++-- 9 files changed, 56 insertions(+), 18 deletions(-) diff --git a/modules/connectors/pages/sinks/elastic-search.adoc b/modules/connectors/pages/sinks/elastic-search.adoc index 310fb2d..8d0ee9a 100644 --- a/modules/connectors/pages/sinks/elastic-search.adoc +++ b/modules/connectors/pages/sinks/elastic-search.adoc @@ -33,5 +33,9 @@ include::common:streaming:partial$connectors/sinks/pulsar-config-params.adoc[] Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://pulsar.apache.org/docs/io-elasticsearch-sink[OSS {pulsar} Elasticsearch sink connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that are only present in <>. \ No newline at end of file +Properties provided in the https://pulsar.apache.org/docs/io-elasticsearch-sink[OSS {pulsar} Elasticsearch sink connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that are only present in <> +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] \ No newline at end of file diff --git a/modules/connectors/pages/sinks/kafka.adoc b/modules/connectors/pages/sinks/kafka.adoc index e24db2f..8ac5dd0 100644 --- a/modules/connectors/pages/sinks/kafka.adoc +++ b/modules/connectors/pages/sinks/kafka.adoc @@ -33,5 +33,9 @@ include::common:streaming:partial$connectors/sinks/pulsar-config-params.adoc[] Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://pulsar.apache.org/docs/io-kafka-sink[OSS {pulsar} {kafka-short} sink connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that aren't present in {pulsar} {pulsar-version}. \ No newline at end of file +Properties provided in the https://pulsar.apache.org/docs/io-kafka-sink[OSS {pulsar} {kafka-short} sink connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that aren't present in {pulsar} {pulsar-version} +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] \ No newline at end of file diff --git a/modules/connectors/pages/sinks/kinesis.adoc b/modules/connectors/pages/sinks/kinesis.adoc index 0d8def2..0a18e91 100644 --- a/modules/connectors/pages/sinks/kinesis.adoc +++ b/modules/connectors/pages/sinks/kinesis.adoc @@ -35,5 +35,9 @@ include::common:streaming:partial$connectors/sinks/pulsar-config-params.adoc[] Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://pulsar.apache.org/docs/io-kinesis-sink[OSS {pulsar} Kinesis sink connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that aren't present in {pulsar} {pulsar-version}. \ No newline at end of file +Properties provided in the https://pulsar.apache.org/docs/io-kinesis-sink[OSS {pulsar} Kinesis sink connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that aren't present in {pulsar} {pulsar-version} +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] \ No newline at end of file diff --git a/modules/connectors/pages/sources/debezium-mysql.adoc b/modules/connectors/pages/sources/debezium-mysql.adoc index 5c979ab..ad20125 100644 --- a/modules/connectors/pages/sources/debezium-mysql.adoc +++ b/modules/connectors/pages/sources/debezium-mysql.adoc @@ -33,5 +33,9 @@ include::common:streaming:partial$connectors/sources/pulsar-config-params.adoc[] Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-connector-properties[Debezium connector for MySQL] and the https://pulsar.apache.org/docs/io-debezium-source[OSS {pulsar} Debezium source connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that are only present in <>. \ No newline at end of file +Properties provided in the https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-connector-properties[Debezium connector for MySQL] and the https://pulsar.apache.org/docs/io-debezium-source[OSS {pulsar} Debezium source connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that are only present in <> +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] \ No newline at end of file diff --git a/modules/connectors/pages/sources/debezium-oracle.adoc b/modules/connectors/pages/sources/debezium-oracle.adoc index 2f609f3..47cc171 100644 --- a/modules/connectors/pages/sources/debezium-oracle.adoc +++ b/modules/connectors/pages/sources/debezium-oracle.adoc @@ -36,5 +36,9 @@ include::common:streaming:partial$connectors/sources/pulsar-config-params.adoc[] Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-connector-properties[Debezium connector for Oracle] and the https://pulsar.apache.org/docs/io-debezium-source[OSS {pulsar} Debezium source connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that are only present in <>. \ No newline at end of file +Properties provided in the https://debezium.io/documentation/reference/stable/connectors/oracle.html#oracle-connector-properties[Debezium connector for Oracle] and the https://pulsar.apache.org/docs/io-debezium-source[OSS {pulsar} Debezium source connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that are only present in <> +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] \ No newline at end of file diff --git a/modules/connectors/pages/sources/debezium-postgres.adoc b/modules/connectors/pages/sources/debezium-postgres.adoc index 2e398b1..9f3ec20 100644 --- a/modules/connectors/pages/sources/debezium-postgres.adoc +++ b/modules/connectors/pages/sources/debezium-postgres.adoc @@ -33,5 +33,9 @@ include::common:streaming:partial$connectors/sources/pulsar-config-params.adoc[] Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-connector-properties[Debezium connector for PostgreSQL] and the https://pulsar.apache.org/docs/io-debezium-source[OSS {pulsar} Debezium source connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that are only present in <>. \ No newline at end of file +Properties provided in the https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-connector-properties[Debezium connector for PostgreSQL] and the https://pulsar.apache.org/docs/io-debezium-source[OSS {pulsar} Debezium source connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that are only present in <> +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] \ No newline at end of file diff --git a/modules/connectors/pages/sources/debezium-sqlserver.adoc b/modules/connectors/pages/sources/debezium-sqlserver.adoc index 5c7d1d9..a2db512 100644 --- a/modules/connectors/pages/sources/debezium-sqlserver.adoc +++ b/modules/connectors/pages/sources/debezium-sqlserver.adoc @@ -53,5 +53,9 @@ Make sure you set the `topic.prefix`, `database.server.name`, and `task.id` prop Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-connector-properties[Debezium connector for SQL Server] and the https://pulsar.apache.org/docs/io-debezium-source[OSS {pulsar} Debezium source connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that are only present in <>. \ No newline at end of file +Properties provided in the https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-connector-properties[Debezium connector for SQL Server] and the https://pulsar.apache.org/docs/io-debezium-source[OSS {pulsar} Debezium source connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that are only present in <> +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] \ No newline at end of file diff --git a/modules/connectors/pages/sources/kafka.adoc b/modules/connectors/pages/sources/kafka.adoc index 49598c4..9c707e7 100644 --- a/modules/connectors/pages/sources/kafka.adoc +++ b/modules/connectors/pages/sources/kafka.adoc @@ -28,5 +28,9 @@ include::common:streaming:partial$connectors/sources/pulsar-config-params.adoc[] Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://pulsar.apache.org/docs/io-kafka-source[OSS {pulsar} {kafka-short} source connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that aren't present in {pulsar} {pulsar-version}. \ No newline at end of file +Properties provided in the https://pulsar.apache.org/docs/io-kafka-source[OSS {pulsar} {kafka-short} source connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that aren't present in {pulsar} {pulsar-version} +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] \ No newline at end of file diff --git a/modules/connectors/pages/sources/kinesis.adoc b/modules/connectors/pages/sources/kinesis.adoc index 5edd895..93ed7a4 100644 --- a/modules/connectors/pages/sources/kinesis.adoc +++ b/modules/connectors/pages/sources/kinesis.adoc @@ -28,5 +28,11 @@ include::common:streaming:partial$connectors/sources/pulsar-config-params.adoc[] Set these properties in the `configs` section of the connector configuration. -Generally, all properties provided in the https://pulsar.apache.org/docs/io-kinesis-source[OSS {pulsar} Kinesis source connector] are supported. -Exceptions include properties that aren't relevant to {product} and properties that aren't present in {pulsar} {pulsar-version}. \ No newline at end of file +Properties provided in the https://pulsar.apache.org/docs/io-kinesis-source[OSS {pulsar} Kinesis source connector] are supported with the following exceptions: + +* Properties that aren't relevant to {product} +* Properties that aren't present in {pulsar} {pulsar-version} +* Properties that are inherently unsupported by your environment +* Properties that are inherently unsupported by your xref:operations:astream-pricing.adoc[cluster deployment type] + +For example, shared clusters cannot use `awsCredentialPluginName:org.apache.pulsar.io.aws.STSAssumeRoleProviderPlugin`. \ No newline at end of file From 568169e734b932d97e8fc03e016113fdaf863a7d Mon Sep 17 00:00:00 2001 From: April M <36110273+aimurphy@users.noreply.github.com> Date: Thu, 17 Sep 2026 14:34:07 -0700 Subject: [PATCH 2/3] codeblock cleaning again --- .../astream-subscriptions-exclusive.adoc | 9 +- .../pages/astream-subscriptions-failover.adoc | 8 +- .../astream-subscriptions-keyshared.adoc | 15 +- .../pages/astream-subscriptions-shared.adoc | 8 +- modules/ROOT/partials/sinks/edit.adoc | 6 +- modules/ROOT/partials/sinks/get-started.adoc | 9 +- modules/ROOT/partials/sources/edit.adoc | 6 +- .../ROOT/partials/sources/get-started.adoc | 9 +- .../partials/subscription-setup-project.adoc | 17 +- .../partials/subscription-start-consumer.adoc | 3 +- modules/apis/pages/api-operations.adoc | 406 +++++++++--------- modules/apis/pages/index.adoc | 159 ++++--- modules/developing/pages/astream-cdc.adoc | 211 +++++---- .../developing/pages/astream-functions.adoc | 35 +- .../pages/clients/csharp-produce-consume.adoc | 5 - .../pages/clients/golang-produce-consume.adoc | 5 - .../pages/clients/java-produce-consume.adoc | 9 - .../pages/clients/nodejs-produce-consume.adoc | 14 +- .../pages/clients/python-produce-consume.adoc | 6 - .../pages/clients/spring-produce-consume.adoc | 5 +- .../pages/configure-pulsar-env.adoc | 3 +- .../pages/gpt-schema-translator.adoc | 58 ++- .../pages/produce-consume-pulsar-client.adoc | 12 +- .../real-time-data-pipelines-tutorial.adoc | 8 +- .../pages/astream-georeplication.adoc | 4 - .../pages/astream-scrape-metrics.adoc | 27 +- .../operations/pages/astream-token-gen.adoc | 16 +- .../pages/monitoring/integration.adoc | 2 - .../pages/monitoring/new-relic.adoc | 1 - .../pages/monitoring/stream-audit-logs.adoc | 41 +- 30 files changed, 514 insertions(+), 603 deletions(-) diff --git a/modules/ROOT/pages/astream-subscriptions-exclusive.adoc b/modules/ROOT/pages/astream-subscriptions-exclusive.adoc index cbea701..539009d 100644 --- a/modules/ROOT/pages/astream-subscriptions-exclusive.adoc +++ b/modules/ROOT/pages/astream-subscriptions-exclusive.adoc @@ -25,7 +25,6 @@ To create a {pulsar-short} exclusive subscription, create a `pulsarConsumer` wit . In `src/main/java/com/datastax/pulsar`, create a `SimplePulsarConsumer.java` file with the following contents: + -.SimplePulsarConsumer.java [source,java,subs="+attributes"] ---- include::ROOT:partial$simplepulsarconsumer.java[] @@ -33,7 +32,6 @@ include::ROOT:partial$simplepulsarconsumer.java[] + Alternatively, you can omit the `.subscriptionType` declaration because `Exclusive` is the default subscription type: + -.SimplePulsarConsumer.java with implied exclusive subscription [source,java] ---- pulsarConsumer = pulsarClient.newConsumer(Schema.JSON(DemoBean.class)) @@ -52,9 +50,10 @@ Alternatively, you can omit the `.subscriptionType` declaration because `Exclusi include::ROOT:partial$subscription-start-consumer.adoc[] -. In a new terminal window, run `SimplePulsarProducer.java` to begin producing messages: +. In a new terminal window, run `SimplePulsarProducer.java` to begin producing messages. ++ +The producer's terminal shows when each message is sent: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 93573631 sent @@ -64,7 +63,6 @@ include::ROOT:partial$subscription-start-consumer.adoc[] + In the `SimplePulsarConsumer` terminal, the consumer begins consuming the produced messages: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":93573631,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} @@ -76,7 +74,6 @@ In the `SimplePulsarConsumer` terminal, the consumer begins consuming the produc + The second consumer cannot subscribe to the topic because the subscription is exclusive: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.Configuration - Configuration has been loaded successfully diff --git a/modules/ROOT/pages/astream-subscriptions-failover.adoc b/modules/ROOT/pages/astream-subscriptions-failover.adoc index 80c39fd..3f55583 100644 --- a/modules/ROOT/pages/astream-subscriptions-failover.adoc +++ b/modules/ROOT/pages/astream-subscriptions-failover.adoc @@ -27,7 +27,6 @@ To create a {pulsar-short} failover subscription, create a `pulsarConsumer` with . In `src/main/java/com/datastax/pulsar`, create a `SimplePulsarConsumer.java` file with the following contents: + -.SimplePulsarConsumer.java [source,java,subs="+attributes"] ---- include::ROOT:partial$simplepulsarconsumer.java[] @@ -37,9 +36,10 @@ include::ROOT:partial$simplepulsarconsumer.java[] include::ROOT:partial$subscription-start-consumer.adoc[] -. In a new terminal window, run `SimplePulsarProducer.java` to begin producing messages: +. In a new terminal window, run `SimplePulsarProducer.java` to begin producing messages. ++ +The producer's terminal shows when each message is sent: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 50585599 sent @@ -52,7 +52,6 @@ include::ROOT:partial$subscription-start-consumer.adoc[] + In the `SimplePulsarConsumer` terminal, the primary consumer begins consuming messages: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":50585599,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} @@ -66,7 +65,6 @@ The backup consumer subscribes to the topic but does not immediately begin consu . In your first `SimplePulsarConsumer` terminal, stop the process (`Ctrl+C`), and then switch to your second `SimplePulsarConsumer` terminal. Notice that the backup consumer begins consuming messages where the first consumer left off: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":73260535,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} diff --git a/modules/ROOT/pages/astream-subscriptions-keyshared.adoc b/modules/ROOT/pages/astream-subscriptions-keyshared.adoc index 6b35656..82f8a46 100644 --- a/modules/ROOT/pages/astream-subscriptions-keyshared.adoc +++ b/modules/ROOT/pages/astream-subscriptions-keyshared.adoc @@ -42,7 +42,6 @@ Running multiple consumers with `autoSplitHashRange` balances the messaging load . In `src/main/java/com/datastax/pulsar`, create a `SimplePulsarConsumer.java` file with the following contents: + -.SimplePulsarConsumer.java [source,java,subs="+attributes"] ---- include::ROOT:partial$simplepulsarconsumer.java[] @@ -50,7 +49,6 @@ include::ROOT:partial$simplepulsarconsumer.java[] . In `pulsarConsumer`, add `.keySharedPolicy(KeySharedPolicy.autoSplitHashRange())`: + -.SimplePulsarConsumer.java with autoSplitHashRange [source,java,subs="+attributes"] ---- ... @@ -80,7 +78,6 @@ This policy requires additional dependencies and producer configuration changes, . In `src/main/java/com/datastax/pulsar`, create a `SimplePulsarConsumer.java` file with the following contents: + -.SimplePulsarConsumer.java [source,java,subs="+attributes"] ---- include::ROOT:partial$simplepulsarconsumer.java[] @@ -88,7 +85,6 @@ include::ROOT:partial$simplepulsarconsumer.java[] . Import the following additional classes that are required for the `stickyHashRange` policy: + -.SimplePulsarConsumer.java [source,java] ---- import org.apache.pulsar.client.api.Range; @@ -100,7 +96,6 @@ import org.apache.pulsar.client.api.SubscriptionType; + The following example sets all possible hashes (`0-65535`) on this subscription to one consumer: + -.SimplePulsarConsumer.java with stickyHashRange for one consumer [source,java,subs="+attributes"] ---- ... @@ -125,7 +120,6 @@ pulsarConsumer = pulsarClient.newConsumer(Schema.JSON(DemoBean.class)) To split the hash range between multiple consumers, add a `Range.of()` argument for each consumer with the assigned hash range. For example: + -.SimplePulsarConsumer.java with stickyHashRange for two consumers [source,java] ---- // Policy assigns half of the hash range to one consumer and half to another @@ -136,7 +130,6 @@ For example: + .. In `SimplePulsarProducer.java`, import the following classes: + -.SimplePulsarProducer.java [source,java] ---- import org.apache.pulsar.client.api.BatcherBuilder; @@ -145,7 +138,6 @@ import org.apache.pulsar.client.api.HashingScheme; .. Configure the `pulsarProducer` to use the `JavaStringHash` hashing scheme: + -.SimplePulsarProducer.java [source,java] ---- pulsarProducer = pulsarClient @@ -165,9 +157,10 @@ pulsarProducer = pulsarClient include::ROOT:partial$subscription-start-consumer.adoc[] -. In a new terminal window, run `SimplePulsarProducer.java` to begin producing messages: +. In a new terminal window, run `SimplePulsarProducer.java` to begin producing messages. ++ +The producer's terminal shows when each message is sent: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 55794190 sent @@ -178,7 +171,6 @@ include::ROOT:partial$subscription-start-consumer.adoc[] + In the `SimplePulsarConsumer` terminal, the consumer begins receiving messages: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":55794190,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} @@ -193,7 +185,6 @@ The auto-hashing policy balances hash ranges across available consumers. If you used sticky hashing with one `Range.of()` argument, then the new consumer cannot subscribe to the topic because the `SimplePulsarConsumer` configuration reserved the entire hash range for the first consumer. For example: + -.Result when using sticky hashing limited to one consumer [source,console] ---- [main] INFO com.datastax.pulsar.Configuration - Configuration has been loaded successfully diff --git a/modules/ROOT/pages/astream-subscriptions-shared.adoc b/modules/ROOT/pages/astream-subscriptions-shared.adoc index 8a4b5ec..a1ec6f7 100644 --- a/modules/ROOT/pages/astream-subscriptions-shared.adoc +++ b/modules/ROOT/pages/astream-subscriptions-shared.adoc @@ -26,7 +26,6 @@ To create a {pulsar-short} shared subscription, create a `pulsarConsumer` with ` . In `src/main/java/com/datastax/pulsar`, create a `SimplePulsarConsumer.java` file with the following contents: + -.SimplePulsarConsumer.java [source,java,subs="+attributes"] ---- include::ROOT:partial$simplepulsarconsumer.java[] @@ -36,9 +35,10 @@ include::ROOT:partial$simplepulsarconsumer.java[] include::ROOT:partial$subscription-start-consumer.adoc[] -. In a new terminal window, run `SimplePulsarProducer.java` to begin producing messages: +. In a new terminal window, run `SimplePulsarProducer.java` to begin producing messages. ++ +The producer's terminal shows when each message is sent: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarProducer - Message 59819331 sent @@ -50,7 +50,6 @@ include::ROOT:partial$subscription-start-consumer.adoc[] + In the `SimplePulsarConsumer` terminal, the consumer begins receiving messages: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":59819331,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} @@ -62,7 +61,6 @@ In the `SimplePulsarConsumer` terminal, the consumer begins receiving messages: + The new consumer subscribes to the topic and consumes messages: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.SimplePulsarConsumer - Message received: {"show_id":70129519,"cast":"LeBron James, Anthony Davis, Kyrie Irving, Damian Lillard, Klay Thompson...","country":"United States","date_added":"July 16, 2021","description":"NBA superstar LeBron James teams up with Bugs Bunny and the rest of the Looney Tunes for this long-awaited sequel.","director":"Malcolm D. Lee","duration":"120 min","listed_in":"Animation, Adventure, Comedy","rating":"PG","release_year":2021,"title":"Space Jam: A New Legacy","type":"Movie"} diff --git a/modules/ROOT/partials/sinks/edit.adoc b/modules/ROOT/partials/sinks/edit.adoc index 8bc4a5b..25188a8 100644 --- a/modules/ROOT/partials/sinks/edit.adoc +++ b/modules/ROOT/partials/sinks/edit.adoc @@ -6,7 +6,8 @@ Additionally, some properties can be modified with specific arguments, such as ` To get the current configuration, see xref:apis:api-operations.adoc#get-sink-connector-configuration-data[Get sink connector configuration data]. -.pulsar-admin CLI +pulsar-admin CLI:: ++ [source,shell,subs="+attributes"] ---- ./bin/pulsar-admin sinks update \ @@ -17,7 +18,8 @@ To get the current configuration, see xref:apis:api-operations.adoc#get-sink-con --parallelism 2 ---- -.{pulsar-short} Admin API +{pulsar-short} Admin API:: ++ [source,shell] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \ diff --git a/modules/ROOT/partials/sinks/get-started.adoc b/modules/ROOT/partials/sinks/get-started.adoc index fce464f..b23ce60 100644 --- a/modules/ROOT/partials/sinks/get-started.adoc +++ b/modules/ROOT/partials/sinks/get-started.adoc @@ -17,7 +17,8 @@ For example: `{connectorType}-sink-prod-us-east-1`. . Create the connector using JSON-formatted connector configuration data. You can pass the configuration directly or with a configuration file. + -.pulsar-admin CLI +pulsar-admin CLI:: ++ [source,shell,subs="+attributes"] ---- ./bin/pulsar-admin sinks create \ @@ -28,7 +29,8 @@ You can pass the configuration directly or with a configuration file. --sink-config-file configs.json ---- + -.{pulsar-short} Admin API +{pulsar-short} Admin API:: ++ [source,shell] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \ @@ -36,7 +38,8 @@ curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$ --form "sinkConfig=@configs.json;type=application/json" ---- + -.Example configuration data structure +Example configuration data structure:: ++ [source,json] ---- include::common:streaming:example$connectors/sinks/{connectorType}/sample-data.json[] diff --git a/modules/ROOT/partials/sources/edit.adoc b/modules/ROOT/partials/sources/edit.adoc index facb65a..4a49221 100644 --- a/modules/ROOT/partials/sources/edit.adoc +++ b/modules/ROOT/partials/sources/edit.adoc @@ -6,7 +6,8 @@ Additionally, some properties can be modified with specific arguments, such as ` To get the current configuration, see xref:apis:api-operations.adoc#get-source-connector-configuration-data[Get source connector configuration data]. -.pulsar-admin CLI +pulsar-admin CLI:: ++ [source,shell,subs="+attributes"] ---- ./bin/pulsar-admin sources update \ @@ -17,7 +18,8 @@ To get the current configuration, see xref:apis:api-operations.adoc#get-source-c --parallelism 2 ---- -.{pulsar-short} Admin API +{pulsar-short} Admin API:: ++ [source,shell] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \ diff --git a/modules/ROOT/partials/sources/get-started.adoc b/modules/ROOT/partials/sources/get-started.adoc index e653b7a..1b51377 100644 --- a/modules/ROOT/partials/sources/get-started.adoc +++ b/modules/ROOT/partials/sources/get-started.adoc @@ -17,7 +17,8 @@ For example: `{connectorType}-source-prod-us-east-1`. . Create the connector using JSON-formatted connector configuration data. You can pass the configuration directly or with a configuration file. + -.pulsar-admin CLI +pulsar-admin CLI:: ++ [source,shell,subs="+attributes"] ---- ./bin/pulsar-admin sources create \ @@ -28,7 +29,8 @@ You can pass the configuration directly or with a configuration file. --source-config-file configs.json ---- + -.{pulsar-short} Admin API +{pulsar-short} Admin API:: ++ [source,shell] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \ @@ -36,7 +38,8 @@ curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE --form "sourceConfig=@mynetty-source-config.json;type=application/json" ---- + -.Example configuration data structure +Example configuration data structure:: ++ [source,json] ---- include::common:streaming:example$connectors/sources/{connectorType}/sample-data.json[] diff --git a/modules/ROOT/partials/subscription-setup-project.adoc b/modules/ROOT/partials/subscription-setup-project.adoc index 3a74af9..a2b180d 100644 --- a/modules/ROOT/partials/subscription-setup-project.adoc +++ b/modules/ROOT/partials/subscription-setup-project.adoc @@ -2,7 +2,6 @@ . Edit the `pom.xml` file to include the following dependencies: + -.pom.xml [source,xml] ---- ---- -. In `src/main/resources`, create the following `application.properties` file with the connection details for your {product} cluster. +. In `src/main/resources`, create an `application.properties` file with the connection details for your {product} cluster. ++ Create the `resources` subdirectory if it doesn't already exist. + -./src/main/resources/application.properties -[source,properties,subs="+quotes"] +[source,plaintext,subs="+quotes"] ---- # --------------------------------------- # Configuration of your Astra Streaming tenant @@ -91,10 +90,10 @@ authentication_token=**ASTRA_APPLICATION_TOKEN** topic_name=my-topic ---- -. In `src/main/java/com/datastax/pulsar`, create the following `Configuration.java` class to load the connection details from `application.properties`. +. In `src/main/java/com/datastax/pulsar`, create a `Configuration.java` class to load the connection details from `application.properties`. ++ Create the `/datastax/pulsar` subdirectories if they don't already exist. + -.Configuration.java [source,java] ---- package com.datastax.pulsar; @@ -201,9 +200,8 @@ public class Configuration { } ---- -. In `src/main/java/com/datastax/pulsar`, create the following `DemoBean.java` class to represent the example messages that will be produced and consumed: +. In `src/main/java/com/datastax/pulsar`, create a `DemoBean.java` class to represent the example messages that will be produced and consumed: + -.DemoBean.java [source,java] ---- package com.datastax.pulsar; @@ -349,9 +347,8 @@ public class DemoBean { } ---- -. In `src/main/java/com/datastax/pulsar`, create a `SimplePulsarProducer.java` file with the following contents: +. In `src/main/java/com/datastax/pulsar`, create a file named `SimplePulsarProducer.java` with the following contents: + -.SimplePulsarProducer.java [source,java] ---- package com.datastax.pulsar; diff --git a/modules/ROOT/partials/subscription-start-consumer.adoc b/modules/ROOT/partials/subscription-start-consumer.adoc index d62dedf..e6e7f27 100644 --- a/modules/ROOT/partials/subscription-start-consumer.adoc +++ b/modules/ROOT/partials/subscription-start-consumer.adoc @@ -1,8 +1,7 @@ . Run `SimplePulsarConsumer.java` to begin consuming messages as the primary consumer. + -The confirmation message and a cursor appear to indicate the consumer is ready: +A confirmation message and a cursor appear when the consumer is ready: + -.Result [source,console] ---- [main] INFO com.datastax.pulsar.Configuration - Configuration has been loaded successfully diff --git a/modules/apis/pages/api-operations.adoc b/modules/apis/pages/api-operations.adoc index 2997131..5164f19 100644 --- a/modules/apis/pages/api-operations.adoc +++ b/modules/apis/pages/api-operations.adoc @@ -61,10 +61,10 @@ To learn how to get these values, see <>. The examples in this guide use environment variables for these values. For example: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sinks/builtinsinks" \ ---header "Authorization: $PULSAR_TOKEN" + --header "Authorization: $PULSAR_TOKEN" ---- === Format responses @@ -85,14 +85,13 @@ Unless otherwise noted, a lack of response indicates that the request was succes Get a list of namespaces in a tenant: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "https://$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- [ @@ -103,7 +102,7 @@ curl -sS --fail -L -X GET "https://$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT" === Create a namespace -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/**NAMESPACE_TO_CREATE**" \ --header "Authorization: Bearer $PULSAR_TOKEN" \ @@ -127,16 +126,15 @@ For more information and all endpoints, see the xref:apis:attachment$pulsar-admi Get the message retention settings for a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/retention" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- The result is a JSON object containing the current message retention settings: -.Result [source,json] ---- { @@ -147,15 +145,15 @@ The result is a JSON object containing the current message retention settings: Edit the message retention settings for a namespace, passing the new settings in the request body: -[source,curl] +[source,bash] ---- curl -L -X POST "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/retention" \ ---header "Content-Type: application/json" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---data '{ - "retentionTimeInMinutes": 360, - "retentionSizeInMB": 102 -}' + --header "Content-Type: application/json" \ + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --data '{ + "retentionTimeInMinutes": 360, + "retentionSizeInMB": 102 + }' ---- include::ROOT:partial$curl-get-template-tip.adoc[] @@ -164,16 +162,15 @@ include::ROOT:partial$curl-get-template-tip.adoc[] Get the backlog quota settings for a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/backlogQuotaMap" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- The result is a JSON object containing the current backlog quota settings: -.Result [source,json] ---- { @@ -188,17 +185,17 @@ The result is a JSON object containing the current backlog quota settings: Edit the backlog quota settings for a namespace, passing the new settings in the request body: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/backlogQuota" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---header "Content-Type: application/json" \ ---data '{ - "limit": -1, - "limitSize": 102400, - "limitTime": 3600, - "policy": "producer_exception" -}' + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --header "Content-Type: application/json" \ + --data '{ + "limit": -1, + "limitSize": 102400, + "limitTime": 3600, + "policy": "producer_exception" + }' ---- include::ROOT:partial$curl-get-template-tip.adoc[] @@ -207,39 +204,39 @@ include::ROOT:partial$curl-get-template-tip.adoc[] Get the message time-to-live (TTL) setting for a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/messageTTL" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- The response is a number representing the TTL duration in seconds, such as `3600`. Edit the message TTL policy for a namespace, passing the new TTL in seconds in the request body: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/messageTTL" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---header "Content-Type: application/json" \ ---data 5000 + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --header "Content-Type: application/json" \ + --data 5000 ---- ==== Automatic topic creation policy Edit the automatic topic creation settings for the specified namespace, passing the new settings in the request body: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/autoTopicCreation" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---header "Content-Type: application/json" \ ---data '{ - "allowAutoTopicCreation": true, - "defaultNumPartitions" 3, - "topicType": "partitioned" -}' + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --header "Content-Type: application/json" \ + --data '{ + "allowAutoTopicCreation": true, + "defaultNumPartitions" 3, + "topicType": "partitioned" + }' ---- `allowAutoTopicCreation` indicates whether automatic topic creation is allowed for the namespace. @@ -254,50 +251,50 @@ If set to `non-partitioned`, then the `defaultNumPartitions` field is ignored. Get the maximum number of consumers allowed for each topic in a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/maxConsumersPerTopic" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- Edit the maximum consumers per topic policy, passing the new limit in the request body: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/maxConsumersPerTopic" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---header "Content-Type: application/json" \ ---data 100 + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --header "Content-Type: application/json" \ + --data 100 ---- ==== Maximum topics per namespace policy Get the maximum number of topics allowed in a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/maxTopicsPerNamespace" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- Edit the maximum topics per namespace policy, passing the new limit in the request body: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/$NAMESPACE/maxTopicsPerNamespace" \ ---header "Content-Type: application/json" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---data 1000 + --header "Content-Type: application/json" \ + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --data 1000 ---- === Delete a namespace -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X DELETE "$WEB_SERVICE_URL/admin/v2/namespaces/$TENANT/**NAMESPACE_TO_DELETE**" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- [#pulsar-admin-api-topic-operations] @@ -311,44 +308,43 @@ There are multiple endpoints you can use to get a list of topics in a namespace * Get all persistent topics in a namespace: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- * Get partitioned persistent topics in a namespace: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/partitioned" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- * Get non-persistent topics in a namespace: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/non-persistent/$TENANT/$NAMESPACE" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- * Get partitioned non-persistent topics in a namespace: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/non-persistent/$TENANT/$NAMESPACE/partitioned" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- For all of these endpoints, the result is a list of topics in the given namespace that match the endpoint's filter. For example, the following result includes partitioned and non-partitioned persistent topics: -.Result [source,json] ---- [ @@ -369,56 +365,56 @@ There are multiple endpoints you can use to create different types of topics: * Create a persistent, non-partitioned topic: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- * Create a persistent, partitioned topic: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/partitions" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---header "Content-Type: application/json" \ ---data $NUM_OF_PARTITIONS + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --header "Content-Type: application/json" \ + --data $NUM_OF_PARTITIONS ---- * Create a non-persistent, non-partitioned topic: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v2/non-persistent/$TENANT/$NAMESPACE/$TOPIC" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- * Create a non-persistent, partitioned topic: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v2/non-persistent/$TENANT/$NAMESPACE/$TOPIC/partitions" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---header "Content-Type: application/json" \ ---data $NUM_OF_PARTITIONS + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --header "Content-Type: application/json" \ + --data $NUM_OF_PARTITIONS ---- === Delete a topic Delete a persistent topic: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X DELETE "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- Delete a non-persistent topic: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X DELETE "$WEB_SERVICE_URL/admin/v2/non-persistent/$TENANT/$NAMESPACE/$TOPIC" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- === Get topic details @@ -430,14 +426,13 @@ For more information and all endpoints, see the xref:apis:attachment$pulsar-admi ==== Get internal statistics for a persistent, non-partitioned topic -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/internalStats" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -474,14 +469,13 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPA ==== Get statistics for a persistent, non-partitioned topic -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/stats" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -522,14 +516,13 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPA ==== Get statistics for a persistent, partitioned topic -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/partitioned-stats" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -641,14 +634,13 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPA ==== Get statistics for all topics in a namespace -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/stats/topics/$TENANT/$NAMESPACE" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -785,14 +777,13 @@ The following sections demonstrate some, but not all, of these endpoints. Get subscriptions for a persistent topic: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/subscriptions" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- [ @@ -807,40 +798,40 @@ Create a subscription on a topic, using the `replicated` query parameter to set * Create a replicated subscription for a persistent topic: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/subscription/$SUBSCRIPTION?replicated=true" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---header "Content-Type: application/json" + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --header "Content-Type: application/json" ---- * Create a non-replicated subscription for a persistent topic: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X PUT "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/subscription/$SUBSCRIPTION?replicated=false" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---header "Content-Type: application/json" + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --header "Content-Type: application/json" ---- === Delete a subscription Delete a subscription from a topic: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X DELETE "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/subscription/$SUBSCRIPTION" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- === Clear a subscription Clear a subscription on a topic by skipping all pending messages: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v2/persistent/$TENANT/$NAMESPACE/$TOPIC/subscription/$SUBSCRIPTION/skip_all" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- == Function operations @@ -849,14 +840,13 @@ Use the following {product} {pulsar-short} Admin API endpoints to manage xref:de === Get functions in a namespace -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPACE" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- [ @@ -866,14 +856,13 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPAC === Get function status -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPACE/$FUNCTION/status" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -903,14 +892,13 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPAC === Get function statistics -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPACE/$FUNCTION/stats" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -953,14 +941,13 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPAC === Get function details -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPACE/$FUNCTION" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -1039,26 +1026,26 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPAC === Start a function -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPACE/$FUNCTION/start" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- === Stop a function -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPACE/$FUNCTION/stop" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- === Restart a function -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/functions/$TENANT/$NAMESPACE/$FUNCTION/restart" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- [#sink-connector-operations] @@ -1079,25 +1066,24 @@ For configuration details, see the documentation for your preferred xref:connect Get a list of sink connectors that are available in your {product} {pulsar-short} tenant: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sinks/builtinsinks" \ ---header "Authorization: Bearer $PULSAR_TOKEN" -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" + | python3 -mjson.tool ---- === Get sinks in a namespace Get a list of sinks deployed in a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- [ @@ -1110,16 +1096,15 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE" \ Get the status of all instances of a sink connector: -[source,curl] +[source,bash] ---- -curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SINK_NAME/status" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool +curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME/status" \ + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- The response includes an `instances` array with one object for each connector instance: -.Result [source,json] ---- { @@ -1148,16 +1133,15 @@ The response includes an `instances` array with one object for each connector in Get the status of an individual instance of a sink connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME/$SINK_INSTANCE_ID/status" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- The response includes a single object with the details for the specified connector instance: -.Result [source,json] ---- { @@ -1190,14 +1174,13 @@ The response includes a single object with the details for the specified connect Get the configuration for an existing sink connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -1258,7 +1241,6 @@ To create a sink connector on a topic, you must provide the xref:connectors:inde The following example configures the built-in {kafka-reg} sink connector in {product}: -.kafka-sink-config.json [source,json] ---- { @@ -1292,11 +1274,11 @@ You can pass the configuration in-line or with a configuration file. The following example uses the previous configuration file example, `kafka-sink-config.json`: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---form "sinkConfig=@kafka-sink-config.json;type=application/json" + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --form "sinkConfig=@kafka-sink-config.json;type=application/json" ---- include::ROOT:partial$curl-file-input-tip.adoc[] @@ -1306,18 +1288,18 @@ include::ROOT:partial$curl-file-input-tip.adoc[] Start all instances of a sink connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME/start" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- Start an individual instance of a sink connector -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME/$SINK_INSTANCE_ID/start" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- [#stop-a-sink-connector] @@ -1325,18 +1307,18 @@ curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$ Stop all instances of a sink connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME/stop" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- Stop an individual instance of a sink connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME/$SINK_INSTANCE_ID/stop" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- [#restart-a-sink-connector] @@ -1344,15 +1326,15 @@ curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$ Restart all instances of a sink connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME/restart" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- Restart an individual instance of a sink connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME/$SINK_INSTANCE_ID/restart" \ --header "Authorization: Bearer $PULSAR_TOKEN" @@ -1363,10 +1345,10 @@ curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$ Delete all instances of a sink connector from a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X DELETE "$WEB_SERVICE_URL/admin/v3/sinks/$TENANT/$NAMESPACE/$SINK_NAME" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- [#source-connector-operations] @@ -1387,24 +1369,23 @@ For configuration details, see the documentation for your preferred xref:connect Get a list of source connectors that are available in your {product} {pulsar-short} tenant: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sources/builtinsources" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- === Get sources in a namespace Get a list of source connectors deployed in a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- [ @@ -1417,16 +1398,15 @@ curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE" Get the status of all instances of a source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME/status" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- The response includes an `instances` array with one object for each connector instance: -.Result [source,json] ---- { @@ -1455,16 +1435,15 @@ The response includes an `instances` array with one object for each connector in Get the status of an individual instance of a source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME/$SOURCE_INSTANCE_ID/status" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- The response includes a single object with the details for the specified connector instance: -.Result [source,json] ---- { @@ -1497,14 +1476,13 @@ The response includes a single object with the details for the specified connect Get the configuration for an existing source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $PULSAR_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -1550,11 +1528,11 @@ To create a source connector on a topic, you must provide the xref:connectors:in You can pass the configuration in-line or with a configuration file. The following example uses a configuration file name `mynetty-source-config.json`: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \ ---header "Authorization: Bearer $PULSAR_TOKEN" \ ---form "sourceConfig=@mynetty-source-config.json;type=application/json" + --header "Authorization: Bearer $PULSAR_TOKEN" \ + --form "sourceConfig=@mynetty-source-config.json;type=application/json" ---- include::ROOT:partial$curl-file-input-tip.adoc[] @@ -1564,18 +1542,18 @@ include::ROOT:partial$curl-file-input-tip.adoc[] Start all instances of a source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME/start" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- Start an individual instance of a source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME/$SOURCE_INSTANCE_ID/start" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- [#stop-a-source-connector] @@ -1583,18 +1561,18 @@ curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE Start all instances of a source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME/stop" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- Start an individual instance of a source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME/$SOURCE_INSTANCE_ID/stop" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- [#restart-a-source-connector] @@ -1602,15 +1580,15 @@ curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE Restart all instances of a source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME/restart" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- Restart an individual instance of a source connector: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME/$SOURCE_INSTANCE_ID/restart" \ --header "Authorization: Bearer $PULSAR_TOKEN" @@ -1621,10 +1599,10 @@ curl -sS --fail -L -X POST "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE Delete all instances of a source connector from a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X DELETE "$WEB_SERVICE_URL/admin/v3/sources/$TENANT/$NAMESPACE/$SOURCE_NAME" \ ---header "Authorization: Bearer $PULSAR_TOKEN" + --header "Authorization: Bearer $PULSAR_TOKEN" ---- == See also diff --git a/modules/apis/pages/index.adoc b/modules/apis/pages/index.adoc index 59e82ae..26262db 100644 --- a/modules/apis/pages/index.adoc +++ b/modules/apis/pages/index.adoc @@ -55,14 +55,13 @@ Use the following {product} {devops-api} endpoints to manage {product} {pulsar-s [#get-tenant-details] === Get tenant details -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v2/streaming/tenants" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $ASTRA_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- [ @@ -120,14 +119,13 @@ curl -sS --fail -L -X GET "https://api.astra.datastax.com/v2/streaming/tenants" Get cloud provider and regions supported by {product}: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v2/streaming/providers" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $ASTRA_TOKEN" \ + | python3 -mjson.tool ---- -.Result [source,json] ---- { @@ -159,36 +157,35 @@ You can do this directly in the request body or with a JSON file. * Create a tenant using in-line input: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "https://api.astra.datastax.com/v2/streaming/tenants" \ ---header "Content-Type: application/json" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---data-raw '{ - "cloudProvider": "aws", - "cloudRegion": "useast2", - "tenantName": "mytenant", - "userEmail": "joshua@example.com" -}' \ -| python3 -mjson.tool + --header "Content-Type: application/json" \ + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --data-raw '{ + "cloudProvider": "aws", + "cloudRegion": "useast2", + "tenantName": "mytenant", + "userEmail": "joshua@example.com" + }' \ + | python3 -mjson.tool ---- * Create a tenant using file input: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "https://api.astra.datastax.com/v2/streaming/tenants" \ ---header "Content-Type: application/json" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---data "@mytenant-config.json" \ -| python3 -mjson.tool + --header "Content-Type: application/json" \ + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --data "@mytenant-config.json" \ + | python3 -mjson.tool ---- + include::ROOT:partial$curl-file-input-tip.adoc[] The output includes the new tenant's details and a `pulsarToken`, which is an initial xref:operations:astream-token-gen.adoc[{pulsar-short} JWT] for the new {pulsar-short} instance: -.Result [source,json] ---- { @@ -221,11 +218,11 @@ The output includes the new tenant's details and a `pulsarToken`, which is an in [#delete-a-tenant] === Delete a tenant -[source,curl] +[source,bash] ---- curl -sS --fail -L -X DELETE "https://api.astra.datastax.com/v2/streaming/tenants/$TENANT/clusters/$CLUSTER" \ ---header "Content-Type: application/json" \ ---header "Authorization: Bearer $ASTRA_TOKEN" + --header "Content-Type: application/json" \ + --header "Authorization: Bearer $ASTRA_TOKEN" ---- == Georeplication operations @@ -237,16 +234,15 @@ For more information, see xref:operations:astream-georeplication.adoc[]. Get the georeplication settings for a namespace: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v2/streaming/replications/$INSTANCE/$TENANT/$NAMESPACE" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ -| python3 -mjson.tool + --header "Authorization: Bearer $ASTRA_TOKEN" \ + | python3 -mjson.tool ---- If georeplication is enabled, the response includes the replication configuration details. -.Result [source,json] ---- { @@ -295,18 +291,18 @@ If georeplication is enabled, the response includes the replication configuratio To enable georeplication, provide the origin and destination clusters, the namespace, bidirectional flag, and the email address (if set for the tenant): -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "https://api.astra.datastax.com/v2/streaming/replications/$INSTANCE/$TENANT/$NAMESPACE" \ ---header "Content-Type: application/json" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---data-raw '{ - "bidirection": true, - "destCluster": "pulsar-aws-uswest2", - "email": "", - "namespace": "ns0", - "originCluster": "pulsar-aws-useast2" -}' + --header "Content-Type: application/json" \ + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --data-raw '{ + "bidirection": true, + "destCluster": "pulsar-aws-uswest2", + "email": "", + "namespace": "ns0", + "originCluster": "pulsar-aws-useast2" + }' ---- You can get the clusters, email, and namespace from <>. @@ -315,18 +311,18 @@ You can get the clusters, email, and namespace from <>. Specify the replication to remove by providing the entire georeplication settings object, including the origin and destination clusters, the namespace, bidirectional flag, and email address (if set for the tenant): -[source,curl] +[source,bash] ---- curl -sS --fail -L -X DELETE "https://api.astra.datastax.com/v2/streaming/replications/$INSTANCE/$TENANT/$NAMESPACE" \ ---header "Content-Type: application/json" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---data-raw '{ - "bidirection": true, - "destCluster": "pulsar-aws-uswest2", - "email": "", - "namespace": "ns0", - "originCluster": "pulsar-aws-useast2" -}' + --header "Content-Type: application/json" \ + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --data-raw '{ + "bidirection": true, + "destCluster": "pulsar-aws-uswest2", + "email": "", + "namespace": "ns0", + "originCluster": "pulsar-aws-useast2" + }' ---- You can get the clusters, email, and namespace from <>. @@ -346,12 +342,12 @@ For {astra} application tokens, use the core xref:astra-db-serverless:api-refere Get a list of token IDs for a given tenant and cluster: -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v2/streaming/tenants/$TENANT/tokens" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---header "X-DataStax-Pulsar-Cluster: $CLUSTER" \ -| python3 -mjson.tool + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --header "X-DataStax-Pulsar-Cluster: $CLUSTER" \ + | python3 -mjson.tool ---- You can get the cluster name from <>. @@ -359,7 +355,6 @@ You can get the cluster name from <>. The result is a list of token objects, each containing the token ID and other metadata. You can use the token ID to <>. -.Result [source,json] ---- [ @@ -379,21 +374,16 @@ Token IDs are also listed in the {astra-ui} for each tenant and cluster. Get the raw string JWT associated with a specific token ID and cluster: -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v2/streaming/tenants/$TENANT/tokens/**PULSAR_TOKEN_ID**" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---header "X-DataStax-Pulsar-Cluster: $CLUSTER" + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --header "X-DataStax-Pulsar-Cluster: $CLUSTER" ---- You can get token IDs from <>, and you can get the cluster name from <>. -.Result -[source,json] ----- -Output: Raw string JWT -eyJhbGciOiJSUzI1NiIsI... ----- +The output is the raw JWT as a string. === Create a {pulsar-short} token @@ -403,43 +393,38 @@ You can get the cluster name from <>. v2 endpoint:: The `v2/streaming/tenants/$TENANT/tokens` endpoint returns a raw string JWT with an unlimited lifetime: + -[source,curl] +[source,bash] ---- curl -sS --fail -L -X POST "https://api.astra.datastax.com/v2/streaming/tenants/$TENANT/tokens" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---header "X-DataStax-Pulsar-Cluster: $CLUSTER" + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --header "X-DataStax-Pulsar-Cluster: $CLUSTER" ---- + -.Result -[source,json] ----- -Output: new raw string JWT -eyJhbGciOiJSUzI1NiIsI... ----- +The output is the raw JWT as a string. v3 endpoint:: The `v3/streaming/tenants/$TENANT/tokens` endpoint returns a JSON response and accepts additional token parameters: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X POST "https://api.astra.datastax.com/v3/streaming/tenants/$TENANT/tokens" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---header "X-DataStax-Pulsar-Cluster: $CLUSTER" \ ---data-raw '{ - "exp": "**EXPIRATION**", - "role": "**ROLE**", - "type": "**TYPE**" -}' + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --header "X-DataStax-Pulsar-Cluster: $CLUSTER" \ + --data-raw '{ + "exp": "**EXPIRATION**", + "role": "**ROLE**", + "type": "**TYPE**" + }' ---- + Replace the following: + +-- * **`EXPIRATION`**: The token lifetime expressed as a number and unit, such as `1d` for one day. * **`ROLE`**: A string describing the token's role. * **`TYPE`**: The token's privilege level, either `admin` or `non-admin`. - +-- + -.Result [source,json] ---- { @@ -453,11 +438,11 @@ Replace the following: Delete a token for a given cluster by token ID: -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X DELETE "https://api.astra.datastax.com/v2/streaming/tenants/$TENANT/tokens/**PULSAR_TOKEN_ID**" \ ---header "Authorization: Bearer $ASTRA_TOKEN" \ ---header "X-DataStax-Pulsar-Cluster: $CLUSTER" + --header "Authorization: Bearer $ASTRA_TOKEN" \ + --header "X-DataStax-Pulsar-Cluster: $CLUSTER" ---- You can get the token ID from <>, and you can get the cluster name from <>. diff --git a/modules/developing/pages/astream-cdc.adoc b/modules/developing/pages/astream-cdc.adoc index 38433cc..2ba7c61 100644 --- a/modules/developing/pages/astream-cdc.adoc +++ b/modules/developing/pages/astream-cdc.adoc @@ -249,7 +249,6 @@ select * from default_keyspace.cdc_demo; + Currently, the table has no rows: + -.Result [source,console] ---- key | c1 @@ -295,11 +294,11 @@ It supports initial CDC setup and extending CDC to new tables in databases where + . Use `{devops-api-ref-url}#tag/Database-Operations/operation/getCDC[GET /v3/databases/**DB_ID**/cdc]` to check the database's existing CDC configuration: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v3/databases/**DB_ID**/cdc" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" ---- + Replace the following: @@ -313,38 +312,38 @@ If the database has an existing CDC configuration, copy the `databaseName`, `tab + For databases where you previously enabled CDC, you only need to include new tables and regions in this `POST` request. + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X POST "https://api.astra.datastax.com/v3/databases/**DB_ID**/cdc" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" \ ---data '{ - "databaseName": "**DB_NAME**", - "tables": [ - { - "tableName": "**TABLE_NAME**", - "keyspaceName": "**KEYSPACE_NAME**" - }, - { - "tableName": "**TABLE_NAME**", - "keyspaceName": "**KEYSPACE_NAME**" - } - ], - "regions": [ - { - "datacenterID": "**DB_ID**-**REGION_SUFFIX**", - "datacenterRegion": "**REGION_NAME**", - "streamingClusterName": "**STREAMING_CLUSTER_NAME**", - "streamingTenantName": "**STREAMING_TENANT_NAME**" - }, - { - "datacenterID": "**DB_ID**-**REGION_SUFFIX**", - "datacenterRegion": "**REGION_NAME**", - "streamingClusterName": "**STREAMING_CLUSTER_NAME**", - "streamingTenantName": "**STREAMING_TENANT_NAME**" - } - ] -}' + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" \ + --data '{ + "databaseName": "**DB_NAME**", + "tables": [ + { + "tableName": "**TABLE_NAME**", + "keyspaceName": "**KEYSPACE_NAME**" + }, + { + "tableName": "**TABLE_NAME**", + "keyspaceName": "**KEYSPACE_NAME**" + } + ], + "regions": [ + { + "datacenterID": "**DB_ID**-**REGION_SUFFIX**", + "datacenterRegion": "**REGION_NAME**", + "streamingClusterName": "**STREAMING_CLUSTER_NAME**", + "streamingTenantName": "**STREAMING_TENANT_NAME**" + }, + { + "datacenterID": "**DB_ID**-**REGION_SUFFIX**", + "datacenterRegion": "**REGION_NAME**", + "streamingClusterName": "**STREAMING_CLUSTER_NAME**", + "streamingTenantName": "**STREAMING_TENANT_NAME**" + } + ] + }' ---- + Provide the following: @@ -366,26 +365,26 @@ You can get these names with the xref:apis:index.adoc#astra-streaming-devops-api `/v2/databases/$DB_ID/datacenters`:: Use `{devops-api-ref-url}#tag/Database-Operations/operation/addDatacenters[POST /v2/databases/**DB_ID**/datacenters]` to add a region to a multi-region database _and_ enable CDC in the same command: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X POST "https://api.astra.datastax.com/v2/databases/**DB_ID**/datacenters" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" \ ---data '{ - [ - { - "tier": "serverless", - "status": "ACTIVE", - "cloudProvider": "**CLOUD_PROVIDER**", - "region": "**REGION_NAME**", - "pcuGroupUUID": "**PCU_GROUP_ID**", - "streamingTenant": { - "streamingClusterName": "**STREAMING_CLUSTER_NAME**", - "streamingTenantName": "**STREAMING_TENANT_NAME**" + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" \ + --data '{ + [ + { + "tier": "serverless", + "status": "ACTIVE", + "cloudProvider": "**CLOUD_PROVIDER**", + "region": "**REGION_NAME**", + "pcuGroupUUID": "**PCU_GROUP_ID**", + "streamingTenant": { + "streamingClusterName": "**STREAMING_CLUSTER_NAME**", + "streamingTenantName": "**STREAMING_TENANT_NAME**" + } } - } - ] -}' + ] + }' ---- + Replace the following: @@ -489,7 +488,6 @@ select * from default_keyspace.cdc_demo; + The demo table now has two rows: + -.Result [source,console] ---- key | c1 @@ -504,10 +502,10 @@ The demo table now has two rows: + For example, if you have an Elasticsearch sink, you can send a `GET` request to your Elasticsearch deployment: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X POST "**ELASTICSEARCH_URL**/**INDEX_NAME**/_search?pretty" \ --header "Authorization: ApiKey '**API_KEY**'" + --header "Authorization: ApiKey '**API_KEY**'" ---- + Replace `**ELASTICSEARCH_URL**`, `**INDEX_NAME**`, and `**API_KEY**` with the values from your https://cloud.elastic.co/[Elasticsearch deployment] that you used to <>. @@ -664,7 +662,6 @@ bin/pulsar-admin topics partitioned-stats persistent://**DATA_TOPIC_STRING** + Replace `**DATA_TOPIC_STRING**` with the `data-` topic string in the format of `**TENANT_NAME**/astracdc/data-**DB_ID**-**KEYSPACE_NAME**.**TABLE_NAME**`. + -.Result [source,json] ---- { @@ -780,11 +777,11 @@ You can check your active CDC configurations in the {astra-ui} or with the {astr By database:: Use `{devops-api-ref-url}#tag/Database-Operations/operation/getCDC[GET /v3/databases/**DB_ID**/cdc]` to get CDC configuration details for an entire database: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v3/databases/**DB_ID**/cdc" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" ---- + Replace `**DB_ID**` with the xref:astra-db-serverless:databases:create-database.adoc#get-db-id[database ID], and replace `**APPLICATION_TOKEN**` with an xref:astra-db-serverless:administration:manage-application-tokens.adoc[{astra-db} application token]. @@ -826,11 +823,11 @@ For multi-region databases with CDC enabled in multiple regions, the `regions` a By table:: Use `{devops-api-ref-url}#tag/Database-Operations/operation/getCDC[GET /v3/databases/**DB_ID**/keyspaces/**KEYSPACE_NAME**/tables/**TABLE_NAME**/cdc]` to get CDC configuration details for a specific table: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v3/databases/**DB_ID**/keyspaces/**KEYSPACE_NAME**/tables/**TABLE_NAME**/cdc" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" ---- + In the path, replace `**DB_ID**` with the xref:astra-db-serverless:databases:create-database.adoc#get-db-id[database ID], and replace `**KEYSPACE_NAME**` and `**TABLE_NAME**` with the name of the keyspace and table where you want to check the CDC configuration. @@ -872,11 +869,11 @@ If CDC is already enabled, the request updates the existing CDC configuration. . Use `{devops-api-ref-url}#tag/Database-Operations/operation/getCDC[GET /v3/databases/**DB_ID**/cdc]` to get the current CDC configuration details for the database: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v3/databases/**DB_ID**/cdc" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" ---- + Replace `**DB_ID**` with the xref:astra-db-serverless:databases:create-database.adoc#get-db-id[database ID], and replace `**APPLICATION_TOKEN**` with an xref:astra-db-serverless:administration:manage-application-tokens.adoc[{astra-db} application token]. @@ -944,36 +941,36 @@ Make sure that you include all existing tables and regions that you want to keep If you omit any existing tables or regions from the request, CDC is disabled for those tables or regions. ==== + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X PUT "https://api.astra.datastax.com/v3/databases/**DB_ID**/cdc" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" \ ---data '{ - "databaseName": "test_db", - "tables": [ - { - "tableName": "table1", - "keyspaceName": "default_keyspace" - }, - { - "tableName": "table2", - "keyspaceName": "default_keyspace" - }, - { - "tableName": "table3", - "keyspaceName": "other_keyspace" - } - ], - "regions": [ - { - "datacenterID": "1234-5678-90080012-2323-1", - "datacenterRegion": "us-east1", - "streamingClusterName": "pulsar-gcp-useast1-dev", - "streamingTenantName": "cdc-streaming" - } - ] -}' + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" \ + --data '{ + "databaseName": "test_db", + "tables": [ + { + "tableName": "table1", + "keyspaceName": "default_keyspace" + }, + { + "tableName": "table2", + "keyspaceName": "default_keyspace" + }, + { + "tableName": "table3", + "keyspaceName": "other_keyspace" + } + ], + "regions": [ + { + "datacenterID": "1234-5678-90080012-2323-1", + "datacenterRegion": "us-east1", + "streamingClusterName": "pulsar-gcp-useast1-dev", + "streamingTenantName": "cdc-streaming" + } + ] + }' ---- + Replace the following: @@ -984,11 +981,11 @@ Replace the following: . To verify that the changes were applied as expected, use `{devops-api-ref-url}#tag/Database-Operations/operation/getCDC[GET /v3/databases/**DB_ID**/cdc]` to get the new CDC configuration details for the database: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v3/databases/**DB_ID**/cdc" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" ---- . If you removed any tables from the existing CDC configuration, decide whether you want to delete the associated namespace, topics, and {product} tenant. @@ -1028,20 +1025,20 @@ Deleting the namespace also deletes the topics within that namespace. Use `{devops-api-ref-url}#tag/Database-Operations/operation/deleteCDC[DELETE /v3/databases/**DB_ID**/cdc]` to disable CDC for a table: -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X DELETE "https://api.astra.datastax.com/v3/databases/**DB_ID**/cdc" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" \ ---data '{ - "databaseID": "**DB_ID**", - "tables": [ - { - "tableName": "**TABLE_NAME**", - "keyspaceName": "**KEYSPACE_NAME**" - } - ] -}' + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" \ + --data '{ + "databaseID": "**DB_ID**", + "tables": [ + { + "tableName": "**TABLE_NAME**", + "keyspaceName": "**KEYSPACE_NAME**" + } + ] + }' ---- Replace the following: diff --git a/modules/developing/pages/astream-functions.adoc b/modules/developing/pages/astream-functions.adoc index 6718c39..e65b22f 100644 --- a/modules/developing/pages/astream-functions.adoc +++ b/modules/developing/pages/astream-functions.adoc @@ -93,7 +93,6 @@ To deploy a Java function, you must create a `.jar` file. . Declare dependencies in `pom.xml`: + -.pom.xml [source,xml] ---- @@ -161,7 +160,6 @@ To deploy a Java function, you must create a `.jar` file. mvn package ---- + -.Result [source,console] ---- [INFO] ------------------------------------------------------------------------ @@ -179,9 +177,8 @@ mvn package . If you haven't done so already, xref:developing:configure-pulsar-env.adoc[set up your environment for the {pulsar-short} binaries]. -. Create a deployment configuration YAML file that defines the function metadata and associated topics: +. Create a deployment configuration YAML file, such as `func-create-config.yaml`,that defines the function metadata and associated topics: + -.func-create-config.yaml [source,yaml,subs="+quotes"] ---- py: **PATH_TO_FUNCTION_ARCHIVE** @@ -240,15 +237,15 @@ If your Python function contains only a single script and no dependencies, you c [source,bash,subs="+quotes"] ---- ./pulsar-admin functions create \ ---py **PATH_TO_PYTHON_FILE** \ ---classname **FILE_NAME**.**CLASS_NAME** \ ---tenant **TENANT_NAME** \ ---namespace **NAMESPACE_NAME** \ ---name **DISPLAY_NAME** \ ---auto-ack true \ ---inputs persistent://**TENANT_NAME**/**NAMESPACE_NAME**/**TOPIC_NAME** \ ---output persistent://**TENANT_NAME**/**NAMESPACE_NAME**/**TOPIC_NAME** \ ---log-topic persistent://**TENANT_NAME**/**NAMESPACE_NAME**/**TOPIC_NAME** + --py **PATH_TO_PYTHON_FILE** \ + --classname **FILE_NAME**.**CLASS_NAME** \ + --tenant **TENANT_NAME** \ + --namespace **NAMESPACE_NAME** \ + --name **DISPLAY_NAME** \ + --auto-ack true \ + --inputs persistent://**TENANT_NAME**/**NAMESPACE_NAME**/**TOPIC_NAME** \ + --output persistent://**TENANT_NAME**/**NAMESPACE_NAME**/**TOPIC_NAME** \ + --log-topic persistent://**TENANT_NAME**/**NAMESPACE_NAME**/**TOPIC_NAME** ---- If there isn't a class in the file, the `className` is only the filename without the extension. @@ -360,8 +357,8 @@ If the function produces output and it has an `output` topic, the output data is [source,bash] ---- ./pulsar-client consume persistent://**TENANT_NAME**/**NAMESPACE_NAME**/**TOPIC_NAME** \ ---subscription-name my-subscription \ ---num-messages 0 # Listen indefinitely + --subscription-name my-subscription \ + --num-messages 0 # Listen indefinitely ---- + . Test your function: @@ -369,10 +366,10 @@ If the function produces output and it has an `output` topic, the output data is [source,bash] ---- ./pulsar-admin functions trigger \ ---name **FUNCTION_DISPLAY_NAME** \ ---tenant **TENANT_NAME** \ ---namespace **NAMESPACE_NAME** \ ---trigger-value "**MESSAGE**" + --name **FUNCTION_DISPLAY_NAME** \ + --tenant **TENANT_NAME** \ + --namespace **NAMESPACE_NAME** \ + --trigger-value "**MESSAGE**" ---- + The trigger sends the message string to the function. diff --git a/modules/developing/pages/clients/csharp-produce-consume.adoc b/modules/developing/pages/clients/csharp-produce-consume.adoc index 54a698a..2655327 100644 --- a/modules/developing/pages/clients/csharp-produce-consume.adoc +++ b/modules/developing/pages/clients/csharp-produce-consume.adoc @@ -29,7 +29,6 @@ dotnet add package DotPulsar --version 2.7.0 . Enter the following code, starting at line 1: + -.Program.cs [source,csharp] ---- using DotPulsar; @@ -62,7 +61,6 @@ include::ROOT:partial$client-variables-table.adoc[] . Create a new `producer` instance, and then instruct it to produce a message in a string format. Messages can also be in formats like JSON, byte, and AVRO. + -.Program.cs [source,csharp] ---- await using var producer = client.NewProducer(Schema.String) @@ -72,7 +70,6 @@ await using var producer = client.NewProducer(Schema.String) . Asynchronously send a single message and wait for acknowledgment: + -.Program.cs [source,csharp] ---- await producer.Send("Hello World"); // Send a message and ignore the returned MessageId @@ -82,7 +79,6 @@ Console.WriteLine("Sent message"); . Create a new `consumer` instance and instruct it to expect messages in string format. The consumer uses broker subscriptions to gather messages. + -.Program.cs [source,csharp] ---- await using var consumer = client.NewConsumer(Schema.String) @@ -97,7 +93,6 @@ In this example, the subscription starts with the `Earliest` unacknowledged mess . Loop through the messages provided to the consumer's subscription and write their contents: + -.Program.cs [source,csharp] ---- var msgCount = 0; diff --git a/modules/developing/pages/clients/golang-produce-consume.adoc b/modules/developing/pages/clients/golang-produce-consume.adoc index bed2886..7f34663 100644 --- a/modules/developing/pages/clients/golang-produce-consume.adoc +++ b/modules/developing/pages/clients/golang-produce-consume.adoc @@ -38,7 +38,6 @@ The new project includes a `main` file and the retrieved {pulsar-short} client p . Open the `main.go` file, and then add the following code to import the required packages, establish an entry point for the application, and create a new instance of a {pulsar-short} client: + -.main.go [source,golang] ---- package main @@ -85,7 +84,6 @@ include::ROOT:partial$client-variables-table.adoc[] . Use the client to create a producer: + -.main.go [source,golang] ---- log.Printf("creating producer...") @@ -110,7 +108,6 @@ The full topic address includes the tenant, namespace, and topic names, as well . Create a `Hello World` message and send it asynchronously: + -.main.go [source,golang] ---- asyncMsg := pulsar.ProducerMessage{ @@ -129,7 +126,6 @@ The full topic address includes the tenant, namespace, and topic names, as well . Use the client to create a new xref:ROOT:astream-subscriptions.adoc[subscription instance] to receive the message: + -.main.go [source,golang] ---- consumer, err := client.Subscribe(pulsar.ConsumerOptions{ @@ -152,7 +148,6 @@ In this example, the subscription starts with the `Earliest` unacknowledged mess . Make the consumer receive and acknowledge (`ack`) messages provided to the subscription: + -.main.go [source,golang] ---- msg, err := consumer.Receive(ctx) diff --git a/modules/developing/pages/clients/java-produce-consume.adoc b/modules/developing/pages/clients/java-produce-consume.adoc index fadcd92..6b8877e 100644 --- a/modules/developing/pages/clients/java-produce-consume.adoc +++ b/modules/developing/pages/clients/java-produce-consume.adoc @@ -30,7 +30,6 @@ mvn archetype:generate \ . Add the {pulsar-short} client dependency in `pom.xml`: + -.pom.xml [source,xml] ---- @@ -43,7 +42,6 @@ mvn archetype:generate \ . For this example, add the following build target in `pom.xml`. This example creates a single artifact. + -.pom.xml [source,xml] ---- @@ -67,7 +65,6 @@ This example creates a single artifact. . If necessary, specify the compiler versions in `pom.xml`: + -.pom.xml [source,xml] ---- @@ -82,7 +79,6 @@ This example creates a single artifact. . Remove any existing content from the file, and then add the following code that imports dependencies, creates a client instance, and configures the instance to use your {product} tenant: + -./src/main/java/org/example/App.java [source,java] ---- package org.example; @@ -121,7 +117,6 @@ include::ROOT:partial$client-variables-table.adoc[] . Use the client to create a producer. The producer builds on the client configuration for directions about what topic to produce messages to. + -./src/main/java/org/example/App.java [source,java] ---- Producer producer = client.newProducer(Schema.STRING) @@ -131,7 +126,6 @@ The producer builds on the client configuration for directions about what topic . Asynchronously send a single message to the broker and wait for acknowledgment, and close the producer: + -./src/main/java/org/example/App.java [source,java] ---- producer.send("Hello World"); @@ -142,7 +136,6 @@ The producer builds on the client configuration for directions about what topic . Create a new consumer instance. This code directs the consumer to watch a certain topic, identifies the subscription for watching topics, and begins the subscription. + -./src/main/java/org/example/App.java [source,java] ---- Consumer consumer = client.newConsumer(Schema.STRING) @@ -153,7 +146,6 @@ This code directs the consumer to watch a certain topic, identifies the subscrip . Receive the messages added by the producer: + -./src/main/java/org/example/App.java [source,java] ---- boolean receivedMsg = false; @@ -176,7 +168,6 @@ This code directs the consumer to watch a certain topic, identifies the subscrip . Clean up and close the class: + -./src/main/java/org/example/App.java [source,java] ---- consumer.close(); diff --git a/modules/developing/pages/clients/nodejs-produce-consume.adoc b/modules/developing/pages/clients/nodejs-produce-consume.adoc index c1ed025..6e1f337 100644 --- a/modules/developing/pages/clients/nodejs-produce-consume.adoc +++ b/modules/developing/pages/clients/nodejs-produce-consume.adoc @@ -18,7 +18,8 @@ Install the {cpp} {pulsar-short} library dependency required by the Node.js {pul {pulsar-short} Node client versions 1.8 and later don't require installation of the {cpp} {pulsar-short} library dependency. -.Ubuntu-based Debian +Ubuntu-based Debian:: ++ [source,shell] ---- wget https://archive.apache.org/dist/pulsar/pulsar-2.10.2/DEB/apache-pulsar-client.deb @@ -30,7 +31,8 @@ sudo apt install -y ./apache-pulsar-client*.deb sudo ldconfig ---- -.Centos/RHEL-based rpm +Centos/RHEL-based rpm:: ++ [source,shell] ---- wget https://archive.apache.org/dist/pulsar/pulsar-2.10.2/RPMS/apache-pulsar-client-2.10.2-1.x86_64.rpm @@ -62,7 +64,6 @@ npm install pulsar-client . In your new project, open the `index.js` file, and then add the following code. + -.index.js [source,javascript] ---- const Pulsar = require("pulsar-client"); @@ -104,7 +105,6 @@ include::ROOT:partial$client-variables-table.adoc[] There are many configuration options for producers. For this example, declare the topic where messages should go. + -.index.js [source,javascript] ---- const producer = await client.createProducer({ @@ -114,7 +114,6 @@ For this example, declare the topic where messages should go. . Send a message and receive acknowledgment: + -.index.js [source,javascript] ---- producer.send({ @@ -125,7 +124,6 @@ For this example, declare the topic where messages should go. . Clean up: + -.index.js [source,javascript] ---- await producer.flush(); @@ -136,7 +134,6 @@ At this point, the script produces a message that waits to be consumed and ackno . Create a new consumer subscription, name the subscription, and declare the topic to watch: + -.index.js [source,javascript] ---- const consumer = await client.subscribe({ @@ -149,7 +146,6 @@ At this point, the script produces a message that waits to be consumed and ackno . Receive messages, write them to the console, and acknowledge receipt with the broker: + -.index.js [source,javascript] ---- const msg = await consumer.receive(); @@ -159,7 +155,6 @@ At this point, the script produces a message that waits to be consumed and ackno . Clean up: + -.index.js [source,javascript] ---- await consumer.close(); @@ -169,7 +164,6 @@ At this point, the script produces a message that waits to be consumed and ackno The complete `index.js` script is as follows: -.index.js [source,javascript] ---- const Pulsar = require("pulsar-client"); diff --git a/modules/developing/pages/clients/python-produce-consume.adoc b/modules/developing/pages/clients/python-produce-consume.adoc index f525452..d2b6432 100644 --- a/modules/developing/pages/clients/python-produce-consume.adoc +++ b/modules/developing/pages/clients/python-produce-consume.adoc @@ -34,7 +34,6 @@ Your IDE might show errors until you have completed the script. . Import the required libraries, and then create a {pulsar-short} client instance with the topic URL and token authentication: + -.index.py [source,python] ---- import pulsar @@ -58,7 +57,6 @@ include::ROOT:partial$client-variables-table.adoc[] . Use the client to create a producer: + -.index.py [source,python] ---- producer = client.create_producer(topic) @@ -66,7 +64,6 @@ producer = client.create_producer(topic) . Send a message: + -.index.py [source,python] ---- producer.send('Hello World'.encode('utf-8')) @@ -74,7 +71,6 @@ producer.send('Hello World'.encode('utf-8')) . Use the Python client instance to create a consumer subscription to the same topic that you sent a message to: + -.index.py [source,python] ---- consumer = client.subscribe(topic, 'my-subscription') @@ -82,7 +78,6 @@ consumer = client.subscribe(topic, 'my-subscription') . Iterate through messages and write their data: + -.index.py [source,python] ---- waitingForMsg = True @@ -110,7 +105,6 @@ client.close() The complete `index.py` script is as follows: -.index.py [source,python] ---- import pulsar diff --git a/modules/developing/pages/clients/spring-produce-consume.adoc b/modules/developing/pages/clients/spring-produce-consume.adoc index 109ebd2..03fc2e3 100644 --- a/modules/developing/pages/clients/spring-produce-consume.adoc +++ b/modules/developing/pages/clients/spring-produce-consume.adoc @@ -26,9 +26,9 @@ image::developing:spring-initializr.png[Spring Initializr] . Click *Generate Project*, download the zip file, and then extract it. . Navigate to `src/main/java`, and then open the `DemoApplication.java` file. ++ This file contains the main method that will run your application with the specified dependencies. + -.DemoApplication.java [source,java] ---- package com.example.demo; @@ -98,7 +98,6 @@ public class DemoApplication . Replace the following values in `DemoApplication.java` with values from your tenant's *Connect* tab in the {astra-ui}: + -.DemoApplication.java [source,java,subs="+quotes"] ---- private static final String serviceUrl = "pulsar+ssl://pulsar-**PROVIDER**-**REGION**.streaming.datastax.com:**PORT**"; @@ -137,7 +136,6 @@ token: *** mvn clean compile ---- + -.Result [source,console] ---- [INFO] ------------------------------------------------------------------------ @@ -155,7 +153,6 @@ mvn clean compile mvn spring-boot:run ---- + -.Result [source,console] ---- Message received: Hello World diff --git a/modules/developing/pages/configure-pulsar-env.adoc b/modules/developing/pages/configure-pulsar-env.adoc index 7d0b348..63b6991 100644 --- a/modules/developing/pages/configure-pulsar-env.adoc +++ b/modules/developing/pages/configure-pulsar-env.adoc @@ -19,6 +19,7 @@ For example, to download {pulsar-short} version `3.1.0`, use patch version `0`. [source,bash,subs="+quotes,+attributes"] ---- PULSAR_VERSION="{pulsar-version}.**PATCH**" + wget https://archive.apache.org/dist/pulsar/pulsar-$PULSAR_VERSION/apache-pulsar-$PULSAR_VERSION-bin.tar.gz ---- @@ -29,7 +30,7 @@ wget https://archive.apache.org/dist/pulsar/pulsar-$PULSAR_VERSION/apache-pulsar tar xvfz apache-pulsar-$PULSAR_VERSION-bin.tar.gz ---- + -The resulting directory is named `apache-pulsar-**PULSAR_VERSON**`. +The resulting directory is named `apache-pulsar-**PULSAR_VERSION**`. This directory is known as the _{pulsar-short} folder_. {pulsar-short} guides assume you are working within this directory. {pulsar-short} commands prefixed by `./bin` indicate the command is run from within the {pulsar-short} folder. diff --git a/modules/developing/pages/gpt-schema-translator.adoc b/modules/developing/pages/gpt-schema-translator.adoc index 25bf719..e405004 100644 --- a/modules/developing/pages/gpt-schema-translator.adoc +++ b/modules/developing/pages/gpt-schema-translator.adoc @@ -37,13 +37,11 @@ If this button isn't available, the GPT schema translator doesn't have an availa + image::developing:two-schemas.png[Schema mapping,320,240] + -.{cass-short}-to-{pulsar-short} schema mapping example -[cols="1,1,1"] -|=== -| {cass-short} table schema | {pulsar-short} JSON schema | Generated mapping - -a| -[source,cql] +The following examples show the progression of the {cass-short}-to-{pulsar-short} schema mapping: ++ +{cass-short} table schema as JSON:: ++ +[source,json] ---- { "primaryKey": { @@ -75,8 +73,9 @@ a| ] } ---- - -a| ++ +{pulsar-short} JSON schema:: ++ [source,json] ---- { @@ -111,15 +110,14 @@ a| ] } ---- - -a| -[source,console] ++ +Generated mapping:: ++ +[source,plaintext] ---- id=key, file1=value.file1, file2=value.file2, file3=value.file3 ---- -|=== - . Save the mapping configuration. . After configuring your {astra-db} sink connector for the given topic and table, messages should flow between the two schemas without error. @@ -130,9 +128,8 @@ You can check the {astra-ui} logs to confirm that the data is flowing into your This example demonstrates how you can generates schema mapping in real time. -. The xref:connectors:sources/data-generator.adoc[Data Generator source connector] generates data for a {pulsar-short} topic with an AVRO schema. +. The xref:connectors:sources/data-generator.adoc[Data Generator source connector] generates data for a {pulsar-short} topic with an AVRO schema: + -.AVRO schema example [source,json] ---- "pulsar_topic_schema": { @@ -365,10 +362,9 @@ This example demonstrates how you can generates schema mapping in real time. } ---- -. The {astra-db} sink connector writes data to the {cass-short} table with a CQL schema. +. The {astra-db} sink connector writes data to the {cass-short} table with the equivalent CQL schema: + -.CQL schema example -[source,cql] +[source,json] ---- "cassandra_table_schemas": { "person": { @@ -410,7 +406,7 @@ image::developing:create-schema-mapping.png[Schema mapping,320,240] + The {gpt-schema-translator} generates an AVRO-to-CQL schema mapping while messages are processed. + -[source,console] +[source,plaintext] ---- passportnumber=value.passportNumber, age=value.age, firstname=value.firstName, lastname=value.lastName ---- @@ -424,10 +420,25 @@ If you don't declare a schema in the {pulsar-short} topic, the schema translator When you create the mapping in the {astra-ui}, you can click *Generate Mapping* to create a generic {pulsar-short} topic schema based on your {cass-short} table schema. If schema mapping isn't possible for the selected table and topic, the *Generate Mapping* button isn't available. -For example, assume you have the following {cass-short} table schema: +The following examples show the same schema in CQL, CQL-as-JSON, and the generated JSON schema mapping: +{cass-short} table schema in CQL:: ++ [source,cql] ---- +CREATE TABLE passport_data ( + passportnumber text, + age varint, + firstname text, + lastname text, + PRIMARY KEY (passportnumber) +); +---- + +{cass-short} table schema in JSON:: ++ +[source,json] +---- { "primaryKey": { "partitionKey": [ @@ -459,8 +470,9 @@ For example, assume you have the following {cass-short} table schema: } ---- -The schema translator would generate the following {pulsar-short} JSON schema mapping based on the given {cass-short} table schema: - +Translated {pulsar-short} JSON schema mapping:: +The schema translator generates the following {pulsar-short} JSON schema mapping based on the {cass-short} table schema: ++ [source,console] ---- passportnumber=value.passportnumber, age=value.age, firstname=value.firstname, lastname=value.lastname diff --git a/modules/developing/pages/produce-consume-pulsar-client.adoc b/modules/developing/pages/produce-consume-pulsar-client.adoc index 7d91992..821b526 100644 --- a/modules/developing/pages/produce-consume-pulsar-client.adoc +++ b/modules/developing/pages/produce-consume-pulsar-client.adoc @@ -31,9 +31,9 @@ For example: [source,shell] ---- ./bin/pulsar-client produce \ -"$TENANT/$NAMESPACE/$TOPIC" \ ---messages "Hi there" \ ---num-produce 1 + "$TENANT/$NAMESPACE/$TOPIC" \ + --messages "Hi there" \ + --num-produce 1 ---- + Your topic now has a new, unacknowledged message. @@ -43,9 +43,9 @@ Your topic now has a new, unacknowledged message. [source,shell] ---- ./bin/pulsar-client consume \ -"$TENANT/$NAMESPACE/$TOPIC" \ ---subscription-name "examples-subscriber" \ ---num-messages 1 + "$TENANT/$NAMESPACE/$TOPIC" \ + --subscription-name "examples-subscriber" \ + --num-messages 1 ---- . Make sure the output includes your message: diff --git a/modules/getting-started/pages/real-time-data-pipelines-tutorial.adoc b/modules/getting-started/pages/real-time-data-pipelines-tutorial.adoc index f3a6f42..ccac0e1 100644 --- a/modules/getting-started/pages/real-time-data-pipelines-tutorial.adoc +++ b/modules/getting-started/pages/real-time-data-pipelines-tutorial.adoc @@ -137,11 +137,8 @@ CREATE TABLE click_data.product_clicks ( + The console prints create statements describing the keyspace itself and the two tables. + -.Result [source,console] ---- -token@cqlsh> describe click_data; - CREATE KEYSPACE click_data WITH replication = {'class': 'NetworkTopologyStrategy', 'us-east-1': '3'} AND durable_writes = true; CREATE TABLE click_data.all_clicks ( @@ -552,13 +549,12 @@ This means the sink consumed a message from the topic, and then wrote the data t select * from click_data.all_clicks; ---- + +Optionally, run `EXPAND ON` before your `SELECT` statement for a more readable output. ++ The result should look similar to the following: + [source,console] ---- -token@cqlsh> EXPAND ON; //this cleans up the output -Now Expanded output is enabled -token@cqlsh> select * from click_data.all_clicks; @ Row 1 ------------------+---------------------------------------- operating_system | Windows diff --git a/modules/operations/pages/astream-georeplication.adoc b/modules/operations/pages/astream-georeplication.adoc index 4b8c5b4..937bf61 100644 --- a/modules/operations/pages/astream-georeplication.adoc +++ b/modules/operations/pages/astream-georeplication.adoc @@ -63,7 +63,6 @@ bin/pulsar-admin tenants get **TENANT_NAME** + In the result, the `allowedClusters` are the clusters where you can replicate the tenant: + -.Result [source,json] ---- { @@ -79,7 +78,6 @@ In the result, the `allowedClusters` are the clusters where you can replicate th bin/pulsar-admin namespaces get-clusters **TENANT_NAME**/**NAMESPACE_NAME** ---- + -.Result [source,json] ---- pulsar-aws-useast1-staging @@ -102,7 +100,6 @@ bin/pulsar-client produce **TENANT_NAME**/**NAMESPACE_NAME**/**TOPIC_NAME** --me + The consumer acknowledges the messages: + -.Result [source,console] ---- ----- got message ----- @@ -160,7 +157,6 @@ bin/pulsar-admin topics stats persistent://**TENANT_NAME**/**NAMESPACE_NAME**/** + In the configuration, `isReplicated` is now `true` for this subscription: + -.config.json [source,json] ---- { diff --git a/modules/operations/pages/astream-scrape-metrics.adoc b/modules/operations/pages/astream-scrape-metrics.adoc index b672cea..8d95d57 100644 --- a/modules/operations/pages/astream-scrape-metrics.adoc +++ b/modules/operations/pages/astream-scrape-metrics.adoc @@ -15,9 +15,8 @@ This doc will show you how to scrape an {product} tenant with Prometheus. . Click the name of your tenant, and then click the *Connect* tab. -. Click *Prometheus* to generate a new configuration file: +. Click *Prometheus* to generate a new `prometheus.yml` configuration file: + -.prometheus.yml [source,yaml,subs="+quotes"] ---- global: @@ -55,7 +54,7 @@ docker pull prom/prometheus . Bind-mount your modified `prometheus.yml` file by running the Prometheus Docker container with a modified path in the `-v` argument: + -[source,shell] +[source,shell,subs="+quotes"] ---- docker run \ -p 9090:9090 \ @@ -97,24 +96,26 @@ image::operations:astream-prometheus-graph.png[Scraping {product} with Prometheu With your Prometheus scrape container running, you can use curl commands to decompress your Prometheus scrape data: -.Deflate +Deflate:: ++ [source,bash] ---- curl -v \ --H "Authorization: Bearer $BEARER_TOKEN" \ --H "Accept-encoding: deflate" \ -https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 \ ---compressed + -H "Authorization: Bearer $BEARER_TOKEN" \ + -H "Accept-encoding: deflate" \ + https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 \ + --compressed ---- -.Gzip +Gzip:: ++ [source,bash] ---- curl -v \ --H "Authorization: Bearer $BEARER_TOKEN" \ --H "Accept-encoding: gzip" \ -https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 \ -| zcat + -H "Authorization: Bearer $BEARER_TOKEN" \ + -H "Accept-encoding: gzip" \ + https://prometheus-aws-useast1.streaming.datastax.com/pulsarmetrics/tenant-1 \ + | zcat ---- Deflate or Gzip will extract your scraped metrics in a format such as the following: diff --git a/modules/operations/pages/astream-token-gen.adoc b/modules/operations/pages/astream-token-gen.adoc index a5ce104..7aa956c 100644 --- a/modules/operations/pages/astream-token-gen.adoc +++ b/modules/operations/pages/astream-token-gen.adoc @@ -67,23 +67,21 @@ For the {devops-api}, the base URL is always `\https://api.astra.datastax.com`. + For example: + -.Billing report by tenant -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v2/stats/namespaces/$TENANT" \ ---header "Accept: application/json" \ ---header "Authorization: Bearer $ASTRA_TOKEN" + --header "Accept: application/json" \ + --header "Authorization: Bearer $ASTRA_TOKEN" ---- {product} {pulsar-short} Admin API:: To send requests to the xref:apis:api-operations.adoc[{product} {pulsar-short} Admin API], use your tenant's web service URL and a {pulsar-short} token. For example: + -.Get built-in sinks -[source,curl] +[source,bash] ---- curl -sS --fail -L -X GET "$WEB_SERVICE_URL/admin/v3/sinks/builtinsinks" \ --H "Authorization: $ASTRA_PULSAR_TOKEN" + --header "Authorization: Bearer $ASTRA_PULSAR_TOKEN" ---- OSS {pulsar-short} REST APIs:: @@ -92,15 +90,13 @@ To send requests to the xref:developing:using-curl.adoc[open source {pulsar} pro {pulsar-short} CLI:: Use a {pulsar-short} token to authenticate with the xref:developing:configure-pulsar-env.adoc[{pulsar-short} CLI]. + -The following commands use a {pulsar-short} token declared in the `pulsar-admin` binary configuration so the token doesn't have to be passed directly on the command line: +The following example commands use a {pulsar-short} token declared in the `pulsar-admin` binary configuration so the token doesn't have to be passed directly on the command line: + -.Monitor a topic's health [source,shell] ---- bin/pulsar-admin topics stats ---- + -.Monitor a connector's health [source,shell] ---- bin/pulsar-admin sinks status diff --git a/modules/operations/pages/monitoring/integration.adoc b/modules/operations/pages/monitoring/integration.adoc index 7647f2d..74c7489 100644 --- a/modules/operations/pages/monitoring/integration.adoc +++ b/modules/operations/pages/monitoring/integration.adoc @@ -23,7 +23,6 @@ For more information, see xref:operations:astream-scrape-metrics.adoc[Scrape met This example uses a job named `astra-pulsar-metrics-demo`. The values in your `config.yml` depend on your tenant configuration. + -.config.yml [source,yaml,subs="+quotes"] ---- - job_name: "astra-pulsar-metrics-demo" @@ -58,7 +57,6 @@ You can change the name of the YAML output file, if desired. . Create a customized values file (`custom-values.yml`) for the Prometheus Community Kubernetes Helm chart. In the 'additionalScrapeConfigsSecret' section, make sure the `name` and `key` match the name if your Kubernetes secret and config file name, such as `astra-secret` and `config.yml`. + -.custom-values.yml [source,yaml,subs="+quotes"] ---- prometheus: diff --git a/modules/operations/pages/monitoring/new-relic.adoc b/modules/operations/pages/monitoring/new-relic.adoc index 480c0f6..7ffbbf4 100644 --- a/modules/operations/pages/monitoring/new-relic.adoc +++ b/modules/operations/pages/monitoring/new-relic.adoc @@ -38,7 +38,6 @@ kubectl create secret generic nr-license-key --from-literal=value=**LICENSE_KEY* . In your extra Prometheus server's `custom-values.yaml` file, add the `remoteWrite` configuration to send local Prometheus metrics to New Relic through `remote_write`: + -.custom-values.yaml [source,yaml,subs="+quotes"] ---- prometheus: diff --git a/modules/operations/pages/monitoring/stream-audit-logs.adoc b/modules/operations/pages/monitoring/stream-audit-logs.adoc index 0e44247..0e4ce69 100644 --- a/modules/operations/pages/monitoring/stream-audit-logs.adoc +++ b/modules/operations/pages/monitoring/stream-audit-logs.adoc @@ -43,46 +43,45 @@ You can use the {devops-api-ref-url}#tag/Organization-Operations/operation/confi . Create the audit log streaming configuration using the values from the tenant's `client.conf` file and your {pulsar-short} configuration: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X POST "https://api.astra.datastax.com/v2/organizations/**ORG_ID**/telemetry/auditLogs" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" ---data '{ - "pulsar": { - "endpoint": "pulsar+**BROKER_SERVICE_URL**", - "auth_strategy": "token", - "topic": "**TOPIC_FULL_NAME**", - "auth_name": "token", - "token": "**PULSAR_AUTHENTICATION_TOKEN**" - } -}' + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" + --data '{ + "pulsar": { + "endpoint": "pulsar+**BROKER_SERVICE_URL**", + "auth_strategy": "token", + "topic": "**TOPIC_FULL_NAME**", + "auth_name": "token", + "token": "**PULSAR_AUTHENTICATION_TOKEN**" + } + }' ---- + Replace the following: + +-- * `**ORG_ID**`: Your {astra} organization ID. * `**APPLICATION_TOKEN**`: Your {astra} application token. * `**BROKER_SERVICE_URL**`: The {product} broker service URL, such as `ssl://pulsar-aws-useast2.streaming.datastax.com:6651`. * `**TOPIC_FULL_NAME**`: The full name of the {product} topic where you want to stream audit logs. * The `auth_strategy` and other authentication details depend on your {pulsar-short} configuration. +-- + -.Result -[source,json] ----- -HTTP/1.1 202 Accepted ----- +A successful request returns `202 Accepted`. . Retrieve and verify the audit log streaming configuration: + -[source,curl,subs="+quotes"] +[source,bash,subs="+quotes"] ---- curl -sS --fail -L -X GET "https://api.astra.datastax.com/v2/organizations/**ORG_ID**/telemetry/auditLogs" \ ---header "Authorization: Bearer **APPLICATION_TOKEN**" \ ---header "Accept: application/json" + --header "Authorization: Bearer **APPLICATION_TOKEN**" \ + --header "Accept: application/json" ---- + -.Result +The response should contain the same data as the previous `POST` request body: ++ [source,json] ---- { From 6ac1fe4bb6047ea83971d4cc79410a6bf01ef9ec Mon Sep 17 00:00:00 2001 From: April M <36110273+aimurphy@users.noreply.github.com> Date: Thu, 17 Sep 2026 14:43:59 -0700 Subject: [PATCH 3/3] table titles --- modules/developing/pages/astream-cdc.adoc | 1 - modules/operations/pages/astream-regions.adoc | 45 +++++++++---------- .../operations/pages/monitoring/index.adoc | 8 ++-- .../pages/private-connectivity.adoc | 4 +- 4 files changed, 28 insertions(+), 30 deletions(-) diff --git a/modules/developing/pages/astream-cdc.adoc b/modules/developing/pages/astream-cdc.adoc index 2ba7c61..e459a69 100644 --- a/modules/developing/pages/astream-cdc.adoc +++ b/modules/developing/pages/astream-cdc.adoc @@ -122,7 +122,6 @@ The conversion for some AVRO structures requires additional tooling that can res The following table describes the conversion of AVRO logical types: -.AVRO complex types [cols="1,1,1,1"] |=== |Name |AVRO type |Fields |Explanation diff --git a/modules/operations/pages/astream-regions.adoc b/modules/operations/pages/astream-regions.adoc index f377bbf..522bd08 100644 --- a/modules/operations/pages/astream-regions.adoc +++ b/modules/operations/pages/astream-regions.adoc @@ -9,82 +9,81 @@ For example, to enable xref:developing:astream-cdc.adoc[CDC for {astra-db}], you Your ElasticSearch and Snowflake deployments can be in different regions than your {product} tenants. -== AWS regions for {product} +If your preferred region isn't available, contact your {company} account representative or {support-url}[IBM Support]. -[cols="1,1"] +[cols=3] |=== -|Region |Location +|Cloud provider |Region |Location +|AWS |`ap-south-1` |Mumbai +|AWS |`ap-southeast-1` |Singapore +|AWS |`ap-southeast-2` |Sydney +|AWS |`eu-central-1` |Frankfurt +|AWS |`eu-west-1` |Ireland +|AWS |`us-east-1` |Virginia +|AWS |`us-east-2` |Ohio +|AWS |`us-west-2` |Oregon -|=== - -== Google Cloud regions for {product} - -[cols="1,1"] -|=== -|Region |Location +|Google Cloud |`australia-southeast1` |Sydney +|Google Cloud |`europe-west1` |Belgium +|Google Cloud |`europe-west3` |Frankfurt +|Google Cloud |`us-central1` |Iowa +|Google Cloud |`us-east1` |South Carolina +|Google Cloud |`us-east4` |Virginia -|=== - -== Microsoft Azure regions for {product} - -[cols="1,1"] -|=== -|Region |Location - +|Azure |`australiaeast` |New South Wales +|Azure |`eastus` |Virginia +|Azure |`westeurope` |Netherlands +|Azure |`westus2` |Washington -|=== - -== Request a region - -If your preferred region isn't available, contact your {company} account representative or {support-url}[IBM Support]. \ No newline at end of file +|=== \ No newline at end of file diff --git a/modules/operations/pages/monitoring/index.adoc b/modules/operations/pages/monitoring/index.adoc index 3468e55..4abb394 100644 --- a/modules/operations/pages/monitoring/index.adoc +++ b/modules/operations/pages/monitoring/index.adoc @@ -11,14 +11,14 @@ Additionally, of the exposed metrics, not all metrics are recommended for extern For a complete {pulsar-short} metrics reference, see: -* https://pulsar.apache.org/docs/reference-metrics/#namespace-metrics[Namespace metrics] +* https://pulsar.apache.org/docs/reference-metrics/#namespace-metrics[{pulsar-short} Namespace metrics] -* https://pulsar.apache.org/docs/reference-metrics/#topic-metrics[Topic metrics] - -For a complete {product} metrics reference, see xref:operations:monitoring/metrics.adoc[]. +* https://pulsar.apache.org/docs/reference-metrics/#topic-metrics[{pulsar-short} Topic metrics] == {product} metrics +For a complete {product} metrics reference, see xref:operations:monitoring/metrics.adoc[]. + === Namespace and topic metrics {product} exposes both namespace and topic level metrics. diff --git a/modules/operations/pages/private-connectivity.adoc b/modules/operations/pages/private-connectivity.adoc index 20a57c7..1736f67 100644 --- a/modules/operations/pages/private-connectivity.adoc +++ b/modules/operations/pages/private-connectivity.adoc @@ -32,7 +32,7 @@ To use a private link service or private endpoint for {product}, do the followin . Contact {support-url}[IBM Support] to request private connectivity for {product}. -=== Private connections for inbound traffic +== Private connections for inbound traffic {product} supports private inbound traffic flowing from your private endpoint to {product}. Inbound traffic includes {pulsar-reg}, {kafka-reg}, and RabbitMQ messaging traffic, as well as Prometheus metrics traffic. @@ -63,7 +63,7 @@ The private link service pattern is the same across cloud providers, but the hos |`prometheus-**PROVIDER**-**REGION**.private.streaming.datastax.com` |=== -=== Private connections for outbound traffic +== Private connections for outbound traffic On a case-by-case basis, {product} can support private outbound traffic flowing from an {product} private endpoint to your private link service.