No cloud account, no problem. Try out change streams in Cloud Spanner locally with a dozen-ish shell commands.

If you have a choice, you should test software against the real thing. The second best option is to use a “fake” that implements the target service’s API. In the cloud, it’s straightforward to spin up a real instance of a service for testing. But there are reasons (e.g. cost or speed) or times (e.g. within a CI pipeline, or rapid testing on your local machine) when an emulator is a better bet.

Let’s say that you wanted to try out Google Cloud Spanner, and it’s useful change streams functionality. Consider creating a real instance and experimenting, but you have an alternative option. The local emulator just added support for change streams, and you can test the whole thing out from the comfort of your own machine. Or, to make life even easier, test it out from a free cloud machine.

With just a Google account (which most everyone has?), you can use a free cloud-based shell and code editor. Just go to shell.cloud.google.com. We’ve loaded this environment up with language CLIs for Java, .NET, Go, and others. It’s got the Docker daemon running. And it’s got our gcloud CLI pre-loaded and ready to go. It’s pretty cool. From here, we can install the Spanner emulator, and run just a few shell commands to see the entire thing in action.

Let’s begin by installing the emulator for Cloud Spanner. It takes just one command.

sudo apt-get install google-cloud-sdk-spanner-emulator

Then we start up the emulator itself with this command:

gcloud emulators spanner start 

After a couple of seconds, I see the emulator running, and listening on two ports.

Great. I want to leave that running while having the freedom to run more commands. It’s easy to spin up new tabs in the Cloud Shell Editor, so I created a new one.

In this new tab, I ran a set of commands that configured the gcloud CLI to work locally with the emulator. The CLI supports the concept of multiple configurations, so we create one that is emulator friendly. Also note that Google Cloud has the idea of “projects.” But if you don’t have a Google Cloud account, you’re ok here. For the emulators, you can use a non-existent value for “project” as I have here.

gcloud config configurations create emulator
gcloud config set auth/disable_credentials true
gcloud config set project local-project
gcloud config set api_endpoint_overrides/spanner http://localhost:9020/

It’s time to create a (local) Spanner instance. I ran this one command to do so. It’s super fast, which makes it great for CI pipeline scenarios. That second command sets the default instance name so that we don’t have to provide an instance value in subsequent commands.

gcloud spanner instances create test-instance \
   --config=emulator-config --description="Test Instance" --nodes=1
gcloud config set spanner/instance test-instance

Now, we need a database in this instance. Spanner supports multiple “dialects”, including PostgreSQL. Here’s how I create a new database.

gcloud spanner databases create example-db --database-dialect=POSTGRESQL

Let’s throw a couple of tables into this database. We’ve got one for Singers, and one for Albums.

gcloud spanner databases ddl update example-db \
--ddl='CREATE TABLE Singers ( SingerId bigint NOT NULL, FirstName varchar(1024), LastName varchar(1024), SingerInfo bytea, PRIMARY KEY (SingerId) )'
gcloud spanner databases ddl update example-db \
--ddl='CREATE TABLE Albums ( SingerId bigint NOT NULL, AlbumId bigint NOT NULL, AlbumTitle varchar, PRIMARY KEY (SingerId, AlbumId) ) INTERLEAVE IN PARENT Singers ON DELETE CASCADE'

Now we’ll insert a handful of rows into each table.

gcloud spanner databases execute-sql example-db \
  --sql="INSERT INTO Singers (SingerId, FirstName, LastName) VALUES (1, 'Marc', 'Richards')"
gcloud spanner databases execute-sql example-db \
  --sql="INSERT INTO Singers (SingerId, FirstName, LastName) VALUES (2, 'Catalina', 'Smith')"
gcloud spanner databases execute-sql example-db   \
  --sql="INSERT INTO Singers (SingerId, FirstName, LastName) VALUES (3, 'Alice', 'Trentor')"
gcloud spanner databases execute-sql example-db   \
  --sql="INSERT INTO Albums (SingerId, AlbumId, AlbumTitle) VALUES (1, 1, 'Total Junk')"
gcloud spanner databases execute-sql example-db   \
  --sql="INSERT INTO Albums (SingerId, AlbumId, AlbumTitle) VALUES (2, 1, 'Green')"

If you want to prove this works (thus far), you can execute regular queries against the new tables. Here’s an example of retrieving the albums.

gcloud spanner databases execute-sql example-db \
    --sql='SELECT SingerId, AlbumId, AlbumTitle FROM Albums'

It’s time to turn on change streams, and this takes an extra step. It doesn’t look like I can smuggle utility commands through the “execute-sql” operation, so we need to run a DDL statement instead. Note that you can create change streams that listen to specific tables or columns. This one listens to anything changing in any table.

gcloud spanner databases ddl update example-db \
--ddl='CREATE CHANGE STREAM EverythingStream FOR ALL'

If you want to prove everything is in place, you can run this command to see all the database objects.

gcloud spanner databases ddl describe example-db --instance=test-instance

I’m now going to open a third tab in the Cloud Shell Editor. This is so that we can continuously tail the change stream results. We’ve created this nice little sample project that lets you tail the change stream. Install the app by running this command in the third tab.

go install github.com/cloudspannerecosystem/spanner-change-streams-tail@latest

Then, in this same tab, we want the Go SDK (which this app uses) to look at the local emulator’s gRPC port instead of the public cloud. Set the environment variable that overrides the default behavior.

export SPANNER_EMULATOR_HOST=localhost:9010

Awesome. Now we start up the change stream app with a single command. You should see it start up and hold waiting for data.

spanner-change-streams-tail -p local-project -i test-instance -d example-db -s everythingstream

Back in the second tab (the first should still be running the emulator, the third is running the change stream tail), let’s add a new record to the Spanner database table. What SHOULD happen is that we see a change record pop up in the third tab.

gcloud spanner databases execute-sql example-db   \
  --sql="INSERT INTO Albums (SingerId, AlbumId, AlbumTitle) VALUES (2, 2, 'Go, Go, Go')"

Sure enough, I see a record pop into the third tab showing the before and after values of the row.

You can mess around with updating records, deleting records, and so on. A change stream is powerful for event sourcing scenarios, or simply feeding data changes to downstream systems.

In this short walkthrough, we tried out the Cloud Shell Editor, spun up the Spanner emulator, and experimented with database change streams. All without needing a Google Cloud account, or installing a lick of software on our own device. Not bad!