Is code a liability or an asset? What it does should be an asset, of course. But there’s a cost to running and maintaining code. Ideally, we take advantage of (managed) services that minimize how much code we have to write to accomplish something.
What if I want to accept document from a partner or legacy business system, send out a request for internal review of that document, and then continue processing? In ye olden days, I’d build file watchers, maybe a database to hold state of in-progress reviews, a poller that notified reviewers, and a web service endpoint to handle responses and update state in the database. That’s potentially a lot of code. Can we get rid of most that?
Google Cloud Workflows recently added a “callback” functionality which makes it easier to create long-running processes with humans in the middle. Let’s build out an event-driven example with minimal code, featuring Cloud Storage, Eventarc, Cloud Workflows, and Cloud Run.
Step 1 – Configure Cloud Storage
Our system depends on new documents getting added to a storage location. That should initiate the processing. Google Cloud Storage is a good choice for an object store.
I created a new bucket named “loan-application-submissions’ in our us-east4 region. At the moment, the bucket is empty.
Step 2 – Create Cloud Run app
The only code in our system is the application that’s used to review the document and acknowledge it. The app accepts a querystring parameter that includes the “callback URL” that points to the specific Workflow instance waiting for the response.
I built a basic Go app with a simple HTML page, and a couple of server-side handlers. Let’s go through the heart of it. Note that the full code sample is on GitHub.
func main() {
fmt.Println("Started up ...")
e := echo.New()
e.Use(middleware.Logger())
e.Use(middleware.Recover())
t := &Template{
Templates: template.Must(template.ParseGlob("web/home.html")),
}
e.Renderer = t
e.GET("/", func(c echo.Context) error {
//load up object with querystring parameters
wf := workflowdata{LoanId: c.QueryParam("loanid"), CallbackUrl: c.QueryParam("callbackurl")}
//passing in the template name (not file name)
return c.Render(http.StatusOK, "home", wf)
})
//respond to POST requests and send message to callback URL
e.POST("/ack", func(c echo.Context) error {
loanid := c.FormValue("loanid")
fmt.Println(loanid)
callbackurl := c.FormValue("callbackurl")
fmt.Println("Sending workflow callback to " + callbackurl)
wf := workflowdata{LoanId: loanid, CallbackUrl: callbackurl}
// Fetch an OAuth2 access token from the metadata server
oauthToken, errAuth := metadata.Get("instance/service-accounts/default/token")
if errAuth != nil {
fmt.Println(errAuth)
}
//load up oauth token
data := OAuth2TokenInfo{}
errJson := json.Unmarshal([]byte(oauthToken), &data)
if errJson != nil {
fmt.Println(errJson.Error())
}
fmt.Printf("OAuth2 token: %s", data.Token)
//setup callback request
workflowReq, errWorkflowReq := http.NewRequest("POST", callbackurl, strings.NewReader("{}"))
if errWorkflowReq != nil {
fmt.Println(errWorkflowReq.Error())
}
//add oauth header
workflowReq.Header.Add("authorization", "Bearer "+data.Token)
workflowReq.Header.Add("accept", "application/json")
workflowReq.Header.Add("content-type", "application/json")
//inboke callback url
client := &http.Client{}
workflowResp, workflowErr := client.Do(workflowReq)
if workflowErr != nil {
fmt.Printf("Error making callback request: %s\n", workflowErr)
}
fmt.Printf("Status code: %d", workflowResp.StatusCode)
return c.Render(http.StatusOK, "home", wf)
})
//simple startup
e.Logger.Fatal(e.Start(":8080"))
}
The “get” request shows the details that came in via the querystrings. The “post” request generates the required OAuth2 token, adds it to the header, and calls back into Google Cloud Workflows. I got stuck for a while because I was sending an ID token and the service expects an access token. There’s a difference! My colleague Guillaume Laforge, who doesn’t even write Go, put together the code I needed to generate the necessary OAuth2 token.
From a local terminal, I ran a single command to push this source code into our fully managed Cloud Run environment:
gcloud run deploy
After a few moments, the app deployed and I loaded it up the browser with some dummy querystring values.
Step 3 – Create Workflow with event-driven trigger
That was it for coding! The rest of our system is composed of managed services. Specifically, Cloud Workflows, and Eventarc which processes events in Google Cloud and triggers consumers.
I created a new Workflow called “workflow-loans” and chose the new “Eventarc” trigger. This means that the Workflow starts up as a result of an event happening elsewhere in Google Cloud.
A new panel popped up and asked me to name my trigger and pick a source. We offer nearly every Google Cloud service as a source for events. See here that I chose Cloud Storage. Once I chose the event provider, I’m offered a contextual set of events. I selected the “finalized” event which fires for any new object added to the bucket.
Then, I’m asked to choose my storage bucket, and we have a nice picker interface. No need to manually type it in. Once I chose my bucket, which resides in a different region from my Workflow, I’m told as much.
The final step is to add the Workflow definition itself. These can be in YAML or JSON. My Workflow accepts some arguments (properties of the Cloud Storage doc, including the file name), and runs through a series of steps. It extracts the loan number from file name, creates a callback endpoint, logs the URL, waits for a callback, and processes the response.
The full Workflow definition is below, and also in my GitHub repo.
main:
params: [args]
steps:
- setup_variables:
#define and assign variables for use in the workflow
assign:
- version: 100 #can be numbers
- filename: ${args.data.name} #name of doc
- log_receipt:
#write a log to share that we started up
call: sys.log
args:
text: ${"Loan doc received"}
- extract_loan_number:
#pull out substring containing loan number
assign:
- loan_number : ${text.substring(filename, 5, 8)}
- create_callback:
#establish a callback endpoint
call: events.create_callback_endpoint
args:
http_callback_method: "POST"
result: callback_details
- print_callback_details:
#print out formatted URL
call: sys.log
args:
severity: "INFO"
# update with the URL of your Cloud Run service
text: ${"Callback URL is https://[INSERT CLOUD RUN URL HERE]?loanid="+ loan_number +"&callbackurl=" + callback_details.url}
- await_callback:
#wait impatiently
call: events.await_callback
args:
callback: ${callback_details}
timeout: 3600
result: callback_request
- print_callback_request:
#wlog the result
call: sys.log
args:
severity: "INFO"
text: ${"Received " + json.encode_to_string(callback_request.http_request)}
- return_callback_result:
return: ${callback_request.http_request}
I deployed the Workflow which also generated the Eventarc trigger itself.
Step 4 – Testing it all out
Let’s see if this serverless, event-driven system now works! To start, I dropped a new PDF named “loan600.pdf” into the designated Storage bucket.
Immediately, Eventarc triggered a Workflow instance because that PDF was uploaded to Cloud Storage. See that the Workflow instance in an “await_callback” stage.
On the same page, notice the logs for the Workflow instance, including the URL for my Cloud Run with all the right querystring parameters loaded.
I plugged that URL into my browser and got my app loaded with the right callback URL.
After clicking the “acknowledge loan submission” button which called back to my running Workflow instance, I switched back to Cloud Workflows and saw that my instance completed successfully.
Summary
There are many ways to solve the problem I called out here. I like this solution. By using Google Cloud Eventarc and Workflows, I eliminated a LOT of code. And since all these services, including Cloud Run, are fully managed serverless services, it only costs me money when it does something. When idle, it costs zero. If you follow along and try it for yourself, let me know how it goes!
Richard Seroter's Architecture Musings 