It was a good day, and I’m doing a quick trip to Sunnyvale tonight to talk to a customer tomorrow about how Google uses AI to improve our own software delivery. Should be fun.
[blog] Customers are putting Gemini to work. Are companies really doing meaningful, business-critical things with generative AI? Some are, yes. This post names them. Related.
[blog] Evaluate open LLMs with Vertex AI and Gemini. Which prompts or models work best for what you’re trying to do? Philipp tries out our managed evaluation service to see what prompts work best on Llama 3.1.
[blog] What drives users to Infrastructure as Code? Brian says that GUIs don’t scale, and chaining together CLI commands is brittle. IaC offers a better way to establish repeatable and maintainable infrastructure setups.
[blog] How streaming LLM APIs work. You probably want big LLM results coming back as a stream versus in a single chunk, and Simon shows how 3 top models handle that.
[blog] Charting Your AI Native Journey. Great read, and thought-provoking in its approach. Guy shows one way to categorize AI tools and gives examples of what falls into each bucket.
I’m flying to Washington DC today with my oldest son to look at colleges. It was a vacation day from work (mostly), but I still did some reading worth sharing.
[blog] Pulumi Google Cloud Provider Version 8.0.0. If you’re using Pulumi already, or just looking to get rolling with a good infrastructure automation strategy, take a look at what’s new.
[article] Pfizer’s Future of Development. How does a big company truly change how they deliver software? I like the focus areas here, and the outcomes are excellent.
Want to get this update sent to you every day? Subscribe to my RSS feed or subscribe via email below:
It’s been a long week! I spent some time this afternoon setting up a new work laptop, and I’m going to foolishly bring it on a trip tomorrow. What could go wrong?
[blog] Keys to a resilient Open Source future. Is AI going to be the best option for open source security? It might be, given the scale of code we’re talking and the volunteer-heavy approach.
[blog] Introducing Netflix’s Key-Value Data Abstraction Layer. Abstractions are tricky to maintain, and can accidentally block you from using unique features underneath. But for scenarios like this, the use case makes sense to me.
[blog] Quitting Time. Perseverance is important, but so is knowing when to quit. What’s your criteria, and can you stick to it?
[blog] Apache Airflow ETL in Google Cloud. The spectrum of hosting options is typically raw compute, managed compute, and managed servcies. That applies here as well.
Want to get this update sent to you every day? Subscribe to my RSS feed or subscribe via email below:
Today was another day of team meetings so I was offline for much of it. But I got some good early-morning reading finished, and you’ll find them in my 400th list below.
[article] Study Finds No DevOps Productivity Gains from Generative AI. Hmm. I haven’t seen findings like this, which is why the article caught my eye. Run your own analysis when you introduce these tools into your environment to see where it makes a positive difference.
I had team meetings today, which made me grateful to work with excellent folks. Before starting this offsite, I read through some fascinating pieces that you’ll find below.
[blog] Advancing Our Chef Infrastructure. I can’t imagine that companies swap out infrastructure-as-code or configuration management investments often. Slack is all-in on Chef, and explains their evolution.
[blog] Should we decompose our monolith? This post looks at the messy world of today’s environments where there’s a mix of monolithic and microservice approaches.
[blog] Legacy Modernization meets GenAI. The ThoughtWorks crew has an interesting tool they’ve built to make modernizations better and more approachable.
[article] How (and why) should teams try distributed pair programming? While some companies are pushing return-to-office, many still have distributed teams that need to work together. This post looks at a study showing what makes distributed pair programming successful.
As hoped for, I mostly skipped doing work this weekend and spent longer-than-expected building an Apollo 11 lunar lander with one of my kids. We had fun. Today was back to the work that pays for these Lego sets.
[blog] Building LLM-powered applications in Go. Eli reminds us that the vast majority of “AI apps” are really just like apps that call LLMs. He shows off a handful of ways to create a RAG server in Go.
[blog] Open Source Foundations Considered Helpful. Do open source foundations just exist to throw lavish conferences? Nah. They add legit value, as James points out here.
[blog] RAG API. Here’s a good post about a managed solution for augmenting your LLM requests with customized data sources. I hadn’t tried this service out yet, and now i want to.
[youtube-video] Safe RAG for LLMs. *Three* RAG pieces today? Let’s embrace the discomfort. I liked this video which looked at patterns for RAG that avoid data leakage.
[article] Predicting developer attrition. How do you know a developer is on their way out the door? This paper looks at attrition and notes how burnout prevention and “opportunities to learn” are big factors.
Today was a good day. I got enough done that I think I can skip most work this weekend, and build that Lego set sitting on my shelf that my kiddo been asking me “when can we build this?” This weekend, son.
[blog] Intro to Ray for AI on Kubernetes. Kaslin explains this open source framework, why you’d use it, and how to get it running on Kubernetes.
[blog] Message Queues in System Design. Classic technology, but there’s always someone learning about it for the first time. Posts like this are a solid intro to how and when to use queues.
I’m starting to wonder if I’m writing too much. Between this daily list, blog posts (new one today), and a couple newsletters each week, maybe I need to use other mediums too. I dunno. Hopefully you’re enjoying it!
[blog] How big up-front design fails. Here are some lessons learned from projects with big upfront planning and design, and why a more iterative approach works better.
[podcast] 274: The Cloud Pod is Still Not Open Source. This latest episode of the podcast that reviews news from AWS, Microsoft and Google spent some time discussing my recent post about technical debt. Neat!
Want to get this update sent to you every day? Subscribe to my RSS feed or subscribe via email below:
School is back in session, and I just met with a handful of teachers at a recent back-to-school night. They’re all figuring out how to account for generative AI tools that students have access to. I say, let’s give teachers the same tools to use. Specifically, what if a teacher wants a quick preliminary grade on book reports submitted by their students? To solve this, I used Gemini Flash 1.5 in Google Cloud Vertex AI in three different ways—one-off in the prompt editor, through code, and via declarative workflow.
Grade Homework in Vertex AI Studio
Maybe you just have one or two papers to grade. Something like Vertex AI Studio is a good choice. Even if you’re not a Google Cloud customer, you can use it for free through this link.
You don’t have any book reports handy to test this with? Me neither. In Vertex AI Studio, I prompted with something like “Write a 300 word book report for Pride and Prejudice from the perspective of an 8th grade student with a good vocabulary and strong writing skills.”
With that handy, let’s “grade” the paper. Here I’m using system instructions to direct the model. I used Gemini to generate the initial version and then I tweaked it a bit. Here’s the instructions:
Task: Evaluate 8th-grade book reports for an honors English class. You are a tough grader. Input: Book report text. Output: Initial letter grade (A, B, C, D, or F) based on: Structure: Clear introduction, body, and conclusion Grammar: Spelling, punctuation, sentence structure. Content: Understanding of the book, critical thinking. Consider: Age and language proficiency of the student.
This tells the model what to look for, the expected quality (“honors English”), and the desired output.
I added the system instructions to the marked section within Vertex AI Studio.
Then I simply pasted the book report into the “prompt” section and submitted the prompt.
I thought this was a pretty good book report, but Gemini 1.5 Flash had a high bar after obeying my system instructions.
This book report shows good understanding of the book and a strong grasp of the themes. However, there are some issues:
Structure: The introduction is a little long and could be more concise. The conclusion is also repetitive. The report could benefit from more clear paragraph breaks to organize the information more effectively.
Grammar: No significant grammar issues.
Content: The student demonstrates good understanding of the novel's major themes and characters. There's a strong focus on plot summary, which is okay, but the report could benefit from more critical analysis. The student needs to go beyond just describing the events to analyze how they impact the themes, characters, or messages of the book. Overall: This report shows good effort, but it needs to be refined.
Grade: B-
If I’m a teacher, this is a fascinating way to get a preliminary assessment of a writing assignment!
Grade Homework Through Code
The above solution works fine for one-off experiences, but how might you scale this AI-assisted grader? Another option is code.
To try this scenario out, I used Cloud Firestore as my document database holding the book reports. I created a collection named “Papers” in the default database and added three documents. Each one holds a different book report.
I think used the Firestore API and Vertex AI API to write some simple Go code that iterates through each Firestore document, calls Vertex AI using the provided system instructions, and then logs out the grade for each report. Note that I could have used a meta framework like LangChain, LlamaIndex, or Firebase Genkit, but I didn’t see the need.
package main
import (
"context"
"fmt"
"log"
"os"
"cloud.google.com/go/firestore"
"cloud.google.com/go/vertexai/genai"
"google.golang.org/api/iterator"
)
func main() {
// get configuration from environment variables
projectID := os.Getenv("PROJECT_ID")
collectionName := os.Getenv("COLLECTION_NAME") // "Papers"
location := os.Getenv("LOCATION") //"us-central1"
modelName := os.Getenv("MODEL_NAME") // "gemini-1.5-flash-001"
ctx := context.Background()
//initialize Vertex AI client
vclient, err := genai.NewClient(ctx, projectID, location)
if err != nil {
log.Fatalf("error creating vertex client: %v\n", err)
}
gemini := vclient.GenerativeModel(modelName)
//add system instructions
gemini.SystemInstruction = &genai.Content{
Parts: []genai.Part{genai.Text(`Task: Evaluate 8th-grade book reports for an honors English class. You are a tough grader. Input: Book report text. Output: Initial letter grade (A, B, C, D, or F) based on: Structure: Clear introduction, body, and conclusion Grammar: Spelling, punctuation, sentence structure. Content: Understanding of the book, critical thinking. Consider: Age and language proficiency of the student.
`)},
}
// Initialize Firestore client
client, err := firestore.NewClient(ctx, projectID)
if err != nil {
log.Fatalf("Failed to create client: %v", err)
}
defer client.Close()
// Get documents from the collection
iter := client.Collection(collectionName).Documents(ctx)
for {
doc, err := iter.Next()
if err != nil {
if err == iterator.Done {
break
}
log.Fatalf("error iterating through documents: %v\n", err)
}
//create the prompt
prompt := genai.Text(doc.Data()["Contents"].(string))
//call the model and get back the result
resp, err := gemini.GenerateContent(ctx, prompt)
if err != nil {
log.Fatalf("error generating context: %v\n", err)
}
//print out the top candidate part in the response
log.Println(resp.Candidates[0].Content.Parts[0])
}
fmt.Println("Successfully iterated through documents!")
}
The code isn’t great, but the results were. I’m also getting more verbose responses from the model, which is cool. This is a much more scalable way to quickly grade all the homework.
Grade Homework in Cloud Workflows
I like the code solution, but maybe I want to run this preliminary grading on a scheduled basis? Every Tuesday night? I could do that with my above code, but how about using a no-code workflow engine? Our Google Cloud Workflows product recently got a Vertex AI connector. Can I make it work with the same system instructions as the above two examples? Yes, yes I can.
I might be the first person to stitch all this together, but it works great. I first retrieved the documents from Firestore, looped through them, and called Vertex AI with the provided system instructions. Here’s the workflow’s YAML definition:
main:
params: [args]
steps:
- init:
assign:
- collection: ${args.collection_name}
- project_id: ${args.project_id}
- location: ${args.location}
- model: ${args.model_name}
- list_documents:
call: googleapis.firestore.v1.projects.databases.documents.list
args:
collectionId: ${collection}
parent: ${"projects/" + project_id + "/databases/(default)/documents"}
result: documents_list
- process_documents:
for:
value: document
in: ${documents_list.documents}
steps:
- ask_llm:
call: googleapis.aiplatform.v1.projects.locations.endpoints.generateContent
args:
model: ${"projects/" + project_id + "/locations/" + location + "/publishers/google/models/" + model}
region: ${location}
body:
contents:
role: "USER"
parts:
text: ${document.fields.Contents.stringValue}
systemInstruction:
role: "USER"
parts:
text: "Task: Evaluate 8th-grade book reports for an honors English class. You are a tough grader. Input: Book report text. Output: Initial letter grade (A, B, C, D, or F) based on: Structure: Clear introduction, body, and conclusion Grammar: Spelling, punctuation, sentence structure. Content: Understanding of the book, critical thinking. Consider: Age and language proficiency of the student."
generation_config:
temperature: 0.5
max_output_tokens: 2048
top_p: 0.8
top_k: 40
result: llm_response
- log_file_name:
call: sys.log
args:
text: ${llm_response}
No code! I executed the workflow, passing in all the runtime arguments.
In just a moment, I saw my workflow running, and “grades” being logged to the console. In real life, I’d probably update the Firestore document with this information. I’d also use Cloud Scheduler to run this on a regular basis.
While I made this post about rescuing educators from the toil of grading papers, you can apply these patterns to all sorts of scenarios. Use prompt editors like Vertex AI Studio for experimentation and finding the right prompt phrasing. Then jump into code to interact with models in a repeatable, programmatic way. And consider low-code tools when model interactions are scheduled, or part of long running processes.
