How Google Lens uses schema markup

20 billion visual searches every month. Two big updates. Today Google announced that they are bringing Lens to iOS, which is a significant milestone, lets take a minute to ponder what does this mean for structured data?

What exactly is coming for iOS users?

First, if you have an iPhone, you’ll find a new Lens option that lets you select and search what’s on your screen within Chrome or the Google app, using whatever gesture comes naturally — like drawing, highlighting or tapping. Whether you’re reading an article, shopping for a product or watching a video, you can use this feature to quickly perform a visual search while browsing, without having to take a screenshot or open a new tab.

Google Lens behind the scenes

Note that real Google algorithms are behind a very thick curtains, and Google doesn't like going around talking about them.

Something like Google Lens most likely operates in two big parts: - Understanding the image - Matching the image to Google Knowledge Graph

When you take a screenshot or a photo, the data from this photo has to be understood. Humans species has perfected this skill over years of evolution of visual cortex. Cat or Coca-Cola? We know exactly what we are looking at. Computers? All they see is a grid of colorful pixels. Red, green, blue. Luckily this problem has really been solved many years ago with invention of Convolutional Neural Networks. These are type of networks that learned how to tell apart a cat from dog. Or a Coca-Cola can from a Pepsi can. There are many flavors of neural networks these days. But most of them operate on a very simple underlying principle called embeddings.

What is an image embedding?

To really understand what are embeddings it's much better to understand which problems they solve. Imagine you have a piece of paper with a word car written on it. Cool, then imagine in the other hand you have a photo of a car, but no text whatsoever. Now you need to establish relationship between them. Do these two object essentially signify the same thing? Absolutely! Yet they are in a different representations: one is textual, another one is visual. So how can we say that these two things are actually the same thing? Embeddings!

For now imagine an smart program that takes in your car text, and turns it into something like [0, 1, 0, 1, 0, 0]. Cool, we will skip details how it is done exactly, but this is the outcome of what we call embedding something or getting embeddings.

Now imagine the same smart program that loads in your photo of a car, and returns [0, 1, 0, 1, 0, 0].

Wow, what do we see? These are the same numbers! Now that we compared embeddings, we can firmly say, yes, these are the same thing indeed!

Embeddings and schema markup

So now that we have introduced embeddings into the mix, you might ask, "Okay, what do they have to do with schema?". In a previous example we had a overly simplistic representation, simple car is indeed not something that we work with on every day basis. Google search works with our websites that are made of HTML and other content. Remember finding Waldo game? Well it's something akin to this! Where Waldo is your useful content, and HTML syntax is everyone else around it.

So what schema is doing? That would be like putting a giant red box around Waldo. So when Google Crawl bot comes to your website, it does not need to look for Waldo, but merely find a red box and shoot him in the forehead. I might be getting carried away here, but essentially schema markup helps Google and other search engines to ingest your website. That then turns into aforementioned embeddings. And then stored somewhere deep in Google's heart.

Let's say you sell a product which is Play-Doh Playset.

Your schema markup might look something like this.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Play-Doh Pizza Delivery Scooter Playset",
  "image": "playdoh-pizza-delivery-scooter.jpg",
  "description": "Create your own Play-Doh pizzas and deliver them on a scooter with this fun playset!",
  "brand": {
    "@type": "Brand",
    "name": "Play-Doh"
  },
  "sku": "B07XYZ123",
  "offers": {
    "@type": "Offer",
    "price": "7.79",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  },
  "aggregateRating": {
    "@type": "AggregateRating",
    "ratingValue": "4.8",
    "reviewCount": "512"
  },
  "review": [
    {
      "@type": "Review",
      "author": "Sophia",
      "datePublished": "2023-01-05",
      "reviewBody": "My kids love making their own pretend pizzas and delivering them on the scooter!",
      "name": "Great for imaginative play",
      "reviewRating": {
        "@type": "Rating",
        "bestRating": "5",
        "ratingValue": "5",
        "worstRating": "1"
      }
    }
  ]
}
</script>

So after your schema is ingested by search engine, it's then turned into embeddings. Without schema you would rely on Google trying to understand an HTML file that usually looks something like this.

<html>
<head>
  <title>Play-Doh Pizza Delivery Scooter Playset</title>
</head>
<body style="background-color: #f9f9f9;">
  <center><h1 style="color: purple;">Play-Doh Pizza Delivery Scooter Playset <font color="blue">(Ages 3+)</font></h1></center>

  <div>
    <p>
      <b style="color: red;">Create</b> your own 
      <i>Play-Doh pizzas</i> and deliver them on a scooter 
      <span style="text-decoration: underline;">with this fun playset!</span>
    </p>
    <br />
    <font size="+2" color="green">Only $7.79!</font>
    <marquee behavior="alternate" bgcolor="#ffffcc" direction="left">Limited time offer! Get yours now!</marquee>
    <div style="border: 5px dotted orange; margin: 10px; padding: 10px;">
      <img src="playdoh-pizza-delivery-scooter.jpg" alt="Pizza Delivery Scooter Playset" width="200" style="float: left; margin-right: 20px;" />
      <ul style="list-style-type: circle;">
        <li><b>Brand:</b> Play-Doh</li>
        <li><b>SKU:</b> B07XYZ123</li>
        <li><b>Rating:</b> 4.8/5</li>
        <li><b>Reviews:</b> 512</li>
      </ul>
      <div style="clear: both;"></div>
    </div>
    <hr />

    <table border="2" cellpadding="5" cellspacing="0" width="80%" align="center" style="background-color: #eee;">
      <tr>
        <td>
          <strong>Review 1:</strong><br />
          <em>Author:</em> Sophia <br />
          <em>Rating:</em> 5/5 <br />
          "My kids love making their own pretend pizzas and delivering them on the scooter!"
        </td>
      </tr>
      <tr>
        <td>
          <strong>Review 2:</strong><br />
          <em>Author:</em> Jacob <br />
          <em>Rating:</em> 4/5 <br />
          "It keeps the children busy, but I wish it came with more colors of Play-Doh."
        </td>
      </tr>
    </table>

    <div style="margin-top: 20px;">
      <a href="https://example.com" style="font-size: 20px; text-decoration: none; color: #fff; background-color: #333; padding: 10px;">Buy Now!</a>
    </div>
  </div>

  <br /><br />

See how much redundant text is there? Realistically complex websites have a lot of HTML syntax and very little content. This is where schema markup comes to the rescue.

Future of schema markup

As the web grows, the sheer volume of information grows as well. Search engines are looking into ways of simplifying their pipelines and cutting their costs. As of today AI is still a "very expensive" tool, compared to Google Search, that has been free since its inception. Schema-like documents drive down those costs, by filtering an unwanted information and streamlining ingestion.

For more posts like this subscribe to our newsletter and follow us on LinkedIn!