Multimodal AI: Images as Inputs
1. Multimodal AI: Images as Inputs
Every lesson in this course so far has dealt with text, like transcripts, descriptions, or customer notes. That's not because text is the only thing that matters, but rather because text is where AI capabilities are the most mature at this time. However, like all things AI, that changes quickly. Real-world operational data includes product photos, scanned receipts, shipping documents, audios from customer calls, photos, hand-filled return labels, and so many more non-text files. They're often sitting in your cloud storage or Snowflake stages already, and until recently, you needed a separate machine learning pipeline to do anything useful with them. But not anymore. Now several AI functions that you already know support non-text inputs. In this lesson, we will cover which ones, how they work, and where they can be useful. Four functions in the AI family now handle non-text data. AI_CLASSIFY and AI_EMBED now support images. For classify, you can pass a product photo and a list of categories, and it returns the most likely classification. And for embed, the function generates a vector embedding for the image, the same way it does for text, which opens up image similarity search. AI_TRANSCRIBE converts audio and video files into text. We mentioned this briefly in a previous lesson, but we'll say a bit more about it here. AI_PARSE_DOCUMENT extracts structured content from document images and scanned PDFs. Invoices, forms, shipping labels, anything where you've got a file that contains information you need as structured data. Let's look at each of these more concretely, starting with AI_CLASSIFY. Before the examples, there is one piece of mechanics that you need to understand. Images don't go into a SQL column as raw bytes. They live in a Snowflake stage, which is the same internal or external cloud storage you use for file-based workflows. You reference the file using stage path, and you wrap that path with the TO_FILE function to create an object reference that the AI functions understand. That's the pattern. File into stage, TO_FILE to create the reference, pass reference to the function. Let's walk through a scenario. Our ski gear retailer receives return shipments. Each return comes with a photo the customer submitted showing the damage or defect. These photos are uploaded to a stage automatically when the return is initiated. Until now, someone had to manually look at each photo and tag it. With AI_CLASSIFY, you could do that in bulk. Quite the time saver, don't you think? But wait, it gets even more useful. You already have customer call transcripts in a separate table. Now you can join the image-based damage classification to the text-based call record and build a richer picture of each return. Rather than disparate pieces, now you have a single record that includes the photo showed, what the customer said on the call, and structured fields extracted from that call. That's much more of a complete picture of a return than either image or the transcript alone gives you. This is more of a composability pattern we've already discussed, but now extended to non-text data. The functions still chain together in SQL like this with images acting as just another input type alongside text. We covered AI_TRANSCRIBE in a previous lesson, but let's recap here. The quick version is that you point the function at an audio or video file on stage and transcribe returns a text transcript. From there, every other AI function applies. You transcribe your call recordings in bulk and then the entire text analysis pipeline you built in earlier lessons runs on the output without any changes. The last function we'll cover in this lesson is a parse document function. AI_PARSE_DOCUMENT handles documents like scanned PDFs, images of forms, or photos of invoices. As you can see in our code, you can pass it a file reference and a description of what you want to extract. The parsed output preserves the document structure. Tables in the document come back as structured data. Forms come back with field names and values. You can then extract specific fields using AI_EXTRACT or query the parsed JSON directly. Multimodal AI is an ever-expanding area. The four functions in this lesson represent where multimodal support stands now in the AI function family, but Snowflake is actively adding support for new modalities and expanding which models and file types each function accepts. Before you build on any of these in production, check the current Snowflake documentation for the list of supported file formats, size limits, and model availability in region. The direction is clearly towards more, not less, but specifics change faster than any course can keep up with. So, now let's recap. Non-text outputs are now supported in several AI functions. AI_CLASSIFY for image classification, AI_EMBED for image embeddings, AI_TRANSCRIBE for audio and video, and AI_PARSE_DOCUMENT for document images and scanned PDFs. Images and files live in a Snowflake stage. You reference them with TO_FILE. The function call syntax is otherwise the same as you've been doing for text. And these functions compose with the rest of the AI family the same way the text-based functions do. The results are the columns that can be joined, filtered, aggregated, and chained from there. This is the same data, just more of it.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.