Get startedGet started for free

Data Model in Vertex AI Feature Store

1. Data Model in Vertex AI Feature Store

Now that you're familiar with the main capabilities of Vertex AI feature store, let's explore the data model and terminology that is used in Vertex AI feature store to describe its resources and components. Vertex AI feature store uses a time series data model to store a series of values for features. This model enables Vertex AI feature store to maintain feature values as they change over over time. Vertex AI feature store organizes resources hierarchically in the following order feature store, entity Type, feature. You must create these resources before you can ingest data into Vertex AI feature store. Let's look at each of these terms in detail. Feature Store, a feature store is a container for storing and managing features. Features are measurable attributes of entities such as customers, products, or orders. Organizations typically create one shared feature store for feature ingestion, serving and sharing across all teams. However, sometimes organizations might choose to create multiple feature stores within the same project to isolate environments. For example, organizations might have separate feature stores for experimentation, testing, and production. Feature stores can be a valuable asset for organizations that are looking to improve their machine learning capabilities. By centralizing the storage and management of features, feature stores can help to improve the efficiency and accuracy of machine learning models. Entity Type, an entity type is a group of features that are related to each other in some way. For example, a movie service might have entity types for movies and users. Each entity type would contain features that are relevant to that type of entity. For example, the movie entity type might contain features such as the movie's title, release date, and genre. The user entity type might contain features such as the user's name, email address, and age. Entity types are used to organize features in a feature store. This can help improve the efficiency of feature management and clarify the relationships between features. When you create an entity type, you specify the following information the name of the entity type, the description of the entity type, the features that belong to the entity type. After you create an entity type, you can use it to store and manage features. For example, you can create a feature called movie_rating and associate it with the entity type movie. This will allow you to store the rating of each movie in the feature store. Entity, an entity is a specific instance of an entity type. For example, movie_one and movie_two are entities of the entity type movie. In a feature store Entity IDs are used to identify entities in a feature store. This can be helpful for tracking entities and for querying entities. Entity IDs must be unique within a feature store. This means that no two entities can have the same ID. Entity IDs must be of the type string. This means that they must be a string of characters. Entity IDs can be used to query entities in a feature store. For example, you can query all entities that have a specific ID. Entity IDs can also be used to track entities. For example, you can track the changes that have been made to an entity over time. Feature, a feature is a measurable property or attribute of an entity type. For example, the movie entity type has features such as Average, Rating, and Title. Features are associated with entity types, which means that they can only be used to describe entities of that type. For example, the average_rating feature can only be used to describe movies. Features must be unique within a given entity type, but they do not need to be globally unique. This means that no two features in the same entity type can have the same name, but two different entity types can have features with the same name. For example, the movie entity type and the user entity type can both have a feature named Title. When reading feature values, you provide the feature and its entity type as part of the request. For example, to read the average_rating feature for the Movie entity type, you would specify the feature name as average_rating and the entity type name as Movie. When creating a feature, you specify its value type. The value type determines what types of values you can store for the feature. For example, the value type for the average_rating feature is double. This means that you can only store floating point numbers for the average_rating feature. For more information about the supported value types, see the value type in the API reference. Feature value in Vertex AI feature Store, feature values are captured at specific time points, which allows a single entity to have multiple values for a given feature. For instance, the movie_one entity can have different values for the average_rating feature, such as 4.4 at one time, and 4.8 at a later time. Each feature value in Vertex AI feature store is associated with a tuple identifier comprising the Entity ID, Feature ID and Timestamp. This identifier is used to retrieve feature values during serving time. Although time is continuous, Vertex AI feature stores discrete values. When you request a feature value at a particular time, T, Vertex AI feature store will return the last stored value at or above time T. Vertex AI feature store will return the latest stored value at or before time, T. For example, if the location information of a car is stored in Vertex AI feature store at times 100 and 110, the location at time 100 will be used for requests made between 100 inclusive and 110 exclusive. If you need a higher resolution, you can interpolate the location between values or increase the sampling rate of your data. Here is an example of how the Vertex AI feature store data model can be used to store and manage feature values. Let's say we have an entity type called Product. There will be multiple entities belonging to that entity type. In the Vertex AI feature store, the ID of each entity is always represented as a string. The product entity type can have many features associated with it. For example, a product could have features such as price, quantity, and color. Each feature has a value which is the value of the feature for a specific entity. For example, the value of the price feature for the product with ID 12345 might be $10. Feature values in most cases will not be static. For example, the number of purchases in the past week for a product might change over time. Hence, in the Vertex AI feature store, each feature value has a corresponding timestamp associated with it. The timestamp indicates when the feature value was recorded. This information can be used to track changes in feature values over time. Feature Ingestion, Feature ingestion is the process of importing feature values computed by your feature engineering jobs into a feature store. Before you can ingest data, the corresponding entity type and features must be defined in the feature store. Vertex AI feature store offers batch and streaming ingestion, letting you add feature values in bulk or in real time. For example, you might have computed source data that lives in locations such as Big Query or Cloud Storage. You can batch ingest data from those sources into a central feature store, so that those feature values can be served in a uniform format. As your source data changes, you can use streaming ingestion to quickly get those changes into your feature store. That way, you have the latest data available for online serving scenarios. Feature Serving, feature serving is the process of exporting stored feature values for training or inference. Vertex AI feature store offers two methods for serving features batch and online. Batch serving is for high throughput and serving large volumes of data for offline processing, like for model training or batch predictions. Online serving is for low latency data retrieval of small batches of data for real time processing, like for online predictions. Entity View, when you retrieve values from a feature store, the service returns an entity view that contains the feature values that you requested. You can think of an entity view as a projection of the features and values that Vertex AI feature store returns from an online or batch serving request. For online serving requests, you can get all or a subset of features for a particular entity type. For batch serving requests, you can get all or a subset of features for one or more entity types. For example, if features are distributed across multiple entity types, you can retrieve them together in a single request which joins those features together. You can then use the results to feed to a machine learning or batch prediction request. Export Data, vertex AI feature store lets you export data from your feature stores so that you can backup and archive feature values. You can choose to export the latest feature values, snapshot or a range of values, full export. For more information, see the Export feature Values Documentation. Before we move into the next section, let's look at requirements for the source data that can be ingested into Vertex AI feature store. Note that these requirements might change over time. For the latest data requirements, check the documentation. For batch ingestion, you can use tables in Big Query or files in Cloud Storage. For streaming ingestion, you can provide the feature values to ingest as part of the API request. Each item or row must have a column for Entity IDs, and the values must be of type string. Your source data value types must match the value types of the destination feature in the feature store. All columns must have a header that are of type string. If you provide a column for feature generation timestamps, use one of the following timestamp formats. For Big Query tables and Big Query views, timestamps must be in the Timestamp column. For Avro, timestamps must be of the type Long and Logical type timestamp micros. For CSV files, timestamps must be in the RFC 339 format. CS V files cannot include array data types. Use Avro or Big Query instead. For array types, you cannot include a null value in the array. However, you can include an empty array.

2. Let's practice!

Create Your Free Account

or

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.