Manage Components

1. Manage Components

Oftentimes, we need to access, delete, or otherwise manage our GX components. Let's learn how to do that.

2. Components

Recall that GX components are Python classes that represent our data and data validation entities. Here is a list of some of the components we've learned about so far. That's a lot of components -- nice job!

3. Component management in GX

By now, you've learned about four key components that do most of the heavy lifting in GX. These are Data Sources, which connect to data and contain Data Assets; Expectation Suites, which house Expectations; Validation Definitions, which validate Expectations against data; and Checkpoints, which group together and automate Validations in a production deployment. These four components behave very similarly. In fact, each of these has a corresponding attribute in the Data Context, through which the components are managed. We'll be focusing on these four attributes in this chapter.

4. Adding components

Recall that to add our Expectation Suites and Validation Definitions to the Data Context, we used the `.add()` method of `.suites` and `.validation_definitions`, respectively. We can also add our Checkpoint to the Data Context, following the same pattern. We use the `context.checkpoints.add()` method, passing in our Checkpoint component to the `checkpoint` argument.

5. Adding a Data Source

In the case of adding Components to a Data Context, Data Sources behave slightly differently. Because Data Sources are configured according to the format of the data, the method for adding them to the Context depends on that data format. Here is a list of Data Source types and associated methods. Don't worry, you don't need to remember all of these.

6. Adding a pandas Data Source

Because we're focusing on Data Sources from in-memory pandas DataFrames, we'll stick with the `.add_pandas()` method we've been using, which just takes the `name` parameter.

7. Retrieving components

We can retrieve our components using the `.get()` method, passing in the GX name of the component to the `name` parameter. For instance, if we want to retrieve our Data Source named "my_pandas_dataframe", we use the `context.data_sources.get()` method, assigning the string "my_pandas_dataframe" to the `name` argument. And that's it! We have our Data Source.

8. Retrieving components

This same pattern also applies to Expectation Suites, Validation Definitions, and Checkpoints.

9. Listing components

Sometimes we forget what Components we even have in our Data Context. In this case, it's helpful to list out all of our components. We do this using the `.all()` method. For instance, we can use the `context.data_sources.all()` method to list out all of our Data Sources. This outputs a dictionary-like object with the name of each Data Source and its associated metadata. In this case, we can see that our Data Context contains one Data Source called "my_pandas_datasource".

10. Listing components

Again, this method can also be used to list out our Expectation Suites, Validation Definitions, and Checkpoints.

11. Deleting components

Finally, we can delete components using the `.delete()` method, passing in the name of the component to the name parameter. We can delete our Data Source using the `context.data_sources.delete()` method. Now the Data Context contains no Data Sources.

12. Deleting components

Of course, this method is the same for Expectation Suites, Validation Definitions, and Checkpoints.

13. Cheat sheet

Here's a summary of the methods we covered in this video for component management. Feel free to refer back to this as you complete the exercises.

14. Let's practice!

Time to start managing some components!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.