Is the source or the destination bad?

In the previous lesson, you used the destination computer as your entity of interest. However, your cybersecurity analyst just told you that it is the infected machines that generate the bad traffic, and will therefore appear as a source, not a destination, in the flows dataset.

The data flows has been preloaded, as well as the list bad of infected IDs and the feature extractor featurizer() from the previous lesson. You also have numpy available as np, AdaBoostClassifier(), and cross_val_score().

This exercise is part of the course

Designing Machine Learning Workflows in Python

View Course

Exercise instructions

Create a data frame where each row is a feature vector for a source_computer. Group by source computer ID in the flows dataset and apply the feature extractor to each group.
Convert the iterator to a data frame by calling list() on it.
Create labels by checking whether each source_computer ID belongs in the list of bads you have been given.
Assess an AdaBoostClassifier() on this data using cross_val_score().

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Group by source computer, and apply the feature extractor
out = flows.____('source_computer').____(featurize)

# Convert the iterator to a dataframe by calling list on it
X = pd.DataFrame(____, index=____)

# Check which sources in X.index are bad to create labels
y = [x in bads for x in ____]

# Report the average accuracy of Adaboost over 3-fold CV
print(np.mean(____(____, X, y)))

Edit and Run Code

Designing Machine Learning Workflows in Python

AdvancedSkill Level

4.8+

9 reviews

In the previous chapters you established a solid foundation in supervised learning, complete with knowledge of deploying models in production but always assumed you a labeled dataset would be available for your analysis. In this chapter, you take on the challenge of modeling data without any, or with very few, labels. This takes you into a journey into anomaly detection, a kind of unsupervised modeling, as well as distance-based learning, where beliefs about what constitutes similarity between two examples can be used in place of labels to help you achieve levels of accuracy comparable to a supervised workflow. Upon completing this chapter, you will clearly stand out from the crowd of data scientists in confidently knowing what tools to use to modify your workflow in order to overcome common real-world challenges.

Exercise 1: Anomaly detection Exercise 2: A simple outlier Exercise 3: LoF contamination Exercise 4: Novelty detection Exercise 5: A simple novelty Exercise 6: Three novelty detectors Exercise 7: Contamination revisited Exercise 8: Distance-based learning Exercise 9: Find the neighbor Exercise 10: Not all metrics agree Exercise 11: Unstructured data Exercise 12: Restricted Levenshtein Exercise 13: Bringing it all together Exercise 14: Concluding remarks