Skip to content

comet_ml.ConfusionMatrix ¶

ConfusionMatrix(
    y_true=None,
    y_predicted=None,
    labels=None,
    matrix=None,
    title="Confusion Matrix",
    row_label="Actual Category",
    column_label="Predicted Category",
    max_examples_per_cell=25,
    max_categories=25,
    winner_function=None,
    index_to_example_function=None,
    cache=True,
    selected=None,
    images=None,
    experiment=None,
    examples=None,
    **kwargs
)

Data structure for holding a confusion matrix of values and their labels.

Create the ConfusionMatrix data structure.

Parameters:

  • y_true –

    (optional) list of vectors representing the targets, or a list of integers representing the correct label. If not provided, then matrix may be provided.

  • y_predicted –

    (optional) list of vectors representing predicted values, or a list of integers representing the output. If not provided, then matrix may be provided.

  • images –

    (optional) a list of data that can be passed to Experiment.log_image()

  • labels –

    (optional) a list of strings that name of the columns and rows, in order. By default, it will be "0" through the number of categories (e.g., rows/columns).

  • matrix –

    (optional) the confusion matrix (list of lists). Must be square, if given. If not given, then it is possible to provide y_true and y_predicted.

  • examples –

    (optional) if matrix is given you may also give examples, a dictionary of {(x, y): [index1, index2, ...]} where x is column and y is row of matrix

  • title –

    (optional) a custom name to be displayed. By default, it is "Confusion Matrix".

  • row_label –

    (optional) label for rows. By default, it is "Actual Category".

  • column_label –

    (optional) label for columns. By default, it is "Predicted Category".

  • max_examples_per_cell –

    (optional) maximum number of examples per cell. By default, it is 25.

  • max_categories –

    (optional) max number of columns and rows to use. By default, it is 25.

  • winner_function –

    (optional) a function that takes in an entire list of rows of patterns, and returns the winning category for each row. By default, it is argmax.

  • index_to_example_function –

    (optional) a function that takes an index and returns either a number, a string, a URL, or a {"sample": str, "assetId": str} dictionary. See below for more info. By default, the function returns a number representing the index of the example.

  • cache –

    (optional) should the results of index_to_example_function be cached and reused? By default, cache is True.

  • selected –

    (optional) None, or list of selected category indices. These are the rows/columns that will be shown. By default, select is None. If the number of categories is greater than max_categories, and selected is not provided, then selected will be computed automatically by selecting the most confused categories.

  • experiment –

    (optional) passed automatically when using Experiment.create_confusion_matrix().

  • kwargs –

    (optional) any extra keywords and their values will be passed onto the index_to_example_function.

Note

The matrix is [row][col] and [real][predicted] order. That way, the data appears as it is display in the confusion matrix in the user interface on comet.com.

Note

Uses winner_function to compute winning categories for y_true and y_predicted, if they are vectors.

# Typically, you can log the y_true/y_predicted or matrix:

>>> experiment = Experiment()

# If you have a y_true and y_predicted:
>>> y_predicted = model.predict(x_test)
>>> experiment.log_confusion_matrix(y_true, y_predicted)

# Or, if you have the categories for y_true or y_predicted
# you can just pass those in:
>>> experiment.log_confusion_matrix([0, 1, 2, 3],
                                    [2, 2, 2, 2]) # guess 2 for all

# Or, if you have already computed the matrix:
>>> experiment.log_confusion_matrix(labels=["one", "two", "three"],
                                    matrix=[[10, 0, 0],
                                            [ 0, 9, 1],
                                            [ 1, 1, 8],])

# However, if you want to reuse examples from previous runs,
# you can reuse a ConfusionMatrix instance. You might want to
# do this if you are creating a series of confusion matrices
# during the training of a model.
# See https://staging.comet.com/docs/quick-start/ for a tutorial.

>>> cm = ConfusionMatrix()
>>> y_predicted = model.predict(x_test)
>>> cm.compute_matrix(y_true, y_predicted)
>>> experiment.log_confusion_matrix(matrix=cm)

# Log again, using previously cached values:
>>> y_predicted = model.predict(x_test)
>>> cm.compute_matrix(y_true, y_predicted)
>>> experiment.log_confusion_matrix(matrix=cm)

For more details and example uses, please see:

Log Confusion Matrix

or:

comet_ml.Experiment.log_confusion_matrix

Attributes¶

column_label instance-attribute ¶

column_label = column_label

experiment instance-attribute ¶

experiment = experiment

images instance-attribute ¶

images = None

index_to_example_function instance-attribute ¶

index_to_example_function = index_to_example_function

labels instance-attribute ¶

labels = labels

max_categories instance-attribute ¶

max_categories = max_categories

max_examples_per_cell instance-attribute ¶

max_examples_per_cell = max_examples_per_cell

row_label instance-attribute ¶

row_label = row_label

selected instance-attribute ¶

selected = sorted(selected) if selected is not None else None

title instance-attribute ¶

title = title

use_cache instance-attribute ¶

use_cache = cache

winner_function instance-attribute ¶

winner_function = winner_function

Functions¶

clear ¶

clear()

Clear the matrices and type.

clear_cache ¶

clear_cache()

Clear the caches.

compute_matrix ¶

compute_matrix(
    y_true, y_predicted, index_to_example_function=None, images=None, **kwargs
)

Compute the confusion matrix.

Parameters:

  • y_true –

    list of vectors representing the targets, or a list of integers representing the correct label

  • y_predicted –

    list of vectors representing predicted values, or a list of integers representing the output

  • images –

    (optional) a list of data that can be passed to Experiment.log_image()

  • index_to_example_function –

    (optional) a function that takes an index and returns either a number, a string, a URL, or a {"sample": str, "assetId": str} dictionary. See below for more info. By default, the function returns a number representing the index of the example.

Note

Uses winner_function to compute winning categories for y_true and y_predicted, if they are vectors.

default_index_to_example_function ¶

default_index_to_example_function(index, **kwargs)

User-provided function.

Parameters:

  • index –

    the index of the pattern being tested

  • kwargs –

    additional keyword arguments for an overridden method

Returns:

  • –

    an integer representing the winning category

  • –

    a string representing an example

  • –

    a string representing an URL (starts with "http")

  • –

    a dictionary containing keys "sample" and "assetId"

The return dictionary is used to link a confusion matrix cell with a Comet asset. In this function, you can create an asset and return a dictionary, like so:

# Example index_to_example_function
def index_to_example_function(index):
    # x_test is user's inputs (just an example):
    image_array = x_test[index]
    # Make an asset name:
    image_name = "confusion-matrix-%05d.png" % index
    # Make an asset:
    results = experiment.log_image(
        image_array, name=image_name, image_shape=(28, 28, 1)
    )
    # Return the example name and assetId
    return {"sample": image_name, "assetId": results["imageId"]}

# Then, pass it to ConfusionMatrix(), or log_confusion_matrix()

default_winner_function ¶

default_winner_function(ndarray)

A default winner function. Takes a list of patterns to apply winner function to.

Parameters:

  • ndarray –

    a 2-D matrix where rows are the patterns

Returns a list of winning categories.

display ¶

display(space=4)

Display an ASCII version of the confusion matrix.

Parameters:

  • space –

    (optional) column width

image_index_to_example_function ¶

image_index_to_example_function(
    index: int, **kwargs: Dict[str, Any]
) -> Optional[Dict[str, Any]]

The internal index_to_example function used when passing in images to the compute_matrix() method.

Parameters:

  • index (int) –

    the index of the pattern being tested

  • kwargs (Dict[str, Any], default: {} ) –

    additional keyword arguments for Experiment.log_image()

Returns:

  • Optional[Dict[str, Any]] –

    a dictionary containing keys "sample" and "assetId"

initialize ¶

initialize()

Initialize the confusion matrix.

need_init ¶

need_init()

Method to call when you want to reset the confusion matrix. Doesn't reset cache.

Note

This method is called automatically when the confusion matrix is logged.

to_json ¶

to_json()

Return the associated confusion matrix as the JSON to upload.

Dec. 2, 2024