cometx¶
_________ ____ ___ ___ / /__ __
/ ___/ __ \/ __ `__ \/ _ \/ __/ |/_/
/ /__/ /_/ / / / / / / __/ /__> <
\___/\____/_/ /_/ /_/\___/\__/_/|_|
Open source extensions for the Comet SDK.
These extensions are created and supported by the community and are not an official project of Comet ML. We welcome contributions!
Installation¶
pip install cometx --upgrade
To use these command-line functions, you can set your Comet API key and URL override using command-line flags:
cometx --api-key="YOUR-COMET-API-KEY" COMMAND
If you are a Comet on-prem user, and your installation does not use smart-keys, you'll also need to set the URL override:
cometx --api-key="YOUR-COMET-API-KEY" --url-override="https://your-companys-comet.com/clientlib/" COMMAND
Usage¶
cometx is composed of a series of commands that are useful independently, and can be used together to create sophisticated tools for ML management.
Commands¶
- cometx admin
- cometx config
- cometx copy
- cometx delete-assets
- cometx download
- cometx list
- cometx log
- cometx reproduce
- cometx smoke-test
- cometx update
For all commands, use the --help flag to get additional information.
cometx also includes a Python API to upload panels and to create and update dashboards. See Panels and Dashboards (Python API).
Global Options¶
These flags are availble before a command:
--api-key API_KEY- Set the COMET_API_KEY--url-override URL_OVERRIDE- Set the COMET_URL_OVERRIDE
This command shows the cometx version:
--version- Display comet_ml version
This command can be used globally, or for individual commands:
-h, --help- Show help message
cometx list¶
This command is used to:
- get a list of all workspaces that you are a member of
- get a list of all projects in a workspace
- get a list of all experiments (by name or key) in a project
cometx list examples:
cometx list WORKSPACE/PROJECT/EXPERIMENT-KEY-OR-NAME
cometx list WORKSPACE/PROJECT
cometx list WORKSPACE
cometx list
Flags¶
-u, --use-name- Use experiment names for experiment folders and listings--query QUERY- Only list experiments that match this Comet query string--debug- Provide debug info
For more information, cometx list --help
cometx download¶
This command is used to:
- download all workspaces, projects, and experiments of workspaces that you are a member of
- download all projects, and experiments of a given workspace
- download all experiments of a given workspace/project
- download artifacts and models from the registry
- download panels
Note: For detailed information on copying experiments from one Comet installation to another, see MIGRATIONS.md.
cometx download examples:
cometx download WORKSPACE/PROJECT/EXPERIMENT-KEY-OR-NAME [RESOURCE ...] [FLAGS ...]
cometx download WORKSPACE/PROJECT [RESOURCE ...] [FLAGS ...]
cometx download WORKSPACE [RESOURCE ...] [FLAGS ...]
cometx download [RESOURCE ...] [FLAGS ...]
Where [RESOURCE ...] is zero or more of the following names:
- run - alias for: code, git, output, graph, and requirements
- system
- others
- parameters
- metadata
- metrics
- assets
- html
- project - alias for: project_notes, project_metadata
If no RESOURCE is given it will download all of them.
Flags¶
--from from- Source of data to copy. Should be: comet, wandb, or neptune-i IGNORE [IGNORE ...], --ignore IGNORE [IGNORE ...]- Resource(s) (or 'experiments') to ignore-j PARALLEL, --parallel PARALLEL- The number of threads to use for parallel downloading; default (None) is based on CPUs-o OUTPUT, --output OUTPUT- Output directory for downloads-u, --use-name- Use experiment names for experiment folders and listings-l, --list- List the items at this level (workspace, project, experiment, artifacts, or model-registry) rather than download--flat- Download the files without subfolders-f, --ask- Queries the user; if flag not included system will answeryesfor all queries--filename FILENAME- Only get resources ending with this--query QUERY- Only download experiments that match this Comet query string--asset-type ASSET_TYPE- Only get assets with this type--sync SYNC- What level to sync at: all, experiment, project, or workspace--debug- Provide debug info
To download artifacts:
cometx download WORKSPACE/artifacts/NAME [FLAGS ...]
cometx download WORKSPACE/artifacts/NAME/VERSION-OR-ALIAS [FLAGS ...]
To download models from the model registry:
cometx download WORKSPACE/model-registry/NAME [FLAGS ...]
cometx download WORKSPACE/model-registry/NAME/VERSION-OR-STAGE [FLAGS ...]
To download panels:
cometx download WORKSPACE/panels/NAME-OR-ID [FLAGS ...]
cometx download WORKSPACE/panels [FLAGS ...]
For more information, cometx download --help
cometx copy¶
This command is used to:
- copy downloaded data to a new experiment
- create a symlink from one project to existing experiments
- copy panels
Note: For detailed information on copying experiments from one Comet installation to another, see MIGRATIONS.md.
cometx copy examples:
cometx copy SOURCE DESTINATION
cometx copy --symlink SOURCE DESTINATION
where SOURCE is:
- if not
--symlink, "WORKSPACE/PROJECT/EXPERIMENT", "WORKSPACE/PROJECT", or "WORKSPACE" folder - if
--symlink, then it is a Comet path to workspace or workspace/project - "WORKSPACE/panels" or "WORKSPACE/panels/PANEL-ZIP-FILENAME" to copy panels
where DESTINATION is:
- WORKSPACE
- WORKSPACE/PROJECT
Not all combinations are possible:
| Destination → Source ↓ | WORKSPACE | WORKSPACE/PROJECT |
|---|---|---|
WORKSPACE/*/* | Copies all projects | N/A |
WORKSPACE/PROJ/* | N/A | Copies all experiments |
WORKSPACE/PROJ/EXP | N/A | Copies experiment |
Asset Types¶
- 3d-image
- 3d-points - deprecated
- audio
- confusion-matrix - may contain assets
- curve
- dataframe
- dataframe-profile
- datagrid
- embeddings - may reference image asset
- histogram2d - not used
- histogram3d - internal only, single histogram, partial logging
- histogram_combined_3d
- image
- llm_data
- model-element
- notebook
- source_code
- tensorflow-model-graph-text - not used
- text-sample
- video
Flags¶
-i IGNORE [IGNORE ...], --ignore IGNORE [IGNORE ...]- Resource(s) (or 'experiments') to ignore--debug- If given, allow debugging--quiet- If given, don't display update info--symlink- Instead of copying, create a link to an experiment in a project--sync- Check to see if experiment name has been created first; if so, skip
For more information, cometx copy --help
cometx log¶
This command is used to log a resource (metrics, parameters, asset, etc) file to a specific experiment or experiments.
cometx log examples:
cometx log WORKSPACE/PROJECT/EXPERIMENT-KEY FILENAME ... --type=TYPE
cometx log WORKSPACE PANEL-ZIP-FILENAME ... --type=panel
cometx log WORKSPACE PANEL.py ... --type=panel
cometx log WORKSPACE PANEL-URL ... --type=panel
cometx log WORKSPACE/PROJECT --type=other --set "key:value"
cometx log WORKSPACE --type=other --set "key:value"
Where TYPE is one of the following names:
- all
- asset
- audio
- code
- image
- metrics
- notebook
- panel
- tensorflow-file
- text-sample
- video
- other
- tensorboard-folder-assets
Flags¶
--type TYPE- The type of item to log--set SET- The key:value to log--query QUERY- A Comet Query string, see https://www.comet.com/docs/v2/api-and-sdk/python-sdk/reference/API/#apiquery--debug- If given, allow debugging--use-base-name- If given, using the basename for logging assets
For more information, cometx log --help
cometx delete-assets¶
To delete experiments assets:
cometx delete-assets WORKSPACE/PROJECT --type=image
cometx delete-assets WORKSPACE/PROJECT/EXPERIMENT --type=all
Type can be valid asset type, including:
- all
- asset
- audio
- code
- image
- notebook
- text-sample
- video
Flags¶
--type TYPE- The type of item to log--debug- If given, allow debugging--query QUERY- Only delete experiments that match this Comet query string
For more information, cometx delete-assets --help
cometx config¶
To enable auto-logging of your notebooks in Jupyter environments:
cometx config --auto-log-notebook yes
To turn auto-logging of notebooks off, use:
cometx config --auto-log-notebook no
If you keep the generated experiment URLs in the notebook, but later edit the notebook, the notebooks will be updated in all of the experiments created in the notebook.
Flags¶
--debug- If given, allow debugging--auto-log-notebook AUTO_LOG_NOTEBOOK- Takes a 1/yes/true, or 0/no/false
For more information, cometx config --help
cometx reproduce¶
cometx reproduce [-h] [--run] [--executable EXECUTABLE] COMET_PATH OUTPUT_DIR
This command is used to reproduce experiments by copying files to a specified output directory.
Flags¶
--run- Run the reproducible script--executable EXECUTABLE- Run the reproducible script with specified executable
For more information, cometx reproduce --help
cometx update¶
cometx update [-h] [--debug] COMET_SOURCE COMET_DESTINATION
To update existing experiments.
cometx update SOURCE DESTINATION
where SOURCE is a folder:
- "WORKSPACE/PROJECT/EXPERIMENT"
- "WORKSPACE/PROJECT"
- "WORKSPACE"
where DESTINATION is a Comet:
- WORKSPACE
- WORKSPACE/PROJECT
Flags¶
--debug- If given, allow debugging
For more information, cometx update --help
cometx admin¶
cometx admin [-h] [--host HOST] [--debug] ACTION [YEAR-MONTH]
To perform admin functions
cometx admin chargeback-report
Flags¶
--host HOST- Override the HOST URL--debug- If given, allow debugging
For more information, cometx admin --help
cometx smoke-test¶
cometx smoke-test [-h] [--exclude [EXCLUDE ...]] [--debug DEBUG] COMET_PATH [include ...]
Perform a smoke test on a Comet installation. Logs results to WORKSPACE/smoke-tests or WORKSPACE/PROJECT.
Examples:
Run all tests:
cometx smoke-test WORKSPACE # project defaults to smoke-tests
cometx smoke-test WORKSPACE/PROJECT
Run everything except mpm tests:
cometx smoke-test WORKSPACE/PROJECT --exclude mpm
Run just optimizer tests:
cometx smoke-test WORKSPACE/PROJECT optimizer
Run just metric tests:
cometx smoke-test WORKSPACE/PROJECT metric
Items to include or exclude:
- optimizer
- mpm
- panel
- opik
- experiment
- metric
- image
- asset
- dataset-info
- confusion-matrix
- embedding
Flags¶
--exclude [EXCLUDE ...]- Items to exclude; any of: asset, confusion-matrix, dataset-info, embedding, experiment, image, metric, mpm, opik, optimizer, panel--debug DEBUG- Show debugging information
For more information, cometx smoke-test --help
Copy/Download Use Cases¶
In this section we'll explore some common scenarios.
- Copy a specific project from one Comet installation to another
- Copy all projects in workspace to a new workspace
- Copy specific experiments in a project to new experiments
1. Copy a specific project from one comet installation to another¶
A useful idiom is to set your Comet environment variables on the line of a command. In this manner, you can set the COMET_URL_OVERRIDE and COMET_API_KEY for different installations.
Of course, you don't have to set the environment variables if you are copying experiments on the same Comet installation.
Here is how you one could download the experiments in WORKSPACE/PROJECT from http://comet.a.com:
cometx --api-key=A-KEY download WORKSPACE/PROJECT
The cometx download command downloads all of the Comet experiment data into local files. Note that WORKSPACE/PROJECT refers to a workspace and project on http://comet.a.com.
One could then copy the downloaded experiment data with a similar command:
cometx --api-key=B-KEY copy WORKSPACE/PROJECT NEW-WORKSPACE/NEW-PROJECT
Note that WORKSPACE/PROJECT now refers to a directory, and NEW-WORKSPACE/NEW-PROJECT refers to a workspace and project on http://comet.b.com.
2. Copy all projects in workspace to a new workspace¶
Similarly, one can copy all of the projects by first downloading them:
cometx --api-key=A-KEY download WORKSPACE
and then copying them:
cometx --api-key=B-KEY copy WORKSPACE NEW-WORKSPACE
3. Copy specific experiments in a project to new experiments¶
Similarly, one can copy a single experiment first downloading it:
cometx --api-key=A-KEY download WORKSPACE/PROJECT/EXPERIMENT-NAME-OR-ID
and then copying it:
cometx --api-key=B-KEY copy WORKSPACE/PROJECT/EXPERIMENT-NAME-OR-ID NEW-WORKSPACE/NEW-PROJECT
Panels and Dashboards (Python API)¶
In addition to the command-line tools, cometx includes a Python API for managing code panels and dashboards programmatically. The cometx.API class extends the comet_ml API class with methods to upload panels and to create and update dashboards. The cometx.views module builds dashboard layouts: sections, charts, filters, and experiment table columns.
Note
These features require cometx 3.6.8 or later. cometx.views also requires comet_ml 3.57.0 or later, which adds the get_views() and create_view() methods.
Upload a panel¶
Use upload_panel_code(workspace, panel_name, code, template_id=None) to upload Python code as a panel in a workspace. It returns a dict that contains the panel's templateId. If you pass a template_id, the existing panel is overwritten in place instead of creating a new panel.
from cometx import API
api = API()
# Create a new panel
result = api.upload_panel_code("my-workspace", "My Panel", "print('hello world')")
panel_id = result["templateId"]
# Update an existing panel's code in place
result = api.upload_panel_code(
"my-workspace", "My Panel", "print('updated')", template_id=panel_id
)
Use upload_panel_zip(workspace, filename, template_id=None) to upload a panel zip file. The template_id argument works in the same way:
# Create a new panel
result = api.upload_panel_zip("my-workspace", "panel.zip")
# Update an existing panel in place
result = api.upload_panel_zip("my-workspace", "panel.zip", template_id="abc123")
Note
Dashboards that use a panel show the updated code after an in-place update. Refresh the browser page to see the change in the Comet UI.
Create a dashboard¶
Use create_dashboard(workspace, project_name, template_name, template_id=None, panels=None) to create a new dashboard in a project:
from cometx import API
api = API()
# Create an empty dashboard
dashboard = api.create_dashboard(
workspace="my-workspace",
project_name="my-project",
template_name="My Dashboard",
)
# Create a dashboard with panels
dashboard = api.create_dashboard(
workspace="my-workspace",
project_name="my-project",
template_name="My Dashboard",
panels=["panel-template-id-1", "panel-template-id-2"],
)
# Create a dashboard cloned from an existing one (copies panel associations)
dashboard = api.create_dashboard(
workspace="my-workspace",
project_name="my-project",
template_name="My Dashboard Copy",
template_id="existing-dashboard-template-id",
)
| Parameter | Type | Description |
|---|---|---|
workspace | str | The workspace name the project belongs to |
project_name | str | The project name to create the dashboard in |
template_name | str | Name for the new dashboard |
template_id | str (optional) | Source dashboard template ID to clone panel associations from |
panels | list (optional) | List of panel templateId values to include in the new dashboard |
Returns a dict that represents the created DashboardTemplate, including its new template_id.
Update a dashboard¶
Use update_dashboard(workspace, project_name, template_id, template_name=None, panels=None) to rename an existing dashboard, replace its panels, or both:
from cometx import API
api = API()
# Rename a dashboard
api.update_dashboard(
workspace="my-workspace",
project_name="my-project",
template_id="existing-dashboard-id",
template_name="Renamed Dashboard",
)
# Replace the panels on a dashboard
api.update_dashboard(
workspace="my-workspace",
project_name="my-project",
template_id="existing-dashboard-id",
panels=["panel-template-id-1", "panel-template-id-2"],
)
| Parameter | Type | Description |
|---|---|---|
workspace | str | The workspace name the project belongs to |
project_name | str | The project name the dashboard belongs to |
template_id | str | The ID of the dashboard to update |
template_name | str (optional) | New name for the dashboard |
panels | list (optional) | List of panel templateId values to set on the dashboard |
Returns a dict that represents the updated DashboardTemplate.
Typical workflow¶
from cometx import API
api = API()
# 1. Upload a panel
result = api.upload_panel_code("my-workspace", "Hello World", "print('hello world')")
panel_id = result["templateId"]
# 2. Create a dashboard with the panel
dashboard = api.create_dashboard(
workspace="my-workspace",
project_name="my-project",
template_name="My Dashboard",
panels=[panel_id],
)
dashboard_id = dashboard["template_id"]
# 3. Later, update the panel code in place
api.upload_panel_code(
"my-workspace", "Hello World", "print('updated')", template_id=panel_id
)
# Refresh your browser to see the updated panel in the Comet UI
# 4. Add more panels to the dashboard
api.update_dashboard(
workspace="my-workspace",
project_name="my-project",
template_id=dashboard_id,
panels=[panel_id, "another-panel-id"],
)
Build dashboard layouts with cometx.views¶
create_dashboard() and update_dashboard() manage the dashboard record: its name, its ID, and the code panels attached to it. They do not set the dashboard layout. The layout (sections, charts, axes, filters, and table columns) is stored in the dashboard's serialized state, and cometx.views builds that state.
| To do this | Use |
|---|---|
| Attach uploaded code panels to a dashboard | API.create_dashboard() / API.update_dashboard() |
| Lay out built-in charts, sections, and filters | cometx.views + API.create_view() |
cometx.views makes no network calls. It returns a comet_ml View object. Pass this object to API.create_view(workspace, project_name, view) for a project dashboard, or to APIExperiment.create_view(view) for the page of a single experiment.
from cometx import API
from cometx.views import Section, build_view, line_panel, scalar_panel
view = build_view(
"Training Overview",
sections=[
Section(
"At a glance",
[
scalar_panel("accuracy", aggregation="max"),
scalar_panel("loss", aggregation="min"),
],
columns=2,
),
Section(
"Curves",
[
line_panel(["loss", "val_loss"], name="Loss", smoothing=0.6),
line_panel(["accuracy", "val_accuracy"], name="Accuracy"),
],
columns=2,
),
],
)
API().create_view("my-workspace", "my-project", view)
The new dashboard appears in the view selector at the top of the project page.
Panel factories¶
Each factory returns a panel dict with a new chartId. Build a new view object for each target. Do not reuse one view object across projects or experiments.
| Factory | Produces |
|---|---|
line_panel(y, x="step", smoothing=0, transform_y=None, locked=False, ...) | Line chart; y can be a list of metrics to overlay |
bar_panel(metric, aggregation="last", plot_type="BAR", group_by_aggregation=None, ...) | Bar, box, or violin plot, one bar per experiment |
scatter_panel(x, y, z=None, metrics=None, params=None, ...) | 2D or 3D scatter plot, one point per experiment |
parallel_panel(params, metrics=(), target="", ...) | Parallel coordinates chart for a hyperparameter sweep |
scalar_panel(metric, aggregation="last", precision=3, ...) | Single-number tile |
image_panel(images=(), step=None, ...) | Image grid across experiments |
data_panel(file_name, axis=0, join_type="outer", ...) | Table from a logged tabular asset |
custom_panel(instance_id, instance_name="", ...) | An instance of a JavaScript panel |
python_panel(template_id, revision_id="", ...) | An instance of a Python panel |
legend_key(value, label=None, source="log_other") builds one entry of a chart's legend. Use it to color a chart by experiment name, tags, or a hyperparameter.
Note
You can create built-in panels from nothing. custom_panel() and python_panel() refer to a panel instance that already exists, so you cannot invent their IDs. Read the IDs from a panel that is already on a dashboard with get_views(), or from the panel's URL in the Comet UI.
Sections¶
Section(title, panels=(), columns=3, expanded=True, height=1) is a titled, collapsible group of panels. The dashboard grid is 6 units wide, so columns must divide 6 evenly (1, 2, 3, or 6). Section calculates the grid position of each panel for you. height is the row height in grid units (maximum 3).
from cometx.views import Section, image_panel, data_panel
section = Section("Media & tables", columns=2, height=2, expanded=False)
section.add(image_panel(["prediction-grid.png"]), data_panel("per_class_metrics.csv"))
add() returns the section, so you can chain calls.
Project dashboards¶
build_view(name, sections, filters=None, table=None, config=None, auto_generate=False, personal=False) assembles sections into a project dashboard View.
| Parameter | Type | Description |
|---|---|---|
name | str | The display name of the dashboard |
sections | list | Section objects, from top to bottom |
filters | str (optional) | Output of query_state(): a saved filter |
table | str (optional) | Output of table_state(): experiment table columns and sorting |
config | dict (optional) | Output of global_config(): dashboard-wide x-axis, smoothing, and sampling |
auto_generate | bool | If True, Comet adds auto-generated panels for newly logged metrics below your sections |
personal | bool | If True, only you can see the dashboard |
Build filters with rule(), all_of(), and any_of():
from cometx.views import (
all_of, any_of, build_view, global_config, query_state, rule, table_state
)
view = build_view(
"Successful runs",
sections=[...],
# accuracy > 0.9 OR loss < 0.1
filters=query_state(
any_of(
all_of(rule("accuracy", "greater", 0.9)),
all_of(rule("loss", "less", 0.1)),
)
),
table=table_state(
columns=["Name", "experimentTags", "learning_rate", "accuracy", "duration"],
sort_by="accuracy",
page_size=50,
),
config=global_config(x_axis="step", smoothing=0.3, sample_size=500),
)
global_config() applies to all panels that do not have locked=True. A locked panel keeps its own axis and smoothing settings.
Experiment views¶
build_experiment_view(name, sections, x_axis="step", smoothing=0, ...) assembles the same sections into a view for the page of one experiment. Pass the result to APIExperiment.create_view(view):
from comet_ml import API as CometAPI
from cometx.views import Section, build_experiment_view, line_panel
experiment = CometAPI().get_experiments("my-workspace", "my-project")[0]
view = build_experiment_view(
"Run layout",
sections=[Section("Curves", [line_panel(["loss", "val_loss"])], columns=2)],
smoothing=0.4,
)
experiment.create_view(view)
Project dashboards and experiment views store their state in different places: project state is in view.v3, and experiment state is in view.v2["panels"]. For this reason, there are two builders. Sections and panels are the same for both. Do not set experiment_key yourself; create_view() sets it for you.
Warning
APIExperiment.create_view() accepts the result of build_view(), but the view that it creates has no panels. Use build_experiment_view() for experiment views.
Inspect a view¶
describe_view(view, indent="") returns a readable outline of the sections, panels, and filters of a view. Use it to examine the results of get_views() before you copy or change them:
from cometx import API
from cometx.views import describe_view
for view in API().get_views("my-workspace", "my-project"):
print(describe_view(view))
Known limitations¶
create_view()always creates a new dashboard. If you call it two times with the same name, you get two dashboards, and there is nodelete_view(). To update a dashboard in place, you need its existingtemplate_id.create_view()cannot keep this ID, because it callsas_portable(), which clears it. Use the lower-level upsert instead:view.template_id = existing_view.template_id api._client.upsert_view(api.get_project(workspace, project)["projectId"], view)Panels for metrics that the project never logged are empty. They do not show an error. When you apply one layout to many projects, read the metric names of each project first, and build only the panels that have data.
as_portable()makes a view movable between projects, workspaces, or accounts. It removestemplate_id,project_id,experiment_key,pinned_experiments, and theexperimentKeyinv2andv3. It does not change experiment keys insidechart_state, so you must manually fix a panel that is pinned to a specific run."Unsaved Changes"views are the per-user scratch state of the UI for a project. They are included in the results ofget_views(). Skip them when you copy views.
Examples¶
The examples/ directory of the cometx repository has runnable scripts for all of the features above. See the examples README for a full walkthrough. Scripts 04, 05, and 06 do a dry run by default; pass --apply to write changes.
| Script | Demonstrates |
|---|---|
00_seed_demo_project.py | Logs six small runs (metrics, parameters, an image, and a CSV table) so that each panel type in the other scripts has data. |
01_inspect_views.py | Reads views at the project level (API.get_views) and the experiment level (APIExperiment.get_views). |
02_build_project_view.py | Builds a project dashboard with most panel types, a saved filter, a configured experiment table, and dashboard-wide smoothing and sampling. |
03_build_experiment_view.py | Builds the same sections as an experiment view. --all applies one layout to every run in a project. |
04_migrate_views.py | Copies views between projects, workspaces, or accounts (--dst-api-key), with --match filtering. |
05_standardize_workspace.py | Applies one standard dashboard to all projects in a workspace. --update updates dashboards in place instead of adding duplicates. |
06_harvest_panels.py | Finds custom JavaScript and Python panels in existing views and rebuilds them into a new view. |
Running Tests¶
WARNING: Running the tests will create experiments, models, assets, etc. in your default workspace if not set otherwise.
To run the tests, you can either export all of these items in the environment:
$ export COMET_USER="<USERNAME>"
$ export COMET_WORKSPACE="<WORKSPACE>"
$ export COMET_API_KEY="<API-KEY>"
$ pytest tests
Or, define workspace and api_key in your ~/.comet.config file:
$ export COMET_USER="<USERNAME>"
$ pytest tests