Giskard Scan Results

5 issues detected

Performance 3

Robustness 2

We found some data slices in your dataset on which your model performance is lower than average. Performance bias may happen for different reasons:

Not enough examples in the low-performing data slice in the training set
Wrong labels in the training set in the low-performing data slice
Drift between your training set and test set

To learn more about causes and solutions, check our guide on performance bias.

Issues

1 major 2 medium

`text` contains "van" Balanced Accuracy = 0.735 (Global = 0.929) -20.89% than global 201 samples affected
(11.6% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "van", the Balanced Accuracy is 20.89% lower than the global Balanced Accuracy.

Examples

	text	label	Predicted `label`
807	Mohammed Christus. Vier Schetsen van eene reis in het Oosten, etc	Non-fiction	Fiction (p = 0.74)
835	Leeuwarden na en voor hare wording als stad, en in hare betrekking tot de Leppa. Eene bydrage tot de Geschiedkundige beschrijving, van W. Eekhoff	Non-fiction	Fiction (p = 0.54)
855	Burgerlijk en kerkelijk gedenkboek van Haamstede, deels ook van Burgh, etc	Fiction	Non-fiction (p = 0.97)

Taxonomy

avid-effect:performance:P0204

`text_length(text)` >= 60.500 AND `text_length(text)` < 69.500 Balanced Accuracy = 0.858 (Global = 0.929) -7.59% than global 149 samples affected
(8.6% of dataset) Show details Hide details

Description

For records in the dataset where `text_length(text)` >= 60.500 AND `text_length(text)` < 69.500, the Balanced Accuracy is 7.59% lower than the global Balanced Accuracy.

Examples

	text	text_length(text)	label	Predicted `label`
252	Bush-Life in Queensland; or, John West's colonial experiences	61	Fiction	Non-fiction (p = 1.00)
736	Die Wanderungen der Kelten. Historisch-kritisch dargelegt, etc	62	Fiction	Non-fiction (p = 0.99)
807	Mohammed Christus. Vier Schetsen van eene reis in het Oosten, etc	65	Non-fiction	Fiction (p = 0.74)

Taxonomy

avid-effect:performance:P0204

`text_length(text)` >= 101.500 AND `text_length(text)` < 147.500 Balanced Accuracy = 0.876 (Global = 0.929) -5.69% than global 301 samples affected
(17.3% of dataset) Show details Hide details

Description

For records in the dataset where `text_length(text)` >= 101.500 AND `text_length(text)` < 147.500, the Balanced Accuracy is 5.69% lower than the global Balanced Accuracy.

Examples

	text	text_length(text)	label	Predicted `label`
805	Verdens Storbyer [Translated from 'Les Capitales du monde.'] Paa Dansk af P. Nansen. Med 322 Afbildninger	105	Non-fiction	Fiction (p = 0.66)
813	De stad Utrecht en hare geschiedenis, voorafgegaan door eene algemeene geschied- en aardrijkskundige beschouwing over de provincie Utrecht	138	Fiction	Non-fiction (p = 0.78)
835	Leeuwarden na en voor hare wording als stad, en in hare betrekking tot de Leppa. Eene bydrage tot de Geschiedkundige beschrijving, van W. Eekhoff	145	Non-fiction	Fiction (p = 0.54)

Taxonomy

avid-effect:performance:P0204

Your model seems to be sensitive to small perturbations in the input data. These perturbations can include adding typos, changing word order, or turning text into uppercase or lowercase. This happens when:

There is not enough diversity in the training data
Overreliance on spurious correlations like the presence of specific word
Use of complex models with large number of parameters that tend to overfit the training data

To learn more about causes and solutions, check our guide on robustness issues.

Issues

1 major 1 medium

Feature `text`

Transform to uppercase

Fail rate = 0.243

243/1000 tested samples (24.3%) changed prediction after perturbation

1000 samples affected
(57.6% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 24.3% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Transform to uppercase(text)	Original prediction	Prediction after perturbation
657	The Wish. A novel by Hermann Sudermann. Translated by Lily Henkel	THE WISH. A NOVEL BY HERMANN SUDERMANN. TRANSLATED BY LILY HENKEL	Fiction (p = 1.00)	Non-fiction (p = 0.81)
289	A Great Gulf Fixed. A tale	A GREAT GULF FIXED. A TALE	Fiction (p = 1.00)	Non-fiction (p = 0.96)
572	Original Sonnets, elegiac, ethic and erotic: with some miscellaneous productions and imitations	ORIGINAL SONNETS, ELEGIAC, ETHIC AND EROTIC: WITH SOME MISCELLANEOUS PRODUCTIONS AND IMITATIONS	Fiction (p = 1.00)	Non-fiction (p = 0.98)

Taxonomy

avid-effect:performance:P0201

Feature `text`

Transform to title case

Fail rate = 0.050

50/1000 tested samples (5.0%) changed prediction after perturbation

1000 samples affected
(57.6% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 5.0% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Transform to title case(text)	Original prediction	Prediction after perturbation
1268	Ça ira! or Danton in the French Revolution. A study	Ça Ira! Or Danton In The French Revolution. A Study	Non-fiction (p = 0.98)	Fiction (p = 1.00)
1078	Two Women in the Klondike. The story of a journey to the gold-fields of Alaska ... With 105 illustrations and map	Two Women In The Klondike. The Story Of A Journey To The Gold-Fields Of Alaska ... With 105 Illustrations And Map	Non-fiction (p = 1.00)	Fiction (p = 0.64)
489	The passionate pilgrim, or Eros and Anteros. By Henry J. Thurstan [pseudonym of F. T. Palgrave]	The Passionate Pilgrim, Or Eros And Anteros. By Henry J. Thurstan [Pseudonym Of F. T. Palgrave]	Non-fiction (p = 0.99)	Fiction (p = 0.87)

Taxonomy

avid-effect:performance:P0201

Debug your issues in the Giskard hub

Install the Giskard hub app to:

Debug and diagnose your scan issues
Save your scan result as a re-executable test suite to benchmark your model
Extend your test suite with our catalog of ready-to-use tests

You can find installation instructions here.

from giskard import GiskardClient

# Create a test suite from your scan results
test_suite = results.generate_test_suite("My first test suite")

# Upload your test suite to your Giskard hub instance
client = GiskardClient("http://localhost:19000", "GISKARD_API_KEY")
client.create_project("my_project_id", "my_project_name")
test_suite.upload(client, "my_project_id")