Giskard Scan Results

5 issues detected

Performance 4

Robustness 1

We found some data slices in your dataset on which your model performance is lower than average. Performance bias may happen for different reasons:

Not enough examples in the low-performing data slice in the training set
Wrong labels in the training set in the low-performing data slice
Drift between your training set and test set

To learn more about causes and solutions, check our guide on performance bias.

Issues

4 medium

`text` contains "movie" Precision = 0.837 (Global = 0.918) -8.81% than global 95 samples affected
(10.9% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "movie", the Precision is 8.81% lower than the global Precision.

Examples

	text	label	Predicted `label`
69	this one is definitely one to skip , even for horror movie fanatics .	LABEL_0	LABEL_1 (p = 0.95)
172	it seems like i have been waiting my whole life for this movie and now i ca n't wait for the sequel .	LABEL_1	LABEL_0 (p = 0.72)
509	a movie that successfully crushes a best selling novel into a timeframe that mandates that you avoid the godzilla sized soda .	LABEL_1	LABEL_0 (p = 0.91)

Taxonomy

avid-effect:performance:P0204

`text_length(text)` < 82.500 AND `text_length(text)` >= 73.500 Recall = 0.870 (Global = 0.935) -6.97% than global 45 samples affected
(5.2% of dataset) Show details Hide details

Description

For records in the dataset where `text_length(text)` < 82.500 AND `text_length(text)` >= 73.500, the Recall is 6.97% lower than the global Recall.

Examples

	text	text_length(text)	label	Predicted `label`
93	if steven soderbergh 's ` solaris ' is a failure it is a glorious failure .	76	LABEL_1	LABEL_0 (p = 0.59)
142	what better message than ` love thyself ' could young women of any size receive ?	82	LABEL_1	LABEL_0 (p = 0.98)
411	i do n't mind having my heartstrings pulled , but do n't treat me like a fool .	80	LABEL_0	LABEL_1 (p = 0.95)

Taxonomy

avid-effect:performance:P0204

`text_length(text)` >= 165.500 AND `text_length(text)` < 183.500 Recall = 0.872 (Global = 0.935) -6.73% than global 61 samples affected
(7.0% of dataset) Show details Hide details

Description

For records in the dataset where `text_length(text)` >= 165.500 AND `text_length(text)` < 183.500, the Recall is 6.73% lower than the global Recall.

Examples

	text	text_length(text)	label	Predicted `label`
266	a coda in every sense , the pinochet case splits time between a minute-by-minute account of the british court 's extradition chess game and the regime 's talking-head survivors .	179	LABEL_1	LABEL_0 (p = 0.85)
282	while there 's something intrinsically funny about sir anthony hopkins saying ` get in the car , bitch , ' this jerry bruckheimer production has little else to offer	166	LABEL_1	LABEL_0 (p = 1.00)
292	the story and the friendship proceeds in such a way that you 're watching a soap opera rather than a chronicle of the ups and downs that accompany lifelong friendships .	170	LABEL_0	LABEL_1 (p = 0.88)

Taxonomy

avid-effect:performance:P0204

`text_length(text)` < 98.500 AND `text_length(text)` >= 86.500 Precision = 0.861 (Global = 0.918) -6.21% than global 76 samples affected
(8.7% of dataset) Show details Hide details

Description

For records in the dataset where `text_length(text)` < 98.500 AND `text_length(text)` >= 86.500, the Precision is 6.21% lower than the global Precision.

Examples

	text	text_length(text)	label	Predicted `label`
115	sam mendes has become valedictorian at the school for soft landings and easy ways out .	88	LABEL_0	LABEL_1 (p = 0.98)
230	reign of fire looks as if it was made without much thought -- and is best watched that way .	93	LABEL_1	LABEL_0 (p = 1.00)
519	moretti 's compelling anatomy of grief and the difficult process of adapting to loss .	87	LABEL_0	LABEL_1 (p = 1.00)

Taxonomy

avid-effect:performance:P0204

Your model seems to be sensitive to small perturbations in the input data. These perturbations can include adding typos, changing word order, or turning text into uppercase or lowercase. This happens when:

There is not enough diversity in the training data
Overreliance on spurious correlations like the presence of specific word
Use of complex models with large number of parameters that tend to overfit the training data

To learn more about causes and solutions, check our guide on robustness issues.

Issues

1 major

Feature `text`

Add typos

Fail rate = 0.125

100/800 tested samples (12.5%) changed prediction after perturbation

800 samples affected
(91.7% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 12.5% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Add typos(text)	Original prediction	Prediction after perturbation
16	the emotions are raw and will strike a nerve with anyone who 's ever had family trauma .	the ekotions are raw andw ill strike a nerve with anyone wgo 's ever had family trauma .	LABEL_1 (p = 1.00)	LABEL_0 (p = 0.89)
22	holden caulfield did it better .	holdsn caulfkeld did t better .	LABEL_1 (p = 0.99)	LABEL_0 (p = 0.98)
36	the weight of the piece , the unerring professionalism of the chilly production , and the fascination embedded in the lurid topic prove recommendation enough .	he weight of the piec e hte unerring professionalism of the chilly production , and the fascination embeded in the lurid topic prove rrcommendatioh enough .	LABEL_1 (p = 1.00)	LABEL_0 (p = 0.98)

Taxonomy

avid-effect:performance:P0201

Debug your issues in the Giskard hub

Install the Giskard hub app to:

Debug and diagnose your scan issues
Save your scan result as a re-executable test suite to benchmark your model
Extend your test suite with our catalog of ready-to-use tests

You can find installation instructions here.

from giskard import GiskardClient

# Create a test suite from your scan results
test_suite = results.generate_test_suite("My first test suite")

# Upload your test suite to your Giskard hub instance
client = GiskardClient("http://localhost:19000", "GISKARD_API_KEY")
client.create_project("my_project_id", "my_project_name")
test_suite.upload(client, "my_project_id")