Giskard Scan Results

8 issues detected

Robustness 5

Performance 3

Your model seems to be sensitive to small perturbations in the input data. These perturbations can include adding typos, changing word order, or turning text into uppercase or lowercase. This happens when:

There is not enough diversity in the training data
Overreliance on spurious correlations like the presence of specific word
Use of complex models with large number of parameters that tend to overfit the training data

To learn more about causes and solutions, check our guide on robustness issues.

Issues

3 major 2 medium

Feature `text`

Transform to uppercase

Fail rate = 0.217

170/782 tested samples (21.74%) changed prediction after perturbation

782 samples affected
(99.7% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 21.74% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Transform to uppercase(text)	Original prediction	Prediction after perturbation
1	Just walked in to #Starbucks and asked for a "tall blonde" Hahahaha #irony	JUST WALKED IN TO #STARBUCKS AND ASKED FOR A "TALL BLONDE" HAHAHAHA #IRONY	irony (p = 0.65)	non_irony (p = 0.78)
9	People who tell people with anxiety to "just stop worrying about it" are my favorite kind of people #not #educateyourself	PEOPLE WHO TELL PEOPLE WITH ANXIETY TO "JUST STOP WORRYING ABOUT IT" ARE MY FAVORITE KIND OF PEOPLE #NOT #EDUCATEYOURSELF	irony (p = 0.87)	non_irony (p = 0.51)
10	Most important thing I've learned in school	MOST IMPORTANT THING I'VE LEARNED IN SCHOOL	irony (p = 0.91)	non_irony (p = 0.71)

Taxonomy

avid-effect:performance:P0201

Feature `text`

Transform to title case

Fail rate = 0.144

113/783 tested samples (14.43%) changed prediction after perturbation

783 samples affected
(99.9% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 14.43% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Transform to title case(text)	Original prediction	Prediction after perturbation
1	Just walked in to #Starbucks and asked for a "tall blonde" Hahahaha #irony	Just Walked In To #Starbucks And Asked For A "Tall Blonde" Hahahaha #Irony	irony (p = 0.65)	non_irony (p = 0.54)
21	The definition of #IRONY would be if a 77-year-old rapper went #viral and took #BITCOIN mainstream. Maybe only way #babyboomers will buy in.	The Definition Of #Irony Would Be If A 77-Year-Old Rapper Went #Viral And Took #Bitcoin Mainstream. Maybe Only Way #Babyboomers Will Buy In.	irony (p = 0.82)	non_irony (p = 0.58)
22	Pretty excited about how you gave up on me. File Under: #sarcasm	Pretty Excited About How You Gave Up On Me. File Under: #Sarcasm	irony (p = 0.51)	non_irony (p = 0.80)

Taxonomy

avid-effect:performance:P0201

Feature `text`

Add typos

Fail rate = 0.132

95/717 tested samples (13.25%) changed prediction after perturbation

717 samples affected
(91.5% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 13.25% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Add typos(text)	Original prediction	Prediction after perturbation
3	@user He is exactly that sort of person. Weirdo!	@user He is exatvly that sort of person. Weirdo!	non_irony (p = 0.79)	irony (p = 0.81)
22	Pretty excited about how you gave up on me. File Under: #sarcasm	Pretty xecite dabout how you gave up kn me. File Under: #sarcasm	irony (p = 0.51)	non_irony (p = 0.78)
27	How dare Charles Barkley have an intelligent conversation about race. #sarcasm #CharlesBarkley	How dare Charles Barkley have an intelilgent onvsersation about race. #wrcasm #CarlesBarkley	non_irony (p = 0.64)	irony (p = 0.69)

Taxonomy

avid-effect:performance:P0201

Feature `text`

Punctuation Removal

Fail rate = 0.093

59/634 tested samples (9.31%) changed prediction after perturbation

634 samples affected
(80.9% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Punctuation Removal”, the model changes its prediction in 9.31% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Punctuation Removal(text)	Original prediction	Prediction after perturbation
23	Who told the #hipsters that #irony was a thing of the Clinton years? Do they not carry history books in used bookstores in #brooklyn ?	Who told the #hipsters that #irony was a thing of the Clinton years Do they not carry history books in used bookstores in #brooklyn	non_irony (p = 0.65)	irony (p = 0.94)
39	On the train and surrounded by posh people, I'm so at home! #not #stickoutlikeasorethumb	On the train and surrounded by posh people I m so at home #not #stickoutlikeasorethumb	irony (p = 0.66)	non_irony (p = 0.81)
40	Stupid #doctors visits is gonna bury me!! Now that's #irony	Stupid #doctors visits is gonna bury me Now that s #irony	irony (p = 0.55)	non_irony (p = 0.77)

Taxonomy

avid-effect:performance:P0201

Feature `text`

Transform to lowercase

Fail rate = 0.055

39/713 tested samples (5.47%) changed prediction after perturbation

713 samples affected
(90.9% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Transform to lowercase”, the model changes its prediction in 5.47% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Transform to lowercase(text)	Original prediction	Prediction after perturbation
19	@user Guess they didn't get the memo reg non-nuclear Baltic sea #sarcasm	@user guess they didn't get the memo reg non-nuclear baltic sea #sarcasm	non_irony (p = 0.51)	irony (p = 0.60)
22	Pretty excited about how you gave up on me. File Under: #sarcasm	pretty excited about how you gave up on me. file under: #sarcasm	irony (p = 0.51)	non_irony (p = 0.54)
30	Nooooooooooo again it's on!!! #PickANewSong #CantStandIt	nooooooooooo again it's on!!! #pickanewsong #cantstandit	non_irony (p = 0.73)	irony (p = 0.75)

Taxonomy

avid-effect:performance:P0201

We found some data slices in your dataset on which your model performance is lower than average. Performance bias may happen for different reasons:

Not enough examples in the low-performing data slice in the training set
Wrong labels in the training set in the low-performing data slice
Drift between your training set and test set

To learn more about causes and solutions, check our guide on performance bias.

Issues

3 major

`text` contains "user" Recall = 0.366 (Global = 0.563) -34.98% than global 290 samples affected
(37.0% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "user", the Recall is 34.98% lower than the global Recall.

Examples

	text	label	Predicted `label`
19	@user Guess they didn't get the memo reg non-nuclear Baltic sea #sarcasm	irony	non_irony (p = 0.51)
25	@user hmm... let me think about that #sarcasm	irony	non_irony (p = 0.91)
47	@user 180 dead on 26/11 n more than 10k our ppl killed in terror attacks till date but not 1 paki show sympathy 2 them #irony	irony	non_irony (p = 0.71)

Taxonomy

avid-effect:performance:P0204

`text` contains "irony" Accuracy = 0.531 (Global = 0.735) -27.73% than global 113 samples affected
(14.4% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "irony", the Accuracy is 27.73% lower than the global Accuracy.

Examples

	text	label	Predicted `label`
23	Who told the #hipsters that #irony was a thing of the Clinton years? Do they not carry history books in used bookstores in #brooklyn ?	irony	non_irony (p = 0.65)
47	@user 180 dead on 26/11 n more than 10k our ppl killed in terror attacks till date but not 1 paki show sympathy 2 them #irony	irony	non_irony (p = 0.71)
65	#Irony RT @user If you're going to give someone a scathing, 1-Star review for poor grammar, FFS use proper grammar.	irony	non_irony (p = 0.71)

Taxonomy

avid-effect:performance:P0204

`text` contains "sarcasm" Accuracy = 0.645 (Global = 0.735) -12.15% than global 110 samples affected
(14.0% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "sarcasm", the Accuracy is 12.15% lower than the global Accuracy.

Examples

	text	label	Predicted `label`
4	So much #sarcasm at work mate 10/10 #boring 100% #dead mate full on #shit absolutely #sleeping mate can't handle the #sarcasm	irony	non_irony (p = 0.93)
6	People complain about my backround pic and all I feel is like "hey don't blame me, Albert E might have spoken those words" #sarcasm #life	irony	non_irony (p = 0.73)
19	@user Guess they didn't get the memo reg non-nuclear Baltic sea #sarcasm	irony	non_irony (p = 0.51)

Taxonomy

avid-effect:performance:P0204

Debug your issues in the Giskard hub

Install the Giskard hub app to:

Debug and diagnose your scan issues
Save your scan result as a re-executable test suite to benchmark your model
Extend your test suite with our catalog of ready-to-use tests

You can find installation instructions here.

from giskard import GiskardClient

# Create a test suite from your scan results
test_suite = results.generate_test_suite("My first test suite")

# Upload your test suite to your Giskard hub instance
client = GiskardClient("http://localhost:19000", "GISKARD_API_KEY")
client.create_project("my_project_id", "my_project_name")
test_suite.upload(client, "my_project_id")