Giskard Scan Results

8 issues detected

Performance 7

Robustness 1

We found some data slices in your dataset on which your model performance is lower than average. Performance bias may happen for different reasons:

Not enough examples in the low-performing data slice in the training set
Wrong labels in the training set in the low-performing data slice
Drift between your training set and test set

To learn more about causes and solutions, check our guide on performance bias.

Issues

2 major 5 medium

`text` contains "said" Precision = 0.737 (Global = 0.917) -19.67% than global 205 samples affected
(9.1% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "said", the Precision is 19.67% lower than the global Precision.

Examples

	text	label	Predicted `label`
115	Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. .	Positive	Neutral (p = 1.00)
129	However , sales returned to growth in April-June 2010 , CEO Pekka Eloholma said .	Positive	Neutral (p = 0.46)
312	Finnish telecoms software developer Tecnomen Oyj said on December 5 , 2006 it won a $ 3.3 mln ( 2.5 mln euro ) order to supply an expansion of the charging and messaging systems for the mobile and fixed networks of Brasil Telecom .	Positive	Neutral (p = 0.98)

Taxonomy

avid-effect:performance:P0204

`text` contains "period" Precision = 0.821 (Global = 0.917) -10.49% than global 173 samples affected
(7.6% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "period", the Precision is 10.49% lower than the global Precision.

Examples

	text	label	Predicted `label`
47	Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 .	Positive	Neutral (p = 0.74)
329	26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier .	Positive	Negative (p = 0.59)
330	8 May 2009 - Finnish liquid handling products and diagnostic test systems maker Biohit Oyj ( HEL : BIOBV ) said today ( 8 May 2009 ) its net loss narrowed to EUR0 .1 m ( USD0 .14 m ) for the first quarter of 2009 from EUR0 .4 m for the same period of 2008 .	Positive	Negative (p = 0.63)

Taxonomy

avid-effect:performance:P0204

`text` contains "finland" Balanced Accuracy = 0.810 (Global = 0.886) -8.58% than global 133 samples affected
(5.9% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "finland", the Balanced Accuracy is 8.58% lower than the global Balanced Accuracy.

Examples

	text	label	Predicted `label`
115	Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. .	Positive	Neutral (p = 1.00)
324	In addition , Cramo and Peab have signed exclusive five-year rental agreements in Finland and have extended their existing rental agreements in the Swedish market for another five years .	Positive	Neutral (p = 1.00)
425	Citigroup , Inc ( NYSE : C ) has announced that its Global Transaction Services ( GTS ) business has been awarded a new mandate by Finland-based Pohjola Bank Group .	Positive	Neutral (p = 0.75)

Taxonomy

avid-effect:performance:P0204

`text` contains "year" Precision = 0.862 (Global = 0.917) -6.02% than global 217 samples affected
(9.6% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "year", the Precision is 6.02% lower than the global Precision.

Examples

	text	label	Predicted `label`
230	Last year the company raised its turnover to approximately 7 million litas EUR 2 mln , from 6.1 million litas in 2004 .	Positive	Neutral (p = 0.63)
321	LONDON MarketWatch -- Nokia nok said it 's won a five-year services contract to run Hutchison Essar 's network operations in nine locations in India .	Positive	Neutral (p = 1.00)
324	In addition , Cramo and Peab have signed exclusive five-year rental agreements in Finland and have extended their existing rental agreements in the Swedish market for another five years .	Positive	Neutral (p = 1.00)

Taxonomy

avid-effect:performance:P0204

`text` contains "eur" Precision = 0.865 (Global = 0.917) -5.68% than global 444 samples affected
(19.6% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "eur", the Precision is 5.68% lower than the global Precision.

Examples

	text	label	Predicted `label`
21	The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million .	Positive	Neutral (p = 1.00)
47	Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 .	Positive	Neutral (p = 0.74)
200	The fair value of the investment properties totaled EUR 2,534.9 mn , up from EUR 2,455.1 mn in 2006 .	Positive	Neutral (p = 1.00)

Taxonomy

avid-effect:performance:P0204

`text` contains "quarter" Precision = 0.866 (Global = 0.917) -5.57% than global 164 samples affected
(7.2% of dataset) Show details Hide details

Description

For records in the dataset where `text` contains "quarter", the Precision is 5.57% lower than the global Precision.

Examples

	text	label	Predicted `label`
328	In the second quarter of 2010 , the group 's pretax loss narrowed to EUR 400,000 from EUR 600,000 .	Positive	Negative (p = 0.98)
329	26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier .	Positive	Negative (p = 0.59)
330	8 May 2009 - Finnish liquid handling products and diagnostic test systems maker Biohit Oyj ( HEL : BIOBV ) said today ( 8 May 2009 ) its net loss narrowed to EUR0 .1 m ( USD0 .14 m ) for the first quarter of 2009 from EUR0 .4 m for the same period of 2008 .	Positive	Negative (p = 0.63)

Taxonomy

avid-effect:performance:P0204

`text_length(text)` >= 122.500 AND `text_length(text)` < 133.500 Balanced Accuracy = 0.839 (Global = 0.886) -5.28% than global 148 samples affected
(6.5% of dataset) Show details Hide details

Description

For records in the dataset where `text_length(text)` >= 122.500 AND `text_length(text)` < 133.500, the Balanced Accuracy is 5.28% lower than the global Balanced Accuracy.

Examples

	text	text_length(text)	label	Predicted `label`
10	Its board of directors will propose a dividend of EUR0 .12 per share for 2010 , up from the EUR0 .08 per share paid in 2009 .	125	Positive	Neutral (p = 1.00)
21	The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million .	132	Positive	Neutral (p = 1.00)
319	Bilfinger investors cheered the agreement , pushing shares up 7 % , or & euro ; 3.30 , to & euro ; 50.29 , in afternoon trade .	127	Positive	Neutral (p = 1.00)

Taxonomy

avid-effect:performance:P0204

Your model seems to be sensitive to small perturbations in the input data. These perturbations can include adding typos, changing word order, or turning text into uppercase or lowercase. This happens when:

There is not enough diversity in the training data
Overreliance on spurious correlations like the presence of specific word
Use of complex models with large number of parameters that tend to overfit the training data

To learn more about causes and solutions, check our guide on robustness issues.

Issues

1 medium

Feature `text`

Add typos

Fail rate = 0.091

91/1000 tested samples (9.1%) changed prediction after perturbation

1000 samples affected
(44.2% of dataset)

Show details

Description

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 9.1% of the cases. We expected the predictions not to be affected by this transformation.

Examples

	text	Add typos(text)	Original prediction	Prediction after perturbation
1143	A huge issue for us is the button placement .	A huge isue for us is the button placment .	Negative (p = 1.00)	Neutral (p = 0.99)
2217	Talentum 's net sales in September were smaller than expected .	Talentum ' net sales in September were skaler tham expeted .	Negative (p = 1.00)	Neutral (p = 1.00)
956	Elcoteq has a global network of After Market Service sites which have a long experience in serving Consumer Electronics and Systems Solutions customers .	Elcoteq has a globa network of After Market Service sites which have a long experience in serving Consumer Electrinixs and Systems Solutions ustomers .	Positive (p = 0.55)	Neutral (p = 0.97)

Taxonomy

avid-effect:performance:P0201

Debug your issues in the Giskard hub

Install the Giskard hub app to:

Debug and diagnose your scan issues
Save your scan result as a re-executable test suite to benchmark your model
Extend your test suite with our catalog of ready-to-use tests

You can find installation instructions here.

from giskard import GiskardClient

# Create a test suite from your scan results
test_suite = results.generate_test_suite("My first test suite")

# Upload your test suite to your Giskard hub instance
client = GiskardClient("http://localhost:19000", "GISKARD_API_KEY")
client.create_project("my_project_id", "my_project_name")
test_suite.upload(client, "my_project_id")