ABCDEFGHIJKLMNOPQRSTUVWXYZAA
1
StatusAreaItemDescriptionHow to Test
2
engine.jsonNameEngine name must be presentDoes the name match the repository name?
3
engine.jsonDescriptionEngine description must be presentIs the description tweet-sized and in line with those at https://codeclimate.com/engines
4
engine.jsonMaintainerMaintainer or organization's name must be presentIs the name present?
5
engine.jsonLanguagesLanuage/framework names that engine analyzes must be presentIs the language field present and accurate?
6
engine.jsonVersionEngine version must be presentIs the version present? Version can be an arbitrary string
7
DockerfileMAINTAINERDockerfile maintainer must be presentDoes it match the engine.json maintiner field?
8
DockerfileVOLUMEEngine must mount /code as a volumeIs the directive called in the Dockerfile?
9
DockerfileWORKDIRThe WORKDIR must be set to /code before the CMD is runOther WORKDIR calls are fine as long as /code is set immediately before CMD
10
DockerfileUSERThe user invoking the engine must have UID/GID 9000Is the user created and used in the Dockerfile?
11
DockerfileCMDThe Dockerfile must provide a CMD to run the engineIs the CMD call last in the Dockerfile?
12
DockerfileNo EXPOSEEXPOSE directives are not valid according to the Code Climate SpecificationAre there any EXPOSE directives?
13
DockerfileNo ONBUILDONBUILD directives are not valid according to the Code Climate SpecificationAre there any ONBUILD directives?
14
DockerfileNo ARGARG directives are not valid according to the Code Climate SpecificationAre there any ARG directives?
15
Docker ImageImage sizeBuilt Docker image size must be less than 512MBCheck results of running `docker images` after image is built locally
16
CLI BehaviorFile ScopeScope analysis to source files of target language/frameworkDoes the engine scope file walking as strictly as possible?
17
CLI Behaviorinclude_paths
Use `include_paths` from config.json as the "workspace" of files and paths to analyze
Run codeclimate analyze FILENAME to test
18
CLI Behaviorexclude_pathsDo not analyze files listed in exclude paths in a user's .codeclimate.ymlAdd file to exclude_paths in .codeclimate.yml, run codeclimate analyze
19
CLI BehaviorTool config fileSupport tool's native configuration, e.g. .rubocop.yml, .eslintrc, etc.Run engine, add config file, compare differences
20
CLI BehaviorCustom config functionalityCustom .config.yml configuration should function properlyAdd custom config to .codeclimate.yml, run analyze, compare results
21
CLI BehaviorCustom config documentationCustom .config.yml configuration should be documentedAdd custom config information to README and documentation for docs.codeclimate.com
22
CLI BehaviorEmpty project functionalityDo not throw errors if there are no files to analyzeRun against an empty repo or a repo with no valid files present
23
CLI BehaviorError handlingThe engine should handle all errors raised by the toolRun against 10 repos and make sure there are no obvious errors, examine code for error handling
24
CLI Behavior10 Repos TestThe engine should perform reasonably on repos of all sizesRun against 10 repos and make sure none time out, and small repos run in reasonable time
25
Engine OutputDescriptionClearly describes issue being presented.Do all issues have descriptions? Code constants should also be contained in backticks ``
26
Engine OutputContentInformation intended to educate the user on the nature of the issueContent should be formatted as Markdown. Optional for Style issues.
27
Engine OutputLocationRender all relevant source linese.g. complexity of a function should display entire function
28
Engine OutputCategoriesThe category of the issueIs the category for each issue one of the choices? Multiple categories currently not handled. Category can be one of Bug Risk, Clarity, Compatibility, Complexity, Duplication, Performance, Security, Style
29
Engine OutputRemediation PointsThe amount of effort necessary to resolve reported issue (_not_ severity of the issue).Does every issue type have remediation points? Style issues generally should be awarded the smallest values (50k / 5 mins). 10,000 remediation points is approximately 1 minute.
30
Engine OutputThresholdsIf applicable, quantitative thresholds for triggering an issue to occur are properly tunedAny issue triggered by a numberical threshold should ship with a sensible default
31
.com Behavior10 Repos TestTest 10 repositories for speed, functionality, and gradingThe largest, most complex repos in a given language/framework can sometimes be problematic without proper excludes
32
.com BehaviorIssue description UIDescriptions should look good and not be too longIn the UI, do the descriptions take up more than one line?
33
.com BehaviorIssue description backticks ``Anything referring to specific code should render properly in HTMLClass, method, etc. constant names should be styled as code
34
.com BehaviorRead up formattingReadups have properly formatted and spaced Markdown that look goodIt is better to have no readup than a bad readup
35
.com BehaviorLocation renderingIssue locations cover code properly in code viewIn a project, click a class and look in the code view to make sure issues cover proper lines
36
.com BehaviorGrades and GPAsThe grades and GPAs produced by Code Climate make intuitive sense
Look at the repos from the 10 repos test. Files which have a lot of issues that are medium-hard to fix should not be As. You should be able to see variation in grade results. Some A, B, C, D, and F files as expected. GPAs impacted when expected.
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100