Skip to content

Add --fuzzing-ratio to CLI #2496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Markoutte wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add --fuzzing-ratio to CLI #2496

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2a79a7d
Improve CLI:
Markoutte Aug 10, 2023
57555e4
We can ignore --source option
Markoutte Aug 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Improve CLI: 
1. Fix error with classloader
2. Add an option to control fuzzer ration
  • Loading branch information
@Markoutte
Markoutte committed Aug 10, 2023
commit 2a79a7d41317da64a1d88fed52514a06d773a808
28 changes: 18 additions & 10 deletions utbot-cli/src/main/kotlin/org/utbot/cli/GenerateTestsAbstractCommand.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import com.github.ajalt.clikt.parameters.options.multiple
import com.github.ajalt.clikt.parameters.options.option
import com.github.ajalt.clikt.parameters.options.unique
import com.github.ajalt.clikt.parameters.types.choice
import com.github.ajalt.clikt.parameters.types.double
import com.github.ajalt.clikt.parameters.types.long
import mu.KotlinLogging
import org.utbot.common.PathUtil.classFqnToPath
Expand All @@ -25,14 +26,7 @@ import org.utbot.framework.codegen.domain.testFrameworkByName
import org.utbot.framework.codegen.generator.CodeGenerator
import org.utbot.framework.codegen.generator.CodeGeneratorParams
import org.utbot.framework.codegen.services.language.CgLanguageAssistant
import org.utbot.framework.plugin.api.ClassId
import org.utbot.framework.plugin.api.CodegenLanguage
import org.utbot.framework.plugin.api.ExecutableId
import org.utbot.framework.plugin.api.MethodId
import org.utbot.framework.plugin.api.MockStrategyApi
import org.utbot.framework.plugin.api.TestCaseGenerator
import org.utbot.framework.plugin.api.TreatOverflowAsError
import org.utbot.framework.plugin.api.UtMethodTestSet
import org.utbot.framework.plugin.api.*
import org.utbot.framework.plugin.services.JdkInfoDefaultProvider
import org.utbot.summary.summarizeAll
import java.io.File
Expand Down Expand Up @@ -131,14 +125,20 @@ abstract class GenerateTestsAbstractCommand(name: String, help: String) :
.long()
.default(LONG_GENERATION_TIMEOUT)

private val fuzzingRation by option(

@ghost ghost Feb 26, 2024

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: "ration" seems misused here. Consider calling this property fuzzingRatio instead of fuzzingRation.

"--fuzzing-ratio",
help = "Specify the ratio between symbolic engine and fuzzing"
)
.double()
Comment on lines +128 to +132

@ghost ghost Feb 26, 2024 •
edited by ghost
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to this comment from @tyuldashev, the default is 0.1 fuzzing, 0.9 symbolic. If we take the help text literally, then the default "ratio between symbolic engine and fuzzing" would be 0.9 ÷ 0.1 = 9. But from my understanding of the code, 9 is not a valid value here, and this argument should actually be 0.1 to mimic the default behavior.

Consider revising this option's name and help text to reflect its actual semantics and to document its default behavior. For example:

private val fuzzingFraction by option(
        "-z",
        "--fuzzing-fraction",
        help = "Specifies the fraction of time spent using fuzzing instead of the symbolic engine (default: $DEFAULT_FUZZING_VALUE)"
)
        .double()
        .default(DEFAULT_FUZZING_VALUE)
        .validate { require(it in 0.0 .. 1.0) { "should be between 0.0 and 1.0." } }

where DEFAULT_FUZZING_VALUE is the default value of org.utbot.framework.plugin.api.TestFlow.fuzzingValue: currently 0.1.


protected open val classLoader: URLClassLoader by lazy {
val urls = classPath!!
.split(File.pathSeparator)
.map { uri ->
uri.toPath().toURL()
}
.toTypedArray()
URLClassLoader(urls)
URLClassLoader(urls, null)
}

abstract override fun run()
Expand All @@ -162,7 +162,15 @@ abstract class GenerateTestsAbstractCommand(name: String, help: String) :
targetMethods,
mockStrategy,
chosenClassesToMockAlways,
generationTimeout
generationTimeout,
generate = fuzzingRation?.let { rat ->
testFlow {
generationTimeout = this@GenerateTestsAbstractCommand.generationTimeout
isSymbolicEngineEnabled = rat < 1.0
isFuzzingEnabled = rat > 0.0
fuzzingValue = rat
}
} ?: defaultTestFlow(generationTimeout)
).let {
if (sourceCodeFile != null) it.summarizeAll(searchDirectory, sourceCodeFile.toFile()) else it
}
Expand Down