You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
a set of web/enterprise applications for experimentation in automated system testing.
4
-
5
-
__WARNING__: This repository is going through a major refactoring.
6
-
Most of the documentation is still under construction.
7
-
3
+
a set of web/enterprise applications for scientific research in Software Engineering.
8
4
9
5
We collected several different systems, in different programming languages, like
10
6
Java, Kotlin, JavaScript and C#.
11
-
We also added the drivers for EvoMaster to use those systems.
7
+
In this documentation, we will refer to these projects as System Under Test (SUT).
8
+
Currently, the SUTs are either _REST_ or _GraphQL_ APIs.
12
9
13
-
Note that some of these open-source projects might be no longer supported, whereas others are still developed and updated.
14
-
Once a system is added to EMB, we do not modify nor keep it updated with its current version under development.
15
-
The reason is that we want to keep an easy to use, constant set of case studies for experimentation that can be reliably used throughout the years.
10
+
For each SUT, we implemented _driver_ classes, which can programmatically _start_, _stop_ and _reset_ the state of SUT (e.g., data in SQL databases).
11
+
As well as enable setting up different properties in a _uniform_ way, like choosing TCP port numbers for the HTTP servers.
12
+
If a SUT uses any external services (e.g., a SQL database), these will be automatically started via Docker in these driver classes.
13
+
14
+
15
+
This collection of SUTs was originally assembled for easing experimentation with the fuzzer called [EvoMaster](http://evomaster.org).
16
+
However, finding this type of applications is not trivial among open-source projects.
17
+
Furthermore, it is not simple to sort out all the technical details on how to set these applications up and start them in a simple, uniform approach.
18
+
Therefore, this repository provides the important contribution of providing all these necessary scripts for researchers that need this kind of case study.
16
19
17
20
18
21
## License
19
-
All the code that is new for this repository is released under Apache 2.0 license.
22
+
All the code that is new for this repository (e.g., the driver classes) is released under Apache 2.0 license.
20
23
However, this repository contains as well sources from different open-source
21
24
projects, each one with its own license, as clarified in more details beneath.
22
25
26
+
## Example
23
27
28
+
To see an example of using these drivers with EvoMaster to generate test cases, you can look at this [short video](https://youtu.be/3mYxjgnhLEo) (5 minutes).
29
+
24
30
## Current Case Studies
25
31
32
+
The projects were selected based on searches using keywords on GitHub APIs, using convenience sampling.
33
+
Several SUTs were looked at, in which we discarded the ones that would not compile, would crash at startup, would use obscure/unpopular libraries with no documentation to get them started, are too trivial, student projects, etc.
34
+
Where possible, we tried to prioritize/sort based on number of _stars_ on GitHub.
35
+
36
+
37
+
Note that some of these open-source projects might be no longer supported, whereas others are still developed and updated.
38
+
Once a system is added to EMB, we do not modify nor keep it updated with its current version under development.
39
+
The reason is that we want to keep an easy to use, constant set of case studies for experimentation that can be reliably used throughout the years.
40
+
41
+
The SUTs called _NCS_ (Numerical Case Study) and _SCS_ (String Case study) are artificial, developed by us.
42
+
They are based on numerical and string-based functions previously used in the literature of unit test generation.
43
+
We just re-implemented in different languages, and put them behind a web service.
44
+
45
+
26
46
27
47
### REST: Java/Kotlin
28
48
@@ -38,33 +58,111 @@ projects, each one with its own license, as clarified in more details beneath.
38
58
39
59
* News (LGPL), from [https://github.com/arcuri82/testing_security_development_enterprise_systems](https://github.com/arcuri82/testing_security_development_enterprise_systems)
40
60
41
-
* NCS (not-known license, artificial numerical examples coming from different sources)
42
-
43
-
* SCS (not-known license, artificial string examples coming from different sources)
44
-
45
61
* Restcountries (MPL), from [https://github.com/apilayer/restcountries](https://github.com/apilayer/restcountries)
46
62
47
63
* Languagetool (LGPL), from [https://github.com/languagetool-org/languagetool](https://github.com/languagetool-org/languagetool)
48
64
49
65
* CWA-Verification-Server (Apache), from [https://github.com/corona-warn-app/cwa-verification-server](https://github.com/corona-warn-app/cwa-verification-server)
50
66
67
+
* NCS (not-known license, artificial numerical examples coming from different sources)
68
+
69
+
* SCS (not-known license, artificial string examples coming from different sources)
70
+
71
+
51
72
### REST: JavaScript/TypeScript
52
73
53
74
* Disease-sh-API (GPL), from [https://github.com/disease-sh/API](https://github.com/disease-sh/API)
54
75
55
76
* Cyclotron (MIT), from [https://github.com/ExpediaInceCommercePlatform/cyclotron](https://github.com/ExpediaInceCommercePlatform/cyclotron)
56
77
78
+
* SpaceX-API (Apache-2.0 License), from [https://github.com/r-spacex/SpaceX-API](https://github.com/r-spacex/SpaceX-API)
79
+
80
+
* Realworld-App (ISC), from [https://github.com/lujakob/nestjs-realworld-example-app](https://github.com/lujakob/nestjs-realworld-example-app)
81
+
57
82
* NCS (not-known license, artificial numerical examples coming from different sources)
58
-
83
+
59
84
* SCS (not-known license, artificial string examples coming from different sources)
60
85
61
-
### REST: .Net/C#
86
+
87
+
### REST: .NET/C#
62
88
63
89
* Menu.API (not-known license), from [https://github.com/chayxana/Restaurant-App](https://github.com/chayxana/Restaurant-App)
64
90
65
91
* SampleProject (MIT), from [https://github.com/kgrzybek/sample-dotnet-core-cqrs-api](https://github.com/kgrzybek/sample-dotnet-core-cqrs-api)
66
92
67
-
* Library (not-known license), from [https://github.com/KevinDockx/DocumentingAspNetCoreApisWithOpenAPI](https://github.com/KevinDockx/DocumentingAspNetCoreApisWithOpenAPI)
93
+
* NCS (not-known license, artificial numerical examples coming from different sources)
94
+
95
+
* SCS (not-known license, artificial string examples coming from different sources)
96
+
97
+
98
+
### GraphQL: Java/Kotlin
99
+
100
+
* Spring-Pet-Clinic (Apache ), from [https://github.com/spring-petclinic/spring-petclinic-graphql]()
101
+
102
+
* Patio-Api (GPL), from [https://github.com/patio-team/patio-api]()
103
+
104
+
* Timbuctoo (GPL), from [https://github.com/HuygensING/timbuctoo]()
105
+
106
+
* NCS (not-known license, artificial numerical examples coming from different sources)
107
+
108
+
* SCS (not-known license, artificial string examples coming from different sources)
109
+
110
+
111
+
## Using This Repository
112
+
113
+
Due to several reasons, the software in this repository is not published as a library (e.g., on Maven and NPM).
114
+
To use EMB, you need to clone this repository:
115
+
116
+
```
117
+
git clone https://github.com/EMResearch/EMB.git
118
+
```
119
+
120
+
There are 2 main use cases for EMB:
121
+
122
+
* Run experiments with _EvoMaster_
123
+
124
+
* Run experiments with other tools
125
+
126
+
Everything can be setup by running the script `scripts/dist.py`.
127
+
Note that you will need installed at least JDK 8, JDK 11, NPM and .NET 3.x, as well as Docker.
128
+
Also, you will need to setup environment variables like `JAVA_HOME_8` and `JAVA_HOME_11`.
129
+
The script will issue error messages if any prerequisite is missing.
130
+
Once the script is completed, all the SUTs will be available under the `dist` folder, and a `dist.zip` will be created as well (if `scripts/dist.py` is run with `True` as input).
131
+
132
+
[//]: #(There is also a Docker file to run `dist.py`, named `build.dockerfile`.)
[//]: #(The `dist` folder with all SUTs will be under `/emb/dist`. )
145
+
146
+
147
+
148
+
Note that here the drivers will be built as well besides the SUTs, and the SUT themselves will also have an instrumented version (for white-box testing heuristics) for _EvoMaster_ (this is for JavaScript and .NET, whereas instrumentation for JVM is done at runtime, via an attached JavaAgent).
149
+
150
+
In the built `dist` folder, the files will be organized as follows:
151
+
152
+
* For JVM: `<name>-sut.jar` will be the non-instrumented SUTs, whereas their executable drivers will be called `<name>-evomaster-runner.jar`.
153
+
Instrumentation can be done at runtime by attaching the `evomaster-agent.jar` JavaAgent. If you are running experiments with EvoMaster, this will be automatically attached when running experiments with `exp.py` (available in the EvoMaster's repository). Or it can be attached manually with JVM option `-Devomaster.instrumentation.jar.path=evomaster-agent.jar` when starting the driver.
154
+
* For NodeJS: under the folder `<name>` (for each NodeJS SUT), the SUT is available under `src`, whereas the instrumented version is under `build`.
155
+
* For .NET: currently only the instrumented version is available (WORK IN PROGRESS)
156
+
157
+
Note: the building of .NET SUTs/drivers is disabled by default in `dist.py`.
158
+
The reason is that, at the time of this writing, it is _very_ expensive to publish libraries on NuGet (due to the high costs of code certificates).
159
+
The EM Driver library for .NET would have to be installed manually on local machine before any of the C# drivers can be used.
160
+
161
+
162
+
163
+
For running experiments with EvoMaster, you can also "start" each driver directly from an IDE (e.g., IntelliJ).
164
+
Each of these drivers has a "main" method that is running a REST API (binding on default port 40100), where each operation (like start/stop/reset the SUT) can be called via an HTTP message by EvoMaster.
165
+
For JavaScript, you need to use the files `em-main.js`.
68
166
69
167
### RPC (Thrift/gRPC): Java
70
168
@@ -73,43 +171,37 @@ projects, each one with its own license, as clarified in more details beneath.
73
171
* SCS (not-known license, artificial string examples coming from different sources)
74
172
75
173
76
-
## Build The Systems
174
+
You can also build (and install) each module separately, based on needs.
175
+
For example, a Maven module can be installed with:
77
176
78
-
### Build JDK_8_MAVEN
177
+
``mvn clean install -DskipTests``
79
178
80
-
The folder `cs` (*case study*) contains the source code of the different
81
-
system under tests (SUT) in this benchmark, for JDK 8 and Maven.
179
+
However, it is important to understand how this repository is structured, to be able to effectively navigate through it.
180
+
Each folder represents a set of SUTs (and drivers) that can be built using the same tools.
181
+
For example, the folder `jdk_8_maven` contains all the SUTs that need JDK 8 and are built with Maven.
182
+
On the other hand, the SUTs in the folder `jdk_11_gradle` require JDK 11 and Gradle.
82
183
83
-
The folder `em` (*EvoMaster*) contains the classes needed to be written to enable
84
-
the use of EvoMaster on the SUTs.
85
-
In particular, there are `EmbeddedEvoMasterController` and
86
-
`ExternalEvoMasterController` class implementations for each SUT.
87
-
Note: usually you would write a EvoMaster controller class in the same module
88
-
of the SUTs.
89
-
Here, they are in different modules just to make clear what is needed to implement
90
-
to enable the use of EvoMaster.
184
+
For JVM and .NET, each module has 2 submodules, called `cs` (short for "Case Study") and `em` (short for "EvoMaster").
185
+
`cs` contains all the source code of the different SUTs, whereas `em` contains all the drivers.
186
+
Note: building a top-module will build as well all of its internal submodules.
91
187
188
+
Regarding JavaScript, unfortunately NodeJS does not have a good handling of multi-module projects.
189
+
Each SUT has to be built separately.
190
+
However, for each SUT, we put its source code under a folder called `src`, whereas all the code related to the drivers is under `em`.
92
191
93
-
To compile and generate all the jar files, use the command:
192
+
The driver classes for Java and .NET are called `EmbeddedEvoMasterController`.
193
+
For JavaScript, they are in a script file called `app-driver.js`.
194
+
Note that Java also a different kind of driver called `ExternalEvoMasterController`.
195
+
The difference is that in External the SUT is started on a separated process, and not running in the same JVM of the driver itself.
94
196
95
-
``mvn clean package -DskipTests``
96
197
97
-
Currently, all the case studies do require JDK __8__.
98
-
They will not compile with a different version.
99
198
100
-
_Note_: the case studies do import EvoMaster as a library. Current SNAPSHOT
101
-
versions of the case studies do use the most recent SNAPSHOT version of EvoMaster
102
-
(the two versioning numbers are aligned).
103
-
We do __NOT__ publish the SNAPSHOT dependencies online.
104
-
This means that, if you try to build the project directly, it will fail due to
105
-
missing SNAPSHOT dependencies.
199
+
## Old Versions
106
200
107
-
To use such SNAPSHOT versions, you need first a `mvn install` of EvoMaster on your
108
-
machine (so that the SNAPSHOT jars are created, and put under your `~/.m2` folder).
109
-
However, in the Git repository of EMB, we did tag the versions of EMB that are
110
-
using the published versions of EvoMaster.
201
+
The release of EMB are linked in version number with the release of EvoMaster, as EvoMaster's libraries are used in the drivers (e.g., to clean databases and configure auth info).
202
+
In the Git repository of EMB, we did tag the versions of EMB.
111
203
See the [releases](https://github.com/EMResearch/EMB/releases) page.
112
-
For example, to use version `X` of EvoMaster, you can check out the Git commit
204
+
For example, to use version `X`, you can check out the Git commit
113
205
of EMB tagged with version `X`.
114
206
To see the current available tags, from a command-line you can execute:
115
207
@@ -136,15 +228,20 @@ the following plugin dependency version:
136
228
</plugin>
137
229
```
138
230
139
-
Besides JDK 8, to build from Maven you will also need NPM and NodeJS installed
140
-
on your machine (as some of the projects have GUIs built with JS).
141
231
232
+
### Build *develop* Branch
142
233
143
-
### Build DOTNET_3
234
+
Branch *develop* is using the most recent SNAPSHOT version of _EvoMaster_.
235
+
As that is not published online, you need to clone its repository, and build
236
+
it locally (see its documentation on how to do it).
144
237
145
-
*Documentation under construction*
238
+
To handle JavaScript, unfortunately there is the need for some manual settings.
239
+
However, it needs to be done just once.
146
240
241
+
You need to create _symbolic_ link inside `EMB\js_npm` that points to the `evomaster-client-js` folder in _EvoMaster_.
242
+
How to do this, depends on the Operating System.
243
+
Note that in the following, `<some-path>` should be replaced with the actual real paths of where you cloned the _EvoMaster_ and _EMB_ repositories.
0 commit comments