Remove TensorFlow Serving (TFS) gRPC/REST frontend by atobiszei · Pull Request #4349 · openvinotoolkit/model_server

atobiszei · 2026-07-01T12:27:50Z

Delete all TFS-specific source files: tfs_frontend/*, prediction_service,
model_service, get_model_metadata_impl, dag_tfs
Remove TFS REST parser (TFSRestParser) and associated utilities
Update grpcservermodule, server, http_rest_api_handler to KFS/CAPI only
Move tensorflow_type_utils from tfs_frontend (still needed by
kfs_graph_executor_impl for MediaPipe type conversion)

New file: config_status_utils.[ch]pp for Config Reload API support
without TFS API.

Tensorflow type utils are needed for Mediapipe TF types support in calculators.

Remove optional drogon server build flags, as there's no coming back to net_http from TFS.

Ensemble mapping tests were removed as they would require rewrite to KFS and DAGS will be removed soon anyway,

JIRA:CVS-190152

Copilot

Pull request overview

This PR removes the TensorFlow Serving (TFS) gRPC/REST frontend from OVMS, simplifying the server to KFS + C-API only, while preserving TensorFlow datatype conversion utilities needed for MediaPipe calculators and adding a replacement implementation for Config Reload/Status APIs.

Changes:

Removed TFS frontend implementation (gRPC PredictionService/ModelService, TFS REST parser, net_http pieces) and Bazel dependency on tensorflow_serving.
Introduced config_status_utils.[ch]pp to support Config Reload/Status APIs without relying on TFS protos.
Updated build/test codepaths to use KFS/C-API equivalents and moved TF datatype mapping to tensorflow_type_utils.[ch]pp.

Reviewed changes

Copilot reviewed 72 out of 73 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
WORKSPACE	Drops `tensorflow_serving` external repo and workspace init.
src/tfs_frontend/validation.cpp	Deleted TFS request validation implementation.
src/tfs_frontend/tfs_utils.hpp	Deleted TFS utility header.
src/tfs_frontend/tfs_utils.cpp	Deleted TFS utility implementation.
src/tfs_frontend/tfs_request_utils.hpp	Deleted TFS request extraction helpers.
src/tfs_frontend/tfs_request_utils.cpp	Deleted TFS request extraction helpers impl.
src/tfs_frontend/tfs_backend_impl.cpp	Deleted TFS backend infer glue.
src/tfs_frontend/serialization.hpp	Deleted TFS serialization API.
src/tfs_frontend/serialization.cpp	Deleted TFS serialization implementation.
src/tfs_frontend/deserialization.hpp	Deleted TFS deserialization API.
src/tfs_frontend/deserialization.cpp	Deleted TFS deserialization implementation.
src/test/tfs_rest_parser_nonamed_test.cpp	Removed TFS REST parser tests.
src/test/tfs_rest_parser_binary_inputs_test.cpp	Removed TFS REST binary-input tests.
src/test/text2image_test.cpp	Adds logging include to support SPDLOG usage in tests.
src/test/test_utils.hpp	Removes TFS test helpers/types; keeps KFS/CAPI helpers.
src/test/test_utils.cpp	Removes TFS request/response helpers; keeps KFS/CAPI equivalents.
src/test/test_models.hpp	Adds missing include for `Shape` usage.
src/test/tensor_conversion_test.cpp	Drops TFS tensor types from typed tests; keeps KFS coverage.
src/test/stress_test_utils.hpp	Switches TF datatype helpers to `tensorflow_type_utils`, removes TFS status/metadata loops.
src/test/serialization_tests.cpp	Removes TFS serialization test coverage; keeps KFS/CAPI.
src/test/pythonnode_test.cpp	Replaces TFS utils include with `tensorflow_type_utils`.
src/test/modelversionstatus_test.cpp	Removes dependency on TF enum casting; keeps OVMS state transition checks.
src/test/modelinstance_test.cpp	Reworks unload/guard test to use unload guard instead of TFS metadata build.
src/test/metrics_flow_test.cpp	Removes TFS gRPC/REST metric flows; keeps KFS flows.
src/test/mediapipeflow_test.cpp	Replaces TFS utils include with `tensorflow_type_utils`.
src/test/mediapipe_framework_test.cpp	Replaces TFS utils include with `tensorflow_type_utils`; adjusts includes.
src/test/kfs_rest_test.cpp	Updates REST path expectations to v2 KFS routes.
src/test/get_pipeline_metadata_response_test.cpp	Removes pipeline metadata response tests based on TFS protos.
src/test/get_model_metadata_validation_test.cpp	Removes TFS GetModelMetadata validation tests.
src/test/get_model_metadata_signature_test.cpp	Removes TFS signature conversion tests.
src/test/get_model_metadata_response_test.cpp	Removes TFS metadata response tests.
src/test/get_mediapipe_graph_metadata_response_test.cpp	Removes negative TFS metadata/status test.
src/test/deserialization_tests.cpp	Removes TFS deserialization tests; keeps KFS/CAPI.
src/test/c_api_stress_tests.cpp	Removes TFS status/metadata dependencies from C-API stress tests.
src/tensorflow_type_utils.hpp	Refactors TF datatype mapping declarations into standalone header.
src/tensorflow_type_utils.cpp	New implementation for TF datatype ↔ OVMS precision mapping.
src/server.cpp	Removes TFS gRPC service registrations/includes.
src/servable_name_checker.hpp	Extends interface to expose servable definitions and names.
src/rest_utils.hpp	Removes TFS response JSON utility surface; keeps KFS response JSON.
src/rest_utils.cpp	Removes TF Serving JSON tensor conversion path; keeps KFS response serialization.
src/rest_parser.hpp	Removes `TFSRestParser` and associated enums/types; keeps KFS parser.
src/prediction_service.hpp	Deleted TFS gRPC PredictionService implementation.
src/prediction_service.cpp	Deleted TFS gRPC PredictionService implementation.
src/net_http_async_writer_impl.hpp	Deleted net_http async writer implementation.
src/net_http_async_writer_impl.cpp	Deleted net_http async writer implementation.
src/modelmanager.hpp	Marks `findServableDefinition` / `getServableDefinitionNames` as overrides.
src/model_service.hpp	Deleted TFS ModelService/GetModelStatus implementation.
src/model_service.cpp	Deleted TFS ModelService/GetModelStatus implementation.
src/kfs_frontend/kfs_graph_executor_impl.cpp	Switches TF datatype helpers include to `tensorflow_type_utils`.
src/http_rest_api_handler.hpp	Removes TFS REST endpoint types; keeps config + KFS + v3 + metrics.
src/grpcservermodule.hpp	Removes TFS services from gRPC server module.
src/grpcservermodule.cpp	Removes TFS service registration/ownership.
src/get_model_metadata_impl.hpp	Deleted TFS GetModelMetadata implementation.
src/get_model_metadata_impl.cpp	Deleted TFS GetModelMetadata implementation.
src/dags/dag_tfs.cpp	Deleted TFS DAG instantiation TU.
src/config_status_utils.hpp	New shared helpers for config status/reload model status output.
src/config_status_utils.cpp	Implements model status collection + JSON serialization for config APIs.
src/capi_frontend/capi.cpp	Removes unused includes tied to removed TFS services.
src/BUILD	Removes TFS targets/deps; adds `tensorflow_type_utils` + `config_status_utils`; makes drogon unconditional.
common_settings.bzl	Removes drogon/net_http select flags and related COPTS.
BUILD.bazel	Removes TFS proto/json_tensor deps from root dependency bundle.
.bazelrc	Removes USE_DROGON define toggle.

- Delete all TFS-specific source files: tfs_frontend/*, prediction_service, model_service, get_model_metadata_impl, dag_tfs - Remove TFS REST parser (TFSRestParser) and associated utilities - Update grpcservermodule, server, http_rest_api_handler to KFS/CAPI only - Move tensorflow_type_utils from tfs_frontend (still needed by kfs_graph_executor_impl for MediaPipe type conversion) New file: config_status_utils.[ch]pp for Config Reload API support without TFS API. JIRA:CVS-190152

- Delete net_http_async_writer_impl.hpp/.cpp (dead code; httpservermodule.cpp always calls createAndStartDrogonHttpServer, comment in http_server.cpp: 'no going back to net_http') - Remove libnet_http_server Bazel target from src/BUILD - Replace enable_drogon select in ovms_lib deps with unconditional Drogon deps - Remove @tensorflow_serving//:__subpackages__ from cc_shared_library - Remove enable_net_http config_setting from common_settings.bzl - Remove stale TFS protos (prediction_service_cc_proto, model_service_cc_proto, json_tensor) from ovms_dependencies in BUILD.bazel - Remove @tensorflow_serving git_repository and tf_serving_workspace() from WORKSPACE — no remaining usages after above cleanup

- net_http is gone; Drogon is the only HTTP server implementation - Remove enable_drogon config_setting and USE_DROGON define from common_settings.bzl - Remove COPTS_DROGON (-DUSE_DROGON=0/1) — flag unused in any source file - Remove --define=USE_DROGON=1 from .bazelrc - Make multi_part_parser_drogon_test.cpp unconditional in ovms_test srcs

dkalinowski · 2026-07-03T08:32:32Z

-    remote = "https://github.com/tensorflow/serving.git",
-    tag = "2.18.0",
-    patch_args = ["-p1"],
-    patches = ["net_http.patch", "listen.patch", "partial_2.18.patch"]


remove the patches from the repo as well

dkalinowski · 2026-07-03T08:41:10Z

In src/execution_context.hpp there are TensorflowServing API defines which then correlate to metrics. I think we need it removed, together with metrics support for TFS endpoints. I see in model_metric_reporter.cpp that we still support those.

in src/predict_request_validation_utils.hpp I dont think we need to forward declare TFS types and assert like that:

    if ((std::is_same<RequestType, ::inference::ModelInferRequest>::value || std::is_same<RequestType, ::tensorflow::serving::PredictRequest>::value) && choice == ValidationChoice::OUTPUT) {
        return StatusCode::NOT_IMPLEMENTED;
    }

ensemble_flow_custom_node_tests.cpp tests use tensorflow::serving::PredictRequest and tensorflow::serving::PredictResponse - can we remove those tests? the same goes for gather_node_test.cpp

in pipelinedefinitionstatus_test.cpp this should no longer be needed:

using namespace tensorflow;
using namespace tensorflow::serving;

mzegla

Do we have validation team confirmation that this change does not introduce any regression?

mzegla · 2026-07-03T13:04:40Z

+    for (const auto& servableName : servableNames) {
+        std::vector<ModelVersionStatusDetails> versions;
+
+        auto model_ptr = modelProvider.findModelByName(servableName);


Suggested change

auto model_ptr = modelProvider.findModelByName(servableName);

auto modelPtr = modelProvider.findModelByName(servableName);

mzegla · 2026-07-03T13:04:53Z

+
+        auto model_ptr = modelProvider.findModelByName(servableName);
+        if (model_ptr) {
+            auto modelVersionsInstances = model_ptr->getModelVersionsMapCopy();


Suggested change

auto modelVersionsInstances = model_ptr->getModelVersionsMapCopy();

auto modelVersionsInstances = modelPtr->getModelVersionsMapCopy();

mzegla · 2026-07-03T13:05:06Z

+        std::vector<ModelVersionStatusDetails> versions;
+
+        auto model_ptr = modelProvider.findModelByName(servableName);
+        if (model_ptr) {


Suggested change

if (model_ptr) {

if (modelPtr) {

mzegla · 2026-07-03T13:05:37Z

+                continue;
+            }
+            INCREMENT_IF_ENABLED(svsd->getMetricReporter().getGetModelStatusRequestSuccessMetric(context));
+            auto [state, error_code] = svsd->getStatus().convertToModelStatus();


Suggested change

auto [state, error_code] = svsd->getStatus().convertToModelStatus();

auto [state, errorCode] = svsd->getStatus().convertToModelStatus();

mzegla · 2026-07-03T13:05:46Z

+            ModelVersionStatusDetails details{
+                svsd->getVersion(),
+                state,
+                error_code,


Suggested change

error_code,

errorCode,

mzegla · 2026-07-03T13:05:54Z

+                svsd->getVersion(),
+                state,
+                error_code,
+                ModelVersionStatusErrorCodeToString(error_code)};


Suggested change

ModelVersionStatusErrorCodeToString(error_code)};

ModelVersionStatusErrorCodeToString(errorCode)};

mzegla · 2026-07-03T13:57:30Z

-    GetModelStatus,
-    GetModelMetadata,
-    ConfigReload,
+// note since removal of TFS V3 endpoints (from OpenAI are now also accepted as V1)


Suggested change

// note since removal of TFS V3 endpoints (from OpenAI are now also accepted as V1)

// note since removal of TFS, V3 endpoints (from OpenAI) are now also accepted as V1

mzegla · 2026-07-03T14:04:00Z

-#include "../logging.hpp"
-#include "../profiler.hpp"
-#include "../status.hpp"
+using TFSDataType = tensorflow::DataType;  // TODO @atobiszei since we dont have TFS now we can rename to TFDataType?


What about this TODO? Should it be merged and addressed later or is it a leftover?

atobiszei requested a review from Copilot July 1, 2026 12:33

Copilot started reviewing on behalf of atobiszei July 1, 2026 12:34 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread src/config_status_utils.cpp Outdated

Comment thread src/config_status_utils.cpp

Comment thread src/test/modelinstance_test.cpp Outdated

Comment thread src/http_rest_api_handler.hpp Outdated

atobiszei added 4 commits July 2, 2026 09:55

Review fixes

c49ad29

atobiszei force-pushed the atobisze_remove_tfs branch from bc27bc8 to c49ad29 Compare July 2, 2026 10:56

atobiszei marked this pull request as ready for review July 2, 2026 11:08

atobiszei requested review from dkalinowski and mzegla July 2, 2026 12:10

dkalinowski reviewed Jul 3, 2026

View reviewed changes

Demo & docs updates

17955bb

mzegla reviewed Jul 3, 2026

View reviewed changes

	auto model_ptr = modelProvider.findModelByName(servableName);
	auto modelPtr = modelProvider.findModelByName(servableName);

	auto modelVersionsInstances = model_ptr->getModelVersionsMapCopy();
	auto modelVersionsInstances = modelPtr->getModelVersionsMapCopy();

	auto [state, error_code] = svsd->getStatus().convertToModelStatus();
	auto [state, errorCode] = svsd->getStatus().convertToModelStatus();

	ModelVersionStatusErrorCodeToString(error_code)};
	ModelVersionStatusErrorCodeToString(errorCode)};

	// note since removal of TFS V3 endpoints (from OpenAI are now also accepted as V1)
	// note since removal of TFS, V3 endpoints (from OpenAI) are now also accepted as V1

Uh oh!

Conversation

atobiszei commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dkalinowski commented Jul 3, 2026

Uh oh!

mzegla left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

atobiszei commented Jul 1, 2026 •

edited

Loading