Issue with Learning Service API – "Something went wrong in server while processing the request"

[rajnishdargan]

Hi @pallakartheekreddy
When we initially installed the services using the Sunbird Installer, the learning-service image that was being pulled via the Helm chart was:
shouryasolanki/learning-service:latest

In the current installer Helm chart, we observed that the image has been updated to:
sunbirded.azurecr.io/learning-service

We have accordingly updated our configuration to use:
sunbirded.azurecr.io/learning-service:latest

However, when we hit the following API:

curl --location 'https://<learning-service-domain>/learning-service/assessment/v3/items/create' \
--header 'Content-Type: application/json' \
--header 'Cookie: AWSALB=<REDACTED>; AWSALBCORS=<REDACTED>' \
--data '{
  "request": {
    "assessment_item": {
      "objectType": "AssessmentItem",
      "metadata": {
        "code": "NA",
        "isShuffleOption": false,
        "body": "{\"data\":{\"plugin\":{\"id\":\"org.ekstep.questionunit.mcq\",\"version\":\"1.3\",\"templateId\":\"horizontalMCQ\"},\"data\":{\"question\":{\"text\":\"<p>5+5 = ?</p>\\n\",\"image\":\"\",\"audio\":\"\",\"audioName\":\"\",\"hint\":\"\"},\"options\":[{\"text\":\"<p>10</p>\\n\",\"image\":\"\",\"audio\":\"\",\"audioName\":\"\",\"hint\":\"\",\"isCorrect\":true},{\"text\":\"<p>20</p>\\n\",\"image\":\"\",\"audio\":\"\",\"audioName\":\"\",\"hint\":\"\",\"isCorrect\":false}],\"questionCount\":0,\"media\":[]},\"config\":{\"metadata\":{\"max_score\":1,\"isShuffleOption\":false,\"isPartialScore\":true,\"evalUnordered\":false,\"templateType\":\"Horizontal\",\"name\":\"5+5 = ?\\n\",\"title\":\"5+5 = ?\\n\",\"copyright\":\"scp-channel\",\"qlevel\":\"EASY\",\"category\":\"MCQ\"},\"max_time\":0,\"max_score\":1,\"partial_scoring\":true,\"layout\":\"Horizontal\",\"isShuffleOption\":false,\"questionCount\":1,\"evalUnordered\":false},\"media\":[]}}",
        "itemType": "UNIT",
        "version": 2,
        "category": "MCQ",
        "createdBy": "<user-id>",
        "channel": "scp-channel",
        "type": "mcq",
        "template": "NA",
        "template_id": "NA",
        "framework": "level1-framework",
        "max_score": 1,
        "isPartialScore": true,
        "evalUnordered": false,
        "templateType": "Horizontal",
        "name": "5+5 = ?\n",
        "title": "5+5 = ?\n",
        "copyright": "scp-channel",
        "qlevel": "EASY",
        "options": [
          {
            "answer": true,
            "value": {
              "type": "text",
              "asset": "1"
            }
          }
        ]
      },
      "outRelations": []
    }
  }
}'

We receive the following error response:

{
  "id": "ekstep.learning.item.create",
  "ver": "1.0",
  "ts": "2025-04-30T08:43:30ZZ",
  "params": {
    "resmsgid": "efa45ab5-302f-42b6-9f70-95eb464548f3",
    "msgid": "fff526fe-ed41-4bb4-9387-46367b53f944",
    "err": "SYSTEM_ERROR",
    "status": "failed",
    "errmsg": "Something went wrong in server while processing the request"
  },
  "responseCode": "SERVER_ERROR",
  "result": {}
}

Strangely, no related logs appear in the learning-service logs — it always shows the same default output.

NOTE: Picked up JDK_JAVA_OPTIONS:  --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
25-Apr-2025 07:20:26.935 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.62
25-Apr-2025 07:20:26.952 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Mar 31 2022 14:34:15 UTC
25-Apr-2025 07:20:26.953 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.62.0
25-Apr-2025 07:20:26.954 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
25-Apr-2025 07:20:26.955 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.10.219-208.866.amzn2.x86_64
25-Apr-2025 07:20:26.956 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
25-Apr-2025 07:20:26.956 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/local/openjdk-11
25-Apr-2025 07:20:26.956 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           11.0.15+10
25-Apr-2025 07:20:26.957 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
25-Apr-2025 07:20:26.957 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /usr/local/tomcat
25-Apr-2025 07:20:26.959 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /usr/local/tomcat
25-Apr-2025 07:20:27.046 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.lang=ALL-UNNAMED
25-Apr-2025 07:20:27.046 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.io=ALL-UNNAMED
25-Apr-2025 07:20:27.047 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util=ALL-UNNAMED
25-Apr-2025 07:20:27.047 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
25-Apr-2025 07:20:27.047 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
25-Apr-2025 07:20:27.047 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties
25-Apr-2025 07:20:27.048 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
25-Apr-2025 07:20:27.048 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dconfig.file=/usr/local/tomcat/config/application.conf
25-Apr-2025 07:20:27.048 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
25-Apr-2025 07:20:27.048 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
25-Apr-2025 07:20:27.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
25-Apr-2025 07:20:27.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
25-Apr-2025 07:20:27.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat
25-Apr-2025 07:20:27.049 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
25-Apr-2025 07:20:27.050 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
25-Apr-2025 07:20:27.061 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.32] using APR version [1.7.0].
25-Apr-2025 07:20:27.061 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [true].
25-Apr-2025 07:20:27.061 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
25-Apr-2025 07:20:27.074 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1n  15 Mar 2022]
25-Apr-2025 07:20:28.552 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
25-Apr-2025 07:20:28.661 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [2674] milliseconds
25-Apr-2025 07:20:28.871 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
25-Apr-2025 07:20:28.920 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.62]
25-Apr-2025 07:20:28.940 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive [/usr/local/tomcat/webapps/learning-service.war]
25-Apr-2025 07:20:50.625 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
log4j:WARN No appenders could be found for logger (org.springframework.web.context.ContextLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2025-04-25 07:20:59,348 main WARN No Root logger was configured, creating default ERROR-level Root logger with Console appender
log4j:WARN Continuable parsing error 2 and column 42
log4j:WARN Document root element "Configuration", must match DOCTYPE root "null".
log4j:WARN Continuable parsing error 2 and column 42
log4j:WARN Document is invalid: no grammar found.
log4j:ERROR DOM element is - not a <log4j:configuration> element.
[INFO] [04/25/2025 07:21:08.946] [LearningActorSystem-akka.actor.default-dispatcher-4] [akka://LearningActorSystem/deadLetters] Message [java.lang.String] from Actor[akka://LearningActorSystem/user/$g/$a#-964430872] to Actor[akka://LearningActorSystem/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://LearningActorSystem/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
25-Apr-2025 07:21:09.436 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive [/usr/local/tomcat/webapps/learning-service.war] has finished in [40,496] ms
25-Apr-2025 07:21:09.443 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
25-Apr-2025 07:21:09.464 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [40802] milliseconds
[INFO] [04/25/2025 07:21:39.400] [LearningActorSystem-akka.actor.default-dispatcher-4] [akka://LearningActorSystem/deadLetters] Message [org.sunbird.common.dto.Response] from Actor[akka://LearningActorSystem/user/$g/$a#-964430872] to Actor[akka://LearningActorSystem/deadLetters] was not delivered. [2] dead letters encountered. If this is not an expected behavior, then [Actor[akka://LearningActorSystem/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.netty.util.internal.ReflectionUtil (file:/usr/local/tomcat/webapps/learning-service/WEB-INF/lib/netty-all-4.1.16.Final.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of io.netty.util.internal.ReflectionUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

We are using the following ConfigMap for the learning service (omitted here for brevity).
sunbird-learning-config--describe--1746002753.txt
Could you please help us identify what might be causing the API to fail with this server error?

@AbhilashKD @siddhi0719 @vinod303 @coolbung @snehal0904

[aimansharief]

Hey @rajnishdargan, could you SSH into the learning-service pod and navigate to the /usr/local/tomcat/logs directory. You'll find log files there with names starting with localhost_access_log. (e.g., localhost_access_log.2025-04-30.txt). Could you check for any errors in that file and share them with me?

[AbhilashKD]

Hey @aiman Sharief ***@***.***> Please find the attached file below.

…

On Wed, Apr 30, 2025 at 5:25 PM Aiman Sharief ***@***.***> wrote: Hey @rajnishdargan <https://github.com/rajnishdargan>, could you SSH into the learning-service pod and navigate to the /usr/local/tomcat/logs directory. You'll find log files there with names starting with localhost_access_log. (e.g., localhost_access_log.2025-04-30.txt). Could you check for any errors in that file and share them with me? — Reply to this email directly, view it on GitHub <#244 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5SL2YJPPDRAUZ22OCCGWCD24C24HAVCNFSM6AAAAAB4FCQOWOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOJZGI2DINA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[AbhilashKD]

logs.zip

[rajnishdargan]

@aimansharief Any update on learning service issue?

[aimansharief]

@rajnishdargan Actually there might be some issue with the docker file or some code configurations. Because I'm unable to see the application logs. I'll try to fix this and let you know

[aimansharief]

Hey @rajnishdargan , can you please try this health API and share the response

curl --location --globoff --request GET '{{learningservice_host}}/health' \
--header 'Content-Type: application/json' \
--data ''

[rajnishdargan]

@aimansharief health endpoint gives this response

    "id": "learning-service.health",
    "ver": "1.0",
    "ts": "2025-05-06T05:26:04ZZ",
    "params": {
        "resmsgid": "4c919dbd-f7a1-4d9a-ad4c-4b34bcb21795",
        "msgid": null,
        "err": null,
        "status": "successful",
        "errmsg": null
    },
    "responseCode": "OK",
    "result": {
        "checks": [
            {
                "healthy": true,
                "name": "domain graph"
            },
            {
                "healthy": true,
                "name": "redis cache"
            },
            {
                "healthy": true,
                "name": "cassandra db"
            }
        ],
        "healthy": true
    }
}

[AbhilashKD]

@aimansharief
Check below I have run the command in the learning pod itself :

root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# curl --location --globoff --request GET 'http://localhost:8080/health' --header 'Content-Type: application/json' --data ''
<!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/9.0.62</h3></body></html>root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# 
root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# 
root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# 
root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# 
root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# 
root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# curl --location --globoff --request GET 'http://localhost:8080/learning-service/health' --header 'Content-Type: application/json' --data ''
{"id":"learning-service.health","ver":"1.0","ts":"2025-05-06T05:28:53ZZ","params":{"resmsgid":"68f6d59f-bc3b-43b0-999f-926be9b61e75","msgid":null,"err":null,"status":"successful","errmsg":null},"responseCode":"OK","result":{"checks":[{"healthy":true,"name":"domain graph"},{"healthy":true,"name":"redis cache"},{"healthy":true,"name":"cassandra db"}],"healthy":true}}root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# 
root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# 
root@learning-694bc75c9b-xbjl9:/usr/local/tomcat# 

[aimansharief]

Looks like the endpoint isn't correct.
Can you update the endpoint with this http://localhost:8080/learning-service/health

[aimansharief]

@rajnishdargan I checked the code and found that the service-level logs are stored at /data/logs. So, whenever you trigger the API /learning-service/assessment/v3/items/create, please check the logs and share the relevant log entry with me.