Request is not always captured when page sends it right after loading

[andrebcmarinho]

Hello! I'm loading a webpage that is always doing the same POST request and most of the times I'm able to get request body using this library. However, sometimes when shouldInterceptRequest() is called, webViewRequest.type is HTML instead of XML_HTTP and in that case POST request body is not available. Please note that webViewRequest.method is always POST as expected.

Any idea of what might be happening and how to fix this? Thank you

[acsbendi]

Hello, thanks for the report!

This means that the library's main request recorder didn't work, that is, no request was recorded. Could you please provide more details on where this happens so it can be debugged? For example, on what website does it happen?

[andrebcmarinho]

Hello @acsbendi! I've created a simple page to be able to replicate this issue: http://ynap-debug.atwebpages.com/
If you load this page multiple times, it will eventually fail after a few time.
Would you be able to take a look on your side? Thank you

[acsbendi]

@andrebcmarinho I've tried to reproduce this issue, but I couldn't, the request were logged correctly every time for me, producing the following log output:

Sending request from WebView: 
                                                                                                      Type: XML_HTTP
                                                                                                      URL: http://ynap-debug.atwebpages.com/redirect.php?rand=612
                                                                                                      Method: POST
                                                                                                      Body: {
                                                                                                                      "Id": 78912,
                                                                                                                      "Customer": "Jason Sweet",
                                                                                                                      "Quantity": 1,
                                                                                                                      "Price": 18.00
                                                                                                                    }
                                                                                                      Headers: 
                                                                                                           referer: http://ynap-debug.atwebpages.com/
                                                                                                           cookie: 
                                                                                                           origin: http://ynap-debug.atwebpages.com
                                                                                                           content-type: application/json
                                                                                                           cache-control: private
                                                                                                           accept: application/json
                                                                                                           user-agent: Mozilla/5.0 (Linux; Android 12; SM-A528B Build/SP1A.210812.016; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/111.0.5563.116 Mobile Safari/537.36
                                                                                                      Trace: 
                                                                                                        at XMLHttpRequest.send (<anonymous>:68:17)
                                                                                                        at doPostOriginal (http://ynap-debug.atwebpages.com/:28:21)
                                                                                                        at HTMLDocument.<anonymous> (http://ynap-debug.atwebpages.com/:47:17)
                                                                                                      Encoding type (form submissions only): null
                                                                                                      Is for main frame? false
                                                                                                      Is redirect? false
                                                                                                      Has gesture? false

Can you try to create a project with a minimal reproducible example?

[andrebcmarinho]

@acsbendi I've created a quick sample app where we can reproduce this issue. You can find it here

If you click multiple times on the "Request" button you will see on the logs with "debug-intercept" tag that we sometimes get this error. I'm using an emulator with API31. Thank you

[acsbendi]

@andrebcmarinho Thanks for the sample app, I've managed to reproduce the issue with it.

As I've mentioned before, this problem happens because the request recorder didn't work. I've spent some time debugging why, and the reason I found is that the script that sets up the recorder doesn't get executed fast enough after the page starts loading. The only way to resolve this is to inject the recorder setup code directly into the HTML page so that we can guarantee that it gets executed before any other Javascript. This requires sending the original main GET request (e.g. http://ynap-debug.atwebpages.com/ in your example) and modifying the returned body, as described in #2. So once that functionality is implemented, we can fix this issue too.

I've edited the issue's title so that it reflects my findings.