Random NullReference and InvalidCast crashes in mono class libraries #1188

Activity

1. Belorus

   changed the title [-]Random NullRef crashes in SSL/TLS stack[/-] [+]Random NullRef crashes in SSL/TLS (not only) stack[/+] on Jan 12, 2018
2. grendello

   assigned

   marek-safar

   and

   baulig

   on Jan 12, 2018
3. marek-safar

   unassigned

   marek-safar

   and

   baulig

   on Jan 15, 2018
4. marek-safar commented on Jan 15, 2018

   

   marek-safar

   on Jan 15, 2018

   Contributor

   More actions

   /cc @brendanzagaeski could Mono part of the problem be addressed by your latest fix
5. brendanzagaeski commented on Jan 16, 2018

   

   brendanzagaeski

   on Jan 16, 2018

   Contributor

   More actions

   Hmm. These look a bit different. I quickly checked to see how unobserved exceptions from Tasks look on Xamarin.Android. As it turns out, exception telemetry coming from the UnobservedTaskException event would by default not include the android.runtime.JavaProxyThrowable Java type because the exception would be sent before the transition out of the C# runtime. I think the presence of android.runtime.JavaProxyThrowable in the exceptions in the initial comment means the exceptions have propagated to a Java thread by the time they are sent, and that wouldn't happen by default for the unobserved exceptions targeted by my recent fix.
6. jonpryor commented on Jan 16, 2018

   

   jonpryor

   on Jan 16, 2018

   Contributor

   More actions

   @brendanzagaeski, @marek-safar: Related: Issue #1144.

   I think the presence of android.runtime.JavaProxyThrowable in the exceptions in the initial comment means the exceptions have propagated to a Java thread by the time they are sent

   Not quite; the presence of JavaProxyThrowable means that the exception was propagated to a Java stack frame.

   For example, consider Android calling an Activity.OnCreate() override:

   [Activity]
   partial class BadExample : Activity {
       protected override void OnCreate(Bundle bundle) {
           throw new InvalidOperationException("Bwa-ha-ha-ha");
       }
   }

   In the above, we'll have a Java stack frame calling BadExample.OnCreate() (through a delegate/JNI), and thus a stack of:

   Java Method > Managed Method (`BadActivity.OnCreate()`)
   

   When BadActivity.OnCreate() throws InvalidOperationException, it is wrapped into a JavaProxyThrowable and then JNIEnv::Throw() is called, making the JavaProxyThrowable a "pending" exception. When control returns to Java Method, the pending JavaProxyThrowable is thrown in the JVM, allowing normal catch and finally blocks to execute.

   Unfortunately (Issue #1144) we neglected to "unwrap" the JavaProxyThrowable before raising the AppDomain.UnhandleException event, which is probably why it's showing up in all of those stack traces...IFF a Java > Managed transition occurred, anyway.

   👍React with 👍2Reacted by Brendan Zagaeski and Grigory
7. jonpryor commented on Jan 16, 2018

   

   jonpryor

   on Jan 16, 2018

   Contributor

   More actions

   @Belorus wrote:

   we'll definitely try SGEN in next release

   Xamarin.Android doesn't support Boehm. You're already using SGen -- it's the only GC that Xamarin.Android supports.

   Crashes on Androids:

   How are you collecting those stack traces? They seem oddly truncated, in that I would expect more than two calling frames. Maybe even the Java-side stack trace?
8. Belorus commented on Jan 16, 2018

   

   Belorus

   on Jan 16, 2018

   Author

   More actions

   @jonpryor Via HockeyApp SDK 4.1.5
   https://snag.gy/3JqLQY.jpg
   https://snag.gy/reazTl.jpg
9. jonpryor commented on Jan 17, 2018

   

   jonpryor

   on Jan 17, 2018

   Contributor

   More actions

   @Belorus: I don't recognize that interface. :-(

   I can only assume it's a web UI to some crash reporter, which means I need to ask what the crash reporter is, how it works, and (most importantly) where is the rest of the stack trace?

   :-/
10. jonpryor commented on Jan 17, 2018

    

    jonpryor

    on Jan 17, 2018

    Contributor

    More actions

    I can only assume it's a web UI to some crash reporter

    I can only assume I need to spend more time on reading comprehension, as you said it was "HockeyApp SDK 4.1.5".

    Doh!

    Unfortunately that doesn't immediately answer the "how does it work", though a bit of spelunking suggests that AppDomain.UnhandledException is used as well as java.lang.Thread.setDefaultUncaughtExceptionHandler().

    The possible answer to "where is the rest of the stack trace" is probably "< insert gnashing of teeth here >" ^H^H^H I mean "They use Throwable.printStackTrace(), with no calls to Throwable.getMessage()."

    JavaProxyThrowable provides the result of innerException.ToString() as the "message" for the base class, accessible via Throwable.getMessage(). (This is because I didn't know of a way to add stack frames to a java.lang.Throwable, apparently having completely overlooked Throwable.setStackTrace().

    At present, Throwable.getMessage() is the only way to get the inner/managed stack trace held by the JavaProxyThrowable instance, and it appears that (conjecture!) -- while Throwable.getMessage() is seemingly called -- which is presumably why we get the System.NullReferenceException: Object reference not set to an instance of an object message -- it's being truncated to one line.

    Which isn't particularly helpful.
11. Belorus commented on Jan 17, 2018

    

    Belorus

    on Jan 17, 2018

    Author

    More actions

    Today i saw the winner (https://snag.gy/9R6ewZ.jpg):

    Xamarin caused by: android.runtime.JavaProxyThrowable: System.NullReferenceException: Object reference not set to an instance of an object
    

    We have realtime logs on client devices, there i saw the following (Still truncated, although received not via HockeyApp):

    AppDomain.CurrentDomain.UnhandledException. UtcNow is 2018-01-15T11:40:31. --> System.NullReferenceException: Object reference not set to an instance of an object
    

    In code it looks like (simplified):

    AppDomain.CurrentDomain.UnhandledException += OnCurrentDomainUnhandledException;
    
    // ...
    
    private static void OnCurrentDomainUnhandledException(object sender, UnhandledExceptionEventArgs e)
    {
        if (e.ExceptionObject is Exception exception)
        {
            Log.Fatal($"AppDomain.CurrentDomain.UnhandledException. UtcNow is {DateTime.UtcNow:s}. --> " + exception.ToString());
        }
    }
    

    Looks like exception that we get in event handler is already very lets say... consise.
12. baulig commented on Jan 17, 2018

    

    baulig

    on Jan 17, 2018

    Contributor

    More actions

    I'm not exactly sure why, but this is using the legacy TLS stack, which is no longer supported and will soon be deleted.

    Looking over your stack traces:

    1. Legacy TLS, unsupported and scheduled for removal
    2. same
    3. We would need a full stack trace here - where's the SslStream used, what may have disposed it.
    4. Client Certificates are not supported - though we should only throw NotImplemented / NotSupported in the setter and not throw at all in the getter.
    5. This property returns an instance field - I don't see how we could null-ref here:

        internal ServerCertValidationCallback ServerCertValidationCallback {
            get { return certValidationCallback; }
        }
    

    6. Legacy TLS, see 1.

    I am very confused about these stack traces and don't really understand what's happening here. If this is related to TLS at all, then we should try to at least use BTLS on Android. Testing with the new web stack (once it gets integrated into Android) would be helpful as well.

    Let me know if there's anything I can help you with - at the moment, I'm not sure what I could possibly do because I just don't understand what's happening.
13. Belorus

    changed the title [-]Random NullRef crashes in SSL/TLS (not only) stack[/-] [+]Random NullReference and InvalidCast crashes in mono class libraries[/+] on Jan 17, 2018
14. 1 remaining item

    Load more

15. Belorus commented on Jan 19, 2018

    

    Belorus

    on Jan 19, 2018

    Last edited by Belorus

    Author

    More actions

    
    This is crash chart of one of our apps after transition from XA7.x (don't remember exactly) to 8.1
    As you see most crashes are not logic related, these are issues of runtime. We didn't have any of these before.

    
16. Belorus commented on Jan 19, 2018

    

    Belorus

    on Jan 19, 2018

    Last edited by Belorus

    Author

    More actions

    Is there any way to downgrade Xamarin Andoid to 7.4 inside VS2017?
    I've downloaded https://xamarin.azureedge.net/XamarinforVisualStudio/Windows/Xamarin.VisualStudio_4.6.3.4.msi but it doesn't change anything after installation.
17. joj commented on Jan 19, 2018

    

    joj

    on Jan 19, 2018

    Last edited by joj

    Member

    More actions

    @Belorus the msi is only installing for VS2015. There is no easy way to downgrade an installed package in VS2017 AFAIK. I may be wrong, though.
18. brendanzagaeski commented on Jan 19, 2018

    

    brendanzagaeski

    on Jan 19, 2018

    Contributor

    More actions

    I'm not sure if it will be helpful for this particular scenario (Xamarin.Android 7.1.0.43 might be too old for what you need), but for reference, Visual Studio 2017 version 15.0 is available from the Downloads section of my.VisualStudio.com to align with the Visual Studio servicing guidelines.

    (Related: Installing an earlier release of Visual Studio 2017.)
19. akravch commented on Apr 10, 2019

    

    akravch

    on Apr 10, 2019

    More actions

    Hi, sorry for necroposting, but are there any updates regarding this? We are still experiencing this issue with the crashes around TLS, even thought we've updated Xamarin and VS since then:

    Android SDK Tools 1.16
    Visual Studio Professional 2017 15.9.28307.222
    Microsoft .NET Native SDK 15.0.24211
    Xamarin.Android 9.1.4.2 (HEAD/8255f42fc)
    SSL/TLS implementation Native TPS 1.2+

    Here is a few of the crashes:

    Xamarin caused by: android.runtime.JavaProxyThrowable: System.NullReferenceException: Object reference not set to an instance of an object
      at (wrapper xdomain-invoke) Mono.Security.Protocol.Tls.TlsStream.Write(byte[])
      at (wrapper remoting-invoke-with-check) Mono.Security.Protocol.Tls.TlsStream.Write(byte[])
    

    Xamarin caused by: android.runtime.JavaProxyThrowable: System.NullReferenceException: Object reference not set to an instance of an object
      at (wrapper xdomain-invoke) Mono.Security.Protocol.Tls.TlsStream.get_EOF()
      at (wrapper remoting-invoke-with-check) Mono.Security.Protocol.Tls.TlsStream.get_EOF()
    

    Xamarin caused by: android.runtime.JavaProxyThrowable: System.NullReferenceException: Object reference not set to an instance of an object
      at (wrapper xdomain-invoke) System.Net.HttpWebRequest.get_TlsSettings()
      at (wrapper remoting-invoke-with-check) System.Net.HttpWebRequest.get_TlsSettings()
    
20. jonathanpeppers commented on Jun 1, 2022

    

    jonathanpeppers

    on Jun 1, 2022

    Member

    More actions

    Would anyone be able to comment if this issue is solved in .NET 6 Android projects? Thanks!
21. jonathanpeppers

    self-assigned this

    on Jun 1, 2022
22. jonathanpeppers

    added

    need-infoIssues that need more information from the author.Issues that need more information from the author.

    on Jun 1, 2022
24. jonathanpeppers

    added this to the Under Consideration milestone on Jun 1, 2022
25. jonathanpeppers

    added

    Area: Mono RuntimeMono-related issues: BCL bugs, AOT issues, etc.Mono-related issues: BCL bugs, AOT issues, etc.

    Area: App RuntimeIssues in `libmonodroid.so`.Issues in `libmonodroid.so`.

    on Jun 1, 2022
27. ghost closed this as completedon Jun 8, 2022
28. ghost locked as resolved and limited conversation to collaborators on Jul 9, 2022