Error with SQL Server and smalldatetime field

[endersonmaia]

When I use a SQL query with some smalldatetime field, I get this error :

[ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.

{
    "exception" => "Missing Valuefier handling for full class name=org.jruby.RubyObject, simple name=RubyObject"
    "backtrace" => [
      "org.logstash.Valuefier.convertNonCollection(Valuefier.java:51)"
      "org.logstash.Valuefier.convert(Valuefier.java:90)"
      "org.logstash.ConvertedMap$1.visit(ConvertedMap.java:43)"
      "org.jruby.RubyHash.visitLimited(RubyHash.java:648)"
      "org.jruby.RubyHash.visitAll(RubyHash.java:634)"
      "org.logstash.ConvertedMap.newFromRubyHash(ConvertedMap.java:39)"
      "org.logstash.Valuefier.convert(Valuefier.java:63)"
      "org.logstash.ConvertedList.newFromRubyArray(ConvertedList.java:43)"
      "org.logstash.Valuefier.convert(Valuefier.java:66)"
      "org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_set_field(JrubyEventExtLibrary.java:128)"
      "org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$2$0$ruby_set_field.gen)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)"
      "org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:219)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)"
      "org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.IfNode.interpret(IfNode.java:118)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)"
      "org.jruby.ast.FCallOneArgBlockPassNode.interpret(FCallOneArgBlockPassNode.java:32)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)"
      "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)"
      "org.jruby.ast.FCallOneArgBlockNode.interpret(FCallOneArgBlockNode.java:34)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.ast.IfNode.interpret(IfNode.java:118)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)"
      "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)"
      "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)"
      "org.jruby.runtime.Block.yield(Block.java:142)"
      "org.jruby.RubyArray.eachCommon(RubyArray.java:1606)"
      "org.jruby.RubyArray.each(RubyArray.java:1613)"
      "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)"
      "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)"
      "org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)"
      "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)"
      "org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)"
      "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)"
      "org.jruby.ast.DAsgnNode.interpret(DAsgnNode.java:110)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)"
      "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)"
      "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)"
      "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)"
      "org.jruby.runtime.Block.call(Block.java:101)"
      "org.jruby.RubyProc.call(RubyProc.java:300)"
      "org.jruby.internal.runtime.methods.ProcMethod.call(ProcMethod.java:64)"
      "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)"
      "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)"
      "rubyjit.LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170.block_0$RUBY$__file__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:353)"
      "rubyjit$LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170$block_0$RUBY$__file__.call(rubyjit$LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170$block_0$RUBY$__file__)"
      "org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:159)"
      "org.jruby.runtime.CompiledBlock19.call(CompiledBlock19.java:87)"
      "org.jruby.runtime.Block.call(Block.java:101)"
      "org.jruby.RubyProc.call(RubyProc.java:300)"
      "org.jruby.RubyProc.call19(RubyProc.java:281)"
      "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)"
      "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)"
      "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)"
      "rubyjit.LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170.block_0$RUBY$__file__(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:259)"
      "rubyjit$LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170$block_0$RUBY$__file__.call(rubyjit$LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170$block_0$RUBY$__file__)"
      "org.jruby.runtime.CompiledBlock19.yield(CompiledBlock19.java:135)"
      "org.jruby.runtime.Block.yield(Block.java:142)"
      "org.jruby.RubyHash$13.visit(RubyHash.java:1355)"
      "org.jruby.RubyHash.visitLimited(RubyHash.java:648)"
      "org.jruby.RubyHash.visitAll(RubyHash.java:634)"
      "org.jruby.RubyHash.iteratorVisitAll(RubyHash.java:1306)"
      "org.jruby.RubyHash.each_pairCommon(RubyHash.java:1351)"
      "org.jruby.RubyHash.each19(RubyHash.java:1342)"
      "org.jruby.RubyHash$INVOKER$i$0$0$each19.call(RubyHash$INVOKER$i$0$0$each19.gen)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)"
      "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)"
      "rubyjit.LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb:258)"
      "rubyjit.LogStash::Util::WrappedAckedQueue::ReadBatch$$each_460b01f11b82d55a13c70efd2aabe7c62fbc86591442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_acked_queue.rb)"
      "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:161)"
      "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)"
      "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)"
      "rubyjit.LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170.chained_0_rescue_1$RUBY$SYNTHETIC__file__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352)"
      "rubyjit.LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)"
      "rubyjit.LogStash::Pipeline$$filter_batch_e0d70a513c9b572e72031cc1bba279d4ffc82a801442407170.__file__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)"
      "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)"
      "org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.ast.WhileNode.interpret(WhileNode.java:131)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)"
      "org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)"
      "org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:219)"
      "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)"
      "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)"
      "org.jruby.ast.FCallTwoArgNode.interpret(FCallTwoArgNode.java:38)"
      "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)"
      "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)"
      "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)"
      "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)"
      "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)"
      "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)"
      "org.jruby.runtime.Block.call(Block.java:101)"
      "org.jruby.RubyProc.call(RubyProc.java:300)"
      "org.jruby.RubyProc.call(RubyProc.java:230)"
      "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)"
      "java.lang.Thread.run(Thread.java:745)"
  ]
}

[endersonmaia]

I tryied to replicate the error, and created a new table with some fields and a smalldatetime one, but couldn't replicate the error.

I simplifyied the setup and tested with my actual database, and the error is not showing.

I got to the conclusion the error was with a smalldatetime field, 'cause when I remove this field from the query, I get the error.

I'll try some more different setups to see if I can replicate it.

If I can't replicate, I'll try to show my actual setup (it's big queries and messy, that's why I'm avoiding showing it here).

[endersonmaia]

Close this issue, I couldn't replicate it! Sorry!

[guyboertje]

We do have an outstanding issue with JDBC type plugins that try to add a Date datatype instance to an event which we do not currently support. The Java based Event feature needs to be able to seamlessly convert Ruby datatypes to their equivalent Java datatype and back again losslessly.

I need to modify the code to check for any date or datetime objects and safely convert them.

[guyboertje]

I am leaving this issue open for a while for others benefit.

[endersonmaia]

We do have an outstanding issue with JDBC type plugins

Since you mentioned this, I noticed an inconssistant behavior between input-jdbc and filter-jdbc_streaming around SQL Server numeric data type.

I presume this could be related to this convertions of data types, right ?