Retrieving data from RM Server using Tibco Spotfire

SGolbert

Hi RMers!

I've been trying to retrieve a dataset from inside Tibco Spotfire using a RM web service. I've just retrieved the Titanic dataset and connected to the output port. Then in the web service I select Odata as output format and then JSON (I have tested all other variants as well).

However, I obtain an exception on the Spotfire side.

An exception was thrown by the data connection.<br>Exception text:<br>System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.<br>   at Spotfire.Dxp.Data.Access.CancellationHelper.Execute(CancellationToken token, Action action)<br>   at Spotfire.Dxp.Data.Access.CancellationHelper.Execute[T](CancellationToken token, Func`1 function)<br>   at Spotfire.Dxp.Data.Adapters.OData.ODataAdapterConnection.ListTablesCore(CancellationToken cancellationToken, ReadOnlyCollection`1& tables)<br>   at Spotfire.Dxp.Data.Access.Adapters.DataAdapterConnection.ListTables(CancellationToken cancellationToken, ReadOnlyCollection`1& tables)<br>   at Spotfire.Dxp.Data.Access.Adapters.DataAdapter.TryListTables(IDataAdapterExecutionContext executionContext, AdapterCredentialsValue credentials, CancellationToken cancellationToken, ReadOnlyCollection`1& tables)<br>   at Spotfire.Dxp.Data.Access.Adapters.DataAccessDataSourceLink.<>c__DisplayClass47_1.<ListTables>b__0(CancellationToken token)<br>   at Spotfire.Dxp.Data.Access.Adapters.DataAccessDataSourceLink.ExecuteAdapterMethod(ImpersonationPolicy impersonationPolicy, AdapterMethodDelegate method, Nullable`1 cancellationToken)

I have used user and password authentification, but that doesn't seem to be the problem. Could using anonymous login help? Has anybody tried something similar?

Thanks!

Sebastian

jwpfau

Hi Sebastian,

Have you tried to retrieve the data with another tool like Postman?

It seems to fail during the retrieval of the Service Metadata Document

http://rapidminer-vm.qs.intra:8080/api/rest/process/Titanic?/$metadata

A 404 page would also cause the NPE in Spotfire.

Maybe don't use OData but just some generic JSON parsing.

Greetings,
Jonas

SGolbert

Hi all,

I have used a R script in the end, but it was far easier that I thought:

library("httr")<br>library("jsonlite")<br><br>fullDomain <- "http://server_ip:8080/api/rest/process/Titanic?"<br>r <- GET(fullDomain, <br>         authenticate("sgolbert", "My Password"))<br><br>document <- fromJSON(content(r,"text"))

That actually brings the data to Spotfire, the rest is Spotfire specific. There is a workaround to make the user input user name and password and then encode it to base64, if you are concerned about security.

Regards,

Sebastian

jwpfau

Hi Sebastian,

it's probably an authentication error, but to be sure enable logging in your TIBCO Spotfire client under "Help > Support Diagnostics and Logging" > Logging tab, set the log level to Debug. The log file should contain a better error message.

Greetings,
Jonas

SGolbert

Hi Jonas,

I've done that and here is the log:

2019-01-15T13:23:59,944+01:00 2019-01-15 12:23:59,944 sgolbert [Main Thread] INFO  Spotfire.Dxp.Forms.Application.DiagnosticsDialog [(null)] - The log4net log level (for the root logger) was set to DEBUG<br>2019-01-15T13:24:02,475+01:00 2019-01-15 12:24:02,475 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Leaving modal operation.<br>2019-01-15T13:24:02,506+01:00 2019-01-15 12:24:02,506 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Framework.Menu.UserActionRegistration [(null)] - Dispatched UserAction 'helpSpotfireDiagnostics' from sender 'S&upport Diagnostics and Logging'  to 1 handlers. Time elapsed: 00:00:19.0424008<br>2019-01-15T13:24:06,994+01:00 2019-01-15 12:24:06,994 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Data.Persistence.DisposableDataPoolSectionKey [(null)] - Creating new DisposableDataPoolSectionKey. Tag: EmbeddedData Value: 32b91b60-3ab1-4b03-aee1-ce978e46055d.<br>2019-01-15T13:24:06,995+01:00 2019-01-15 12:24:06,995 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Data.Persistence.DisposableDataPoolSectionKey [(null)] - Creating new DisposableDataPoolSectionKey. Tag: LinkedData Value: 0cd82d2c-7b19-4dda-a38a-3a35e319082f.<br>2019-01-15T13:24:06,997+01:00 2019-01-15 12:24:06,997 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Data.DataManager [(null)] - No save settings defined in document manager, creating new.<br>2019-01-15T13:24:06,998+01:00 2019-01-15 12:24:06,998 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Data.DataManager [(null)] - Default save settings are not applicable<br>2019-01-15T13:24:07,015+01:00 2019-01-15 12:24:07,015 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Entering modal operation.<br>2019-01-15T13:24:15,227+01:00 2019-01-15 12:24:15,227 sgolbert [48] DEBUG Spotfire.Dxp.Framework.HttpClient.HttpClient [(null)] - Create non Spotfire request http://rapidminer-vm.qs.intra:8080/api/rest/process/Titanic?<br>2019-01-15T13:24:15,242+01:00 2019-01-15 12:24:15,242 sgolbert [48] DEBUG Spotfire.Dxp.Services.ServerInformation [(null)] - Setting server information; timeout: 3600000, keep alive: True<br>2019-01-15T13:24:15,399+01:00 2019-01-15 12:24:15,399 sgolbert [48] DEBUG Spotfire.Dxp.Internal.Utilities.Network [(null)] - Machine is part of domain: qs.intra<br>2019-01-15T13:24:15,477+01:00 2019-01-15 12:24:15,477 sgolbert [48] DEBUG Spotfire.Dxp.Services.ServerInformation [(null)] - Adding server to pre-authenticate: *.qs.intra<br>2019-01-15T13:24:15,524+01:00 2019-01-15 12:24:15,524 sgolbert [48] DEBUG Spotfire.Dxp.Framework.HttpClient.HttpClient [(null)] - Create Spotfire request to non spotfire server http://rapidminer-vm.qs.intra:8080/api/rest/process/Titanic?<br>2019-01-15T13:24:15,820+01:00 2019-01-15 12:24:15,820 sgolbert [48] DEBUG Spotfire.Dxp.Framework.HttpClient.HttpClient [(null)] - Create non Spotfire request http://rapidminer-vm.qs.intra:8080/api/rest/process/Titanic?<br>2019-01-15T13:24:15,820+01:00 2019-01-15 12:24:15,820 sgolbert [48] DEBUG Spotfire.Dxp.Framework.HttpClient.HttpClient [(null)] - Create Spotfire request to non spotfire server http://rapidminer-vm.qs.intra:8080/api/rest/process/Titanic?<br>2019-01-15T13:24:16,023+01:00 2019-01-15 12:24:16,023 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Leaving modal operation.<br>2019-01-15T13:24:16,102+01:00 2019-01-15 12:24:16,102 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Entering modal operation.<br>2019-01-15T13:24:16,118+01:00 2019-01-15 12:24:16,118 sgolbert [ProgressOperationThread] DEBUG Spotfire.Dxp.Data.Access.Adapters.AuthenticationModeService [(null)] - Retrieving authentication mode for adapter 'Spotfire.ODataAdapter' (client type: 'DesktopClient').<br>2019-01-15T13:24:16,118+01:00 2019-01-15 12:24:16,118 sgolbert [ProgressOperationThread] DEBUG Spotfire.Dxp.Data.Access.Adapters.NullImpersonationPolicy [(null)] - Impersonating using current user: WindowsIdentity, Name: QS\sgolbert, AuthenticationType: Negotiate, ImpersonationLevel: None, IsAnonymous: False, IsAuthenticated: True, IsGuest: False, IsSystem: False, Token: 7148, Owner: S-1-5-21-2417176989-1276124730-1766474528-3983 (), User: S-1-5-21-2417176989-1276124730-1766474528-3983 ()<br>2019-01-15T13:24:16,133+01:00 2019-01-15 12:24:16,133 sgolbert [47] DEBUG Spotfire.Dxp.Framework.HttpClient.HttpClient [(null)] - Create non Spotfire request http://rapidminer-vm.qs.intra:8080/api/rest/process/Titanic?/$metadata<br>2019-01-15T13:24:16,133+01:00 2019-01-15 12:24:16,133 sgolbert [47] DEBUG Spotfire.Dxp.Framework.HttpClient.HttpClient [(null)] - Create Spotfire request to non spotfire server http://rapidminer-vm.qs.intra:8080/api/rest/process/Titanic?/$metadata<br>2019-01-15T13:24:16,290+01:00 2019-01-15 12:24:16,290 sgolbert [ProgressOperationThread] DEBUG Spotfire.Dxp.Data.Access.CancellationHelper [(null)] - System.AggregateException: One or more errors occurred. ---> System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.<br>   at Spotfire.Dxp.Data.Adapters.OData.ODataAdapterConnection.ListTablesCoreImpl(List`1 tablesBuilder)<br>   at Spotfire.Dxp.Data.Access.CancellationHelper.<>c__DisplayClass1_0`1.<Execute>b__0()<br>   at Spotfire.Dxp.Data.Access.CancellationHelper.<>c__DisplayClass2_0.<Execute>b__0()<br>   at System.Threading.Tasks.Task.Execute()<br>   --- End of inner exception stack trace ---<br>   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)<br>   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)<br>   at Spotfire.Dxp.Data.Access.CancellationHelper.Execute(CancellationToken token, Action action)<br>---> (Inner Exception #0) System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.<br>   at Spotfire.Dxp.Data.Adapters.OData.ODataAdapterConnection.ListTablesCoreImpl(List`1 tablesBuilder)<br>   at Spotfire.Dxp.Data.Access.CancellationHelper.<>c__DisplayClass1_0`1.<Execute>b__0()<br>   at Spotfire.Dxp.Data.Access.CancellationHelper.<>c__DisplayClass2_0.<Execute>b__0()<br>   at System.Threading.Tasks.Task.Execute()<---<br><br>2019-01-15T13:24:16,290+01:00 2019-01-15 12:24:16,290 sgolbert [ProgressOperationThread] DEBUG Spotfire.Dxp.Data.Adapters.OData.ODataAdapterConnection [(null)] - ListTables executed in 158 ms. NullReferenceException<br>2019-01-15T13:24:16,317+01:00 2019-01-15 12:24:16,317 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Leaving modal operation.<br>2019-01-15T13:24:16,331+01:00 2019-01-15 12:24:16,331 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Entering modal operation.<br>2019-01-15T13:24:34,209+01:00 2019-01-15 12:24:34,209 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Leaving modal operation.<br>2019-01-15T13:24:34,291+01:00 2019-01-15 12:24:34,291 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Entering modal operation.<br>2019-01-15T13:24:34,327+01:00 2019-01-15 12:24:34,327 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Forms.Application.FormsApplicationThread [(null)] - Leaving modal operation.<br>2019-01-15T13:24:34,329+01:00 2019-01-15 12:24:34,329 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Framework.DocumentModel.DocumentManager.Document [(null)] - Dispose document for Empty.<br>2019-01-15T13:24:34,331+01:00 2019-01-15 12:24:34,331 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Framework.DocumentModel.DocumentManager.Document [(null)] - Dispose commandHistory for Empty.<br>2019-01-15T13:24:34,332+01:00 2019-01-15 12:24:34,332 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Framework.DocumentModel.DocumentManager.Document [(null)] - Dispose embeddedDataSectionKey 32b91b60-3ab1-4b03-aee1-ce978e46055d for Empty.<br>2019-01-15T13:24:34,333+01:00 2019-01-15 12:24:34,333 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Data.Persistence.DisposableDataPoolSectionKey [(null)] - Decreasing reference count on key. Value: 32b91b60-3ab1-4b03-aee1-ce978e46055d. ReferenceCount before decreasing: 1.<br>2019-01-15T13:24:34,333+01:00 2019-01-15 12:24:34,333 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Data.Persistence.DataPool [(null)] - Clearing section for key 32b91b60-3ab1-4b03-aee1-ce978e46055d, Embedded.<br>2019-01-15T13:24:34,334+01:00 2019-01-15 12:24:34,334 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Framework.DocumentModel.DocumentManager.Document [(null)] - Dispose linkedDataSectionKey 0cd82d2c-7b19-4dda-a38a-3a35e319082f for Empty.<br>2019-01-15T13:24:34,335+01:00 2019-01-15 12:24:34,335 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Data.Persistence.DisposableDataPoolSectionKey [(null)] - Decreasing reference count on key. Value: 0cd82d2c-7b19-4dda-a38a-3a35e319082f. ReferenceCount before decreasing: 1.<br>2019-01-15T13:24:34,335+01:00 2019-01-15 12:24:34,335 sgolbert [Main Thread] DEBUG Spotfire.Dxp.Data.Persistence.DataPool [(null)] - Clearing section for key 0cd82d2c-7b19-4dda-a38a-3a35e319082f, Linked.<br><br>

It is a NullPointer exception. I've been thinking that perhaps RM doesn't send the data in a compatible format.

Regards,

Sebastian

jwpfau

Hi Sebastian,

Have you tried to retrieve the data with another tool like Postman?

It seems to fail during the retrieval of the Service Metadata Document

http://rapidminer-vm.qs.intra:8080/api/rest/process/Titanic?/$metadata

A 404 page would also cause the NPE in Spotfire.

Maybe don't use OData but just some generic JSON parsing.

Greetings,
Jonas

SGolbert

Hi @jwpfau,

I looked into it, getting the data as JSON it's possible by using a R or Python script. This is a bit more complex to what I had in mind, so I will just save the data as Excel or CSV.

Thanks for the answer!

sgenzer

@land ?

SGolbert

Hi all,

I have used a R script in the end, but it was far easier that I thought:

library("httr")<br>library("jsonlite")<br><br>fullDomain <- "http://server_ip:8080/api/rest/process/Titanic?"<br>r <- GET(fullDomain, <br>         authenticate("sgolbert", "My Password"))<br><br>document <- fromJSON(content(r,"text"))

That actually brings the data to Spotfire, the rest is Spotfire specific. There is a workaround to make the user input user name and password and then encode it to base64, if you are concerned about security.

Regards,

Sebastian

sgenzer

Glad you found a solution, @SGolbert. I would strongly encourage you to check out Sebastian's new Web Automation extension with its extensive JSON parsing tools. They are very good.

Scott

SGolbert

I've read that they are good, but they are not free right? Is this coming to the core library someday?

sgenzer

@SGolbert no those tools are sold from Old World Computing (run by @land). TBH his pricing is VERY reasonable and well worth the $$ for what you get.

Scott