Example of pyspark spline atlas #21

Activity

1. wajda commented on Jun 21, 2022

   

   wajda

   on Jun 21, 2022

   Contributor

   More actions

   Atlas integration is out of direct scope for the Spline team. There are a number of examples on the internet, some of them are listed here - https://absaoss.github.io/spline/articles.html
   Make sure you are referring to the more recent examples though, the ones that involve Spline version 0.5+
   Regarding Scala vs Python, there is no real difference with respect to integration with Atlas. If you can make Spline working for your Python job, the rest steps are pretty much the same. The principle is simple - the Spline agent collects the meta data from your job and sends it over to the destination you like. The destination is specified by the "LineageDispatcher" in use. See https://github.com/AbsaOSS/spline-spark-agent#dispatchers
2. AbsaOSS

   locked and limited conversation to collaborators on Jun 21, 2022
3. wajda

   converted this issue into a discussion #22 on Jun 21, 2022