dlt.extract.reference

SourceFactory Objects

class SourceFactory(ABC, Generic[TSourceFunParams, TDltSourceImpl])

View source on GitHub

clone

@abstractmethod
def clone(*,
          name: str = None,
          section: str = None,
          max_table_nesting: int = None,
          root_key: bool = None,
          schema: Schema = None,
          schema_contract: TSchemaContract = None,
          spec: Type[BaseConfiguration] = None,
          parallelized: bool = None,
          _impl_cls: Type[TDltSourceImpl] = DltSource) -> Self

View source on GitHub

Overrides default decorator arguments that will be used to when DltSource instance and returns modified clone.

SourceReference Objects

class SourceReference()

View source on GitHub

SOURCES

A registry of all the decorated sources and resources discovered when importing modules

ref

A fully qualified reference: module.name to the factory instance

expand_shorthand_ref

@classmethod
def expand_shorthand_ref(cls, ref: str) -> List[str]

View source on GitHub

Converts ref into fully qualified form, return one or more alternatives for shorthand notations. Run context is injected if needed. Following formats are recognized

• section.name
• name

find

@classmethod
def find(cls,
         ref: str,
         raise_exec_errors: bool = False,
         import_missing_modules: bool = False,
         _impl_sig: Callable[TSourceFunParams, Any] = None,
         _impl_cls: Type[TDltSourceImpl] = None) -> Any

View source on GitHub

Returns source factory from reference ref. Looks into registry or tries auto-import. Expands shorthand notation into section.name eg. "sql_database" is expanded into "dlt.sources.sql_database.sql_database".

You can control auto-import behavior:

• raise_exec_errors - will re-raise code execution errors in imported modules
• import_missing_modules - will ignore missing dependencies during import by substituting them with dummy modules. this should be only used to manipulate local dev environment

from_reference

@classmethod
def from_reference(cls,
                   ref: str,
                   name: str = None,
                   section: str = None,
                   max_table_nesting: int = None,
                   root_key: bool = None,
                   schema: Schema = None,
                   schema_contract: TSchemaContract = None,
                   spec: Type[BaseConfiguration] = None,
                   parallelized: bool = None,
                   _impl_cls: Type[TDltSourceImpl] = None,
                   source_args: Tuple[Any, ...] = None,
                   source_kwargs: Dict[str, Any] = None) -> TDltSourceImpl

View source on GitHub

Find registered source factory or imports it, then instantiates the DltSource using passed args and kwargs. Passes additional arguments to clone of source factory