Packages

c

ch.ninecode.cim

CIMFileFormat

class CIMFileFormat extends FileFormat

Linear Supertypes
FileFormat, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. CIMFileFormat
  2. FileFormat
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new CIMFileFormat()

Type Members

  1. case class dummy(sup: Element = null) extends Element with Product with Serializable

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def buildReader(sparkSession: SparkSession, dataSchema: StructType, partitionSchema: StructType, requiredSchema: StructType, filters: Seq[Filter], options: Map[String, String], hadoopConf: Configuration): (PartitionedFile) ⇒ Iterator[InternalRow]

    Returns a function that can be used to read a single file in as an Iterator of InternalRow.

    Returns a function that can be used to read a single file in as an Iterator of InternalRow.

    dataSchema

    The global data schema. It can be either specified by the user, or reconciled/merged from all underlying data files. If any partition columns are contained in the files, they are preserved in this schema.

    partitionSchema

    The schema of the partition column row that will be present in each PartitionedFile. These columns should be appended to the rows that are produced by the iterator.

    requiredSchema

    The schema of the data that should be output for each row. This may be a subset of the columns that are present in the file if column pruning has occurred.

    filters

    A set of filters than can optionally be used to reduce the number of rows output

    options

    A set of string -> string configuration options.

    returns

    A function to convert the file into a set of rows.

    Definition Classes
    CIMFileFormat → FileFormat
  6. def buildReaderWithPartitionValues(sparkSession: SparkSession, dataSchema: StructType, partitionSchema: StructType, requiredSchema: StructType, filters: Seq[Filter], options: Map[String, String], hadoopConf: Configuration): (PartitionedFile) ⇒ Iterator[InternalRow]
    Definition Classes
    FileFormat
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  12. def inferSchema(sparkSession: SparkSession, options: Map[String, String], files: Seq[FileStatus]): Option[StructType]

    When possible, this method should return the schema of the given files.

    When possible, this method should return the schema of the given files. Derrick: The schema of a CIM file is always just a single Element

    Definition Classes
    CIMFileFormat → FileFormat
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. def isSplitable(sparkSession: SparkSession, options: Map[String, String], path: Path): Boolean

    Returns whether a file with path could be splitted or not.

    Returns whether a file with path could be splitted or not. Derrick: This is used to allow (or not) the splitting of compressed files, and since the CIM RDF files are not compressed (for now), we return true.

    Definition Classes
    CIMFileFormat → FileFormat
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  18. def prepareWrite(sparkSession: SparkSession, job: Job, options: Map[String, String], dataSchema: StructType): OutputWriterFactory

    Prepares a write job and returns an OutputWriterFactory.

    Prepares a write job and returns an OutputWriterFactory. Currently not supported

    Definition Classes
    CIMFileFormat → FileFormat
  19. def supportBatch(sparkSession: SparkSession, dataSchema: StructType): Boolean
    Definition Classes
    FileFormat
  20. def supportDataType(dataType: DataType): Boolean
    Definition Classes
    FileFormat
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. def vectorTypes(requiredSchema: StructType, partitionSchema: StructType, sqlConf: SQLConf): Option[Seq[String]]
    Definition Classes
    FileFormat
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  26. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated @deprecated
    Deprecated

    (Since version ) see corresponding Javadoc for more information.

Inherited from FileFormat

Inherited from AnyRef

Inherited from Any

Ungrouped