package cim
Spark Common Information Model (CIM) reader. Implements an Apache Spark file reader for CIM classes, generating an RDD for all Element objects and one for each specific CIM class. The reader fits into the overall Spark architecture as shown in the following image:
The architecture follows the standard Relation/InputFormat structure as other readers:
- Alphabetic
- By Inheritance
- cim
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
-
class
CHIM extends Serializable
Common Hierarchical Information Model (CHIM) for parsing CIM RDF files.
-
class
CIMAbout extends CIMRDD with Serializable
Handle "about" processing.
Handle "about" processing.
For each element with id X, if there are elements with "rdf:about='X'", this class merges the about elements into the "primary" element(s).
-
case class
CIMClassInfo(name: String, parseable: CIMParseable[Product], subsetter: CIMSubsetter[_ <: Product], relations: List[CIMRelationship], serializer: Serializer[_ <: Product]) extends Product with Serializable
Container for CIM class information.
Container for CIM class information.
- name
the name of the class without any package prefix, i.e. the original CIM class name
- parseable
the registration and subsetting object for this concrete class
- subsetter
the subsetter for this concrete class
- relations
the relationships of this concrete class
- serializer
the Kryo serializer for this concrete class
-
class
CIMContext extends AnyRef
Context for parsing.
Context for parsing. Contains the raw XML, indexes at which to start and stop parsing, the line number index of newlines within the XML, text coverage set (in debug) and error messages raised while parsing.
-
implicit
class
CIMDataFrameReader extends AnyRef
Adds a method,
cim, to DataFrameReader that allows you to read CIM files using the DataFileReader -
implicit
class
CIMDataFrameWriter[T] extends AnyRef
Adds a method,
cim, to DataFrameWriter that allows you to write CIM files using the DataFileWriter -
class
CIMDeDup extends CIMRDD with Serializable
Handle duplicate processing.
Handle duplicate processing.
For each element with id X, if there are other elements with "rdf:ID='X'", this class chooses only one.
This "duplicates" condition arises, for example, when spatial filters are applied to tile large datasets to partition the export task into smaller subsets, for reasons such as parallelization, memory constraints, etc.
A linear element crossing a tile boundary can be exported in either tile, if it can be determined beforehand which tile. A simpler option is to export such objects in all tiles whose spatial extents includes some of the element. It is also nice to include related elements to make each tile self consistent.
These tiles must then be recombined into the full dataset, which is the task for this component - to delete duplicate elements.
Warnings are generated if the deleted elements are not identical to the elements that are retained.
-
case class
CIMEdgeData(id_cn_1: String, id_cn_2: String, id_equ: String, isZero: Boolean, isConnected: Boolean) extends Product with Serializable
Edge data for topological processing.
Edge data for topological processing.
- id_cn_1
the connectivity node of terminal 0
- id_cn_2
the connectivity node of terminal 1 (or N in the case of multi-terminal devices)
- id_equ
the ch.ninecode.model.ConductingEquipment object associated with the terminals
- isZero
trueif there is no electrical difference between the terminals, i.e. a closed switch, which means the terminals are the same topological node- isConnected
trueif there is a connection between the terminals, i.e. a cable, which means the terminals are the same topological island
- class CIMEdges extends CIMRDD with Serializable
- class CIMFileFormat extends FileFormat
- class CIMInputFormat extends FileInputFormat[String, Element]
-
class
CIMIntegrityChecker extends CIMRDD with Serializable
Check relationships in CIM data.
-
case class
CIMIslandData(node: VertexId, island_label: String, island: VertexId = Long.MaxValue) extends Serializable with Product
Vertex data for island topological processing.
Vertex data for island topological processing.
- node
the minimum (hash code) of equivalent ConnectivityNode (a single topological node)
- island_label
a user friendly label for the island
- island
the minimum (hash code) of all connected ConnectivityNode (single topological island)
-
class
CIMJoin extends CIMRDD with Serializable
Join CIM files from NIS Strom and SAP ISU.
Join CIM files from NIS Strom and SAP ISU.
Resolve ServiceLocation objects based on the Name user attribute.
-
case class
CIMNetworkTopologyProcessor(spark: SparkSession) extends CIMRDD with Product with Serializable
Create a topology.
Create a topology.
Create TopologicalNode and optionally TopologicalIsland RDD that encode the connections between electrical elements and updates the Terminal and ConnectivityNode RDD to reference them.
This creates a "bus-branch" model, but retains the "node-breaker" model.
Based on ConnectivityNode elements and connecting edges, find the topology that has:
- each substation has a single bus (TopologicalNode) at each nominal voltage level for each set of BusbarSection that are connected by closed switches
- eliminates switches based on their open/closed state
- assigns each ConnectivityNode to exactly one TopologicalNode
- assigns each TopologicalNode to exactly one TopologicalIsland (islands are un-connected groups of internally-connected TopologicalNode)
- assigns to each TopologicalNode only one ConnectivityNodeContainer (a new unique generated container or one of the possibly many different existing ConnectivityNodeContainer (Bay, Line, Substation, VoltageLevel) of all the ConnectivityNode with the same TopologicalNode)
There are flags that affect whether a Switch (force_retain_switches) or Fuse (force_retain_fuses) is included in the topology:
To be done eventually: - create EquivalentEquipment (branch, injection, shunt) for an EquivalentNetworkforce_retain_switches Switch.retain Switch.open/
Switch.normalOpenin Topology ForceTrue don’t care don’t care yes ForceFalse don’t care true yes false no Unforced true don’t care yes false true yes false no - spark
The session with CIM RDD defined, for which the topology should be calculated
-
class
CIMNormalize extends CIMRDD with Serializable
Handle normalization.
Handle normalization.
For each element with a 1:N relation, ensure the N referece the 1 and not vice versa.
-
abstract
class
CIMParseable[+A <: Product] extends CIMParser
Typed base class for registration and subsetting.
Typed base class for registration and subsetting. Provides facilities to register subclasses with the CHIM parsing framework and forms the subsetting 'typed object' to specify RDDs of specific CIM classes. Typical usage:
object Terminal extends Parseable[Terminal] { // implement Parser abstract method def parse (context: Context): Terminal = ??? }
- A
The CIM class type.
-
trait
CIMParser extends AnyRef
Provides common infrastructure for parsing CIM elements.
Provides common infrastructure for parsing CIM elements. Subclasses (actually their companion objects) must implement the
parsemethod. Other methods are helpers for parsing the regular XML structure of CIM rdf files. -
trait
CIMRDD extends AnyRef
Access globally named and cached RDD of CIM classes.
Access globally named and cached RDD of CIM classes.
This uses the list of persistent RDDs maintained by Spark to retrieve pre-existing RDD of CIM classes persisted by the CIMReader.
These RDD are strongly typed, and are named according to the corresponding CIM class. For example, there is an
RDD[ACLineSegment]with the name"ACLineSegment"persisted and remembered in thespark.sparkContext.getPersistentRDDs: collection.Map[Int, RDD[_] ]by the CIMReader.Implicit parameters for the SparkSession and error Logger are required. Implicit parameter StorageLevel is required to put().
Declare a class that extends CIMRDD to be able to access the
get()methods:import org.apache.spark.rdd.RDD import org.apache.spark.sql.SparkSession import org.slf4j.Logger import ch.ninecode.CIMRDD import ch.ninecode.model._ class Processor (spark: SparkSession) extends CIMRDD with Serializable { implicit val log: Logger = LoggerFactory.getLogger (getClass) implicit val session = spark def process = { val switches: RDD[Switch] = get[Switch] ⋮ } }
Example: - class CIMRecordReader extends RecordReader[String, Element]
- class CIMRegistrator extends KryoRegistrator
-
class
CIMRelation extends BaseRelation with TableScan with CIMRDD
A Relation for CIM RDF files that can produce all of its tuples as an RDD of Row objects.
A Relation for CIM RDF files that can produce all of its tuples as an RDD of Row objects.
As a side effect, this relation also creates RDD and local temporary views corresponding to the subclasses of each element, and saves each RDD in the persistent RDD list using the CIM class name.
-
case class
CIMRelationship(field: String, clazz: String, this_cardinality: String, mate_cardinality: String) extends Product with Serializable
Relation description between CIM classes.
Relation description between CIM classes.
- field
the name of the field in this CIM class holding the reference to the other class
- clazz
the class name of the other class
- this_cardinality
the cardinality of this side of the relation
- mate_cardinality
the cardinality on the other side of the relation
- abstract class CIMSerializer[T] extends Serializer[T]
-
class
CIMSubsetter[A <: Product] extends Serializable
Subclass extractor.
Subclass extractor.
Extracts the given type of object from the full Element Resilient Distributes Dataset (RDD), to create another RDD of just those elements, and creates a DataFrame of that RDD, and registers it as a temporary table for access via SQL (e.g. JDBC and SparkR::sql()).
Note: This must be serializable and can't depend on the companion objects for the CIM case classes.
-
case class
CIMTopologyOptions(identify_islands: Boolean = false, force_retain_switches: State = Unforced, force_retain_fuses: State = Unforced, force_switch_separate_islands: State = Unforced, force_fuse_separate_islands: State = Unforced, default_switch_open_state: Boolean = false, debug: Boolean = false, storage: StorageLevel = StorageLevel.MEMORY_AND_DISK_SER) extends Product with Serializable
Topological processing options.
Topological processing options.
This class determines the behaviour of the CIMNetworkTopologyProcessor.
- identify_islands
When
true, topological islands are identified in addition to topological nodes. That is,TopologicalNodeobjects generated by the processor will have a validTopologicalIslandattributes that reference generatedTopologicalIslandobjects.- force_retain_switches
Keep Switch and subclasses as two topological node elements irregardless of the
retainedattribute, or theopenandnormalOpenattributes. This is used for alternative scenario calculations. It allows the user to override the behaviour when the processor encounters a Switch or a Switch derived class (e.g. Disconnector), except Fuse and ProtectiveSwitch classes are not included by this flag. The default behaviour ofUnforcedwill use the value of theretainedattribute to identify a node boundary only if the attribute is present in the CIM file and the value istrue. When set toForceTruethe behaviour is equivalent to each Switch having aretainedattribute with valuetrue. When set toForceFalsethe behaviour is equivalent to each Switch having aretainedattribute with valuefalse.- force_retain_fuses
Keep Fuse, ProtectedSwitch and subclasses as two topological node elements irregardless of the
retainedattribute, or theopenandnormalOpenattributes. This is used for fuse specificity calculation. It allows the user to override the normal behaviour when a Fuse is encountered, which is to keep the two terminals as two topological nodes only if theretainedattribute is present and true or theopenattribute is present and has a valuetrueor ifopenattribute is not present and thenormalOpenattribute has a valuetrue. This has the same effect for Fuse objects asforce_retain_switchesdoes for Switch objects.- force_switch_separate_islands
Extends the retain attribute to
TopologicalIslandprocessing forSwitchderived objects, except forFuseandProtectedSwitchobjects. The default ofUnforceduses the open or normalOpen attributes, if present, or thedefault_switch_open_statesetting otherwise, to determine if a switch separates two islands.ForceTruekeeps the switch in two islands, irregardless of the switch state, whileForceFalseensures the switch is in only one island.- force_fuse_separate_islands
Similar functionality as force_switch_separate_islands, but for
Fuseobjects.- default_switch_open_state
Allows changing the behaviour when the processor encounters a Switch that has neither an
openattribute, nornormalOpenattribute. The default behaviour offalseis the same as ifopenandnormalOpenboth specifyfalse.- debug
If
trueadditional tests are performed during topology processing:- unique VertexId for every ConnectivityNode mRID (checks the hash function)
- all edges reference existing ConnectivityNode elements (checks for completeness)
- no voltage transitions (checks that edges that are joined have the same BaseVoltage)
- storage
The storage level for new and replaced CIM RDD.
-
case class
CIMVD(node: VertexId = Long.MaxValue, node_label: String = "", voltage: String = null, container: String = null) extends Product with Serializable
Smaller version of CIMVertexData for identifyNodes().
Smaller version of CIMVertexData for identifyNodes().
- node
the minimum (hash code) of equivalent ConnectivityNode (a single topological node)
- node_label
a user friendly label for the node
- voltage
the nominal voltage of the node
- container
the node container
-
case class
CIMVertexData(island: VertexId = Long.MaxValue, island_label: String = "", node: VertexId = Long.MaxValue, node_label: String = "", voltage: String = null, container: String = null) extends Product with Serializable
Vertex data for topological processing.
Vertex data for topological processing.
- island
the minimum (hash code) of all connected ConnectivityNode (single topological island)
- island_label
a user friendly label for the island
- node
the minimum (hash code) of equivalent ConnectivityNode (a single topological node)
- node_label
a user friendly label for the node
- voltage
the nominal voltage of the node
- container
the node container
- class DefaultSource extends RelationProvider
- class Extremum extends Serializable
- case class PostEdge(id_seq_1: String, id_seq_2: String, id_equ: String, clazz: String, name: String, aliasName: String, description: String, container: String, length: Double, voltage: String, normalOpen: Boolean, ratedCurrent: Double, power: Double, installationDate: String, receivedDate: String, status: String, x1: String, y1: String, x2: String, y2: String) extends Serializable with Product
- case class PreEdge(id_seq_1: String, cn_1: String, id_seq_2: String, cn_2: String, id_equ: String, clazz: String, name: String, aliasName: String, description: String, container: String, length: Double, voltage: String, normalOpen: Boolean, ratedCurrent: Double, location: String, power: Double, status: String) extends Serializable with Product
- sealed trait State extends AnyRef
- case class TopoEdge(id_seq_1: String, id_island_1: String, id_seq_2: String, id_island_2: String, id_equ: String, clazz: String, name: String, aliasName: String, description: String, container: String, length: Double, voltage: String, normalOpen: Boolean, ratedCurrent: Double, power: Double, installationDate: String, receivedDate: String, status: String, x1: String, y1: String, x2: String, y2: String) extends Serializable with Product
- class Worker[C <: Product, P <: Product] extends Serializable
Value Members
- object CHIM extends Serializable
-
object
CIMClasses
Get a list of classes suitable for Kryo registration.
- object CIMContext
-
object
CIMParser
Holds constant members of the Parser trait.
Holds constant members of the Parser trait. Includes constants for use by subclasses and the parser. This parser companion object is only needed because Scala doesn't have a static declaration, and instead invents a "companion object" to hold the trait or class members that should be accessible without an instantiated object.
- object CIMTopologyOptions extends Serializable
- object ForceFalse extends State with Product with Serializable
- object ForceTrue extends State with Product with Serializable
- object Unforced extends State with Product with Serializable