Excel files are called spreadsheets. Powerpoint files are called presentations. Similarly Pipe-It "files" are called projects. Pipe-It projects consists of many component files. Not just single files like most other applications. Pipe-It Project View (*.ppv) file stores the visual content of the project. Pipe-It Project Model (*.ppm) file is most important and stores the logical model describing all the elements in the project their connections, their grouping into composites, the files associated with each resource, the command lines of each scripter etc. Pipe-It Project Linkz (*.ppl) file stores all the variables collected from all text files that are part of the project, their current values and algorithms to locate them in later runs. Pipe-It Project Streamz Library (*.pps) file stores Streamz characterizations and conversions used in the project.
The location of these 4 files is called the root folder of the project. Most other files mentioned under can be located anywhere on the disk. However, to make the project portable it is recommended that all such files are located at or under the root folder.
If the project contains optimization, one or a series of Pipe-It Project Optimization (*.ppo) files are also present. They can be located anywhere.
Since Pipe-It is a multiple application integration framework, allowing the launch of many internal and external applications, a large number of application specific files become part of the project. Many of these files may be represented as Resources in the Pipe-It project and become "known" to the project. Many other files just exist and are created, read and written by applications. All files connected to initiating resources (those that are not downstream of any process) are also mandatory for the proper execution of the project. These files are typically located at the root or in folders under the root.
Designing a Pipe-It project requires some planning. Simple projects might contain only a handful of data files linked to resources and these can live in the project root folder. But real life projects contain hundreds if not thousands of files. It makes sense to organize these files in a sensible folder hierarchy. The project within Pipe-It should also follow the folder structure using composites and sub-composites as far as possible. This helps in cloning of composites in case a new instance of the functionality encapsulated in the composite is to be created. Pipe-It estimates the % of files that resides within a particular sub-folder of the project and offers to the user the possibility of duplicating files. This is really a powerful option as all files, their connections to resources, their Linkz variables, and all process command lines are cloned.
Making order of the hundreds and thousand files is another reason for collecting related files together.
Project is container for all elements of modeling processes. Of course keeping all the elements on same canvas makes the scheme very complex for understanding and tracking processe details. To help with this some or all elements in a project can be organized in Sub-Processes or Composites.
We recommend to use the term Project for meaning whole project. Also it means a set of 4 files that defines the project - logical model + visual model + variable links + Streamz Library (ppm + ppv + ppl + pps).
The term Composite is used for defining some part of whole project that is organized separately from other elements. Actually composites are small sub-projects that works in same way as other elements of canvas.
Because Composite is actually sub-project that defines some operations, it can also have nested composites of second level and so on.
From this point of view, the project is tree of composites:
Or similar but top to bottom structure:
As mentioned in the section on Projects, composites are a way to make a project easier to understand. They will compose parts of a project together into one box for a less cluttered appearance. For example one may make one composite for the reservoir part of the field, and one for the processing part of the field. Each composite may also be run individually. Project can have any amount of nested composites:
Connections from elements within a composite to elements outside (or inside other composites) are made via Sockets that look visually unique for each different type.
It is possible to export complete composites to standalone projects by using:
Selecting the Composite and right.clicking give this dialog. There are two possibilities:
Importing composites is done Selecting the File/Import menu option after which the typical file browser window allows you to find the ppv file to import.
An alternate way is to drag the ppv file, like any other resource, on to the canvas. In this case a dialog to choose insertion of resource or import of composite is offered:
Except for connectors, every element has a number of sockets. When you connect two elements, you actually connect an output socket from one to an input socket on the other. Each socket has a different purpose. Think of connecting a stereo receiver to a set of speakers. You can't just attach your wires at random. Each speaker will have exactly two terminals (sockets), but the receiver might have 20 or 30, all with a different purpose. You have to connect the receiver's "Right Speaker Positive" terminal to the positive terminal on the right-hand speaker, and so forth.
All currently defined resources, manifolds, and distributors have exactly two sockets -- one for input and one for output. By default, they're named Input and Output, respectively. You're free to rename them (except Resources), but you can't delete these sockets or add new ones. Currently defined processes (i.e., Scripters) also start with two sockets, Input and Output, but you can add any number of either type, as well as rename them. Since you can define your own sockets for each process, it's up to you to define how they're to be used and to what they should be connected. Generally, though, if you plan to attach resources of different types to a process, it would make sense to define a different socket for each type of input or output.
From the contextual pop-up menu, bring up the Edit Sockets dialog for the Scripter. There you'll see all of your sockets and be able to edit them. You'll also see which connectors are attached to each socket and in which order (which is also editable).
TIP: If different "types" of resources are connected to a scripter and are to be used for different types of arguments on the command line then it is recommended to create new sockets for each type of connected resource, otherwise the wrong file may end up at the position of the wrong argument.
All elements (except connectors) have sockets. Let's pass through types to show what kinds of sockets can be used in appropriate items.
Resources have exactly two sockets -- one for input and one for output:
Scripters also start with two sockets, Input and Output, but you can delete either one and/or add any number of either type, as well as rename them.
Composites can have all 4 types of sockets due to all available internal elements:
Let's illustrate all possible types of sockets inside composites:
The Runner is a behind-the-scene component of Pipe-It that orchestrates the execution of the project. It checks the status of all the elements, if all the resources have associated files connected, and decides the order of execution. It manages the launching of scripter command lines to the OS and waits for their completion. It determines the complete dependencies of the elements and keeps the elements on hold till all the upstream elements are up to date. The runner is visually invisible to the end user, who will only interact through the GUI.
Pipe-Itc allows running a model from a command line without invoking the GUI at all. In such case the Pipe-It Runner is only being invoked directly.
The rules for running composites are the following:
If one wants to completely isolate a running composite from all outside elements, the easiest way to do that is to deactivate the composite itself (non-recursively). The composite can still be opened and run in that state, but all outside connections will be considered inactive. This could prove to be a useful technique for testing individual composites. Be advised that the results might differ from those obtained by running the complete project, however.
As mentioned previously, it is possible to run only parts of the full project. By navigating into a Composite, and running the project from there, only the current Composite and all nested Composites will be executed.
E.g. if the project below was executed from Composite 3-2, only composites 3-2, 3-2-1, and 3-2-2 will be executed. All other parts of the project will not.
The only exceptions to this are Resources located at a higher level, that are connected directly upstream or downstream to Processes located in the Composite being executed. The upstream Resources will be checked to determine wether the connected Processes need to run, and the downstream Resources will be marked as up-to-date once the connected Processes have completed running.
Pipe-It allows projects to be opened by single user at same time. To prevent inconsistency due to simultaneous modifications to project files, a lock file is created when a user opens a project. The lock file contains information about the user and host name, time stamp of last access (load or save) to project and PID of process.
When you try to open project being in use by another user, you will get a warning which indicates a user name and computer, so you can contact him directly to unlock the project and have possibility to open and modify it.
This section gives glimpses of the showcase projects built with Pipe-It. Both the scale (number of elements) and technical complexity and variety of the projects are impressive.
Over 11,000 Resources in this project and nearly 3500 Processes (application launches).
Master project that performs the production optimization.
Almost 1500 Resources and 750 Processes organized in 600 Composites.
Almost 20,000 elements covering each and every field of the NCS. Historical data updated automatically from NPD's Fact Pages. Decline Curve based forecast with stream management allows flexible and simplified what-if scenarios.
Pipe-It projects consists of many components. The Pipe-It Project View (*.ppv) file stores the visual content of the project. The Pipe-It Project Model (*.ppm) file is most important and stores the logical model describing all the elements in the project, their connections, their grouping into composites, the files associated with each resource, the command lines of each scripter etc. The Pipe-It Project Linkz (*.ppl) file stores all the variables collected from all text files that are part of the project, their current values and algorithms to locate them in later runs.
The location of these 3 files is called the root folder of the project. Most other files mentioned under can be located anywhere on the disk. However, to make the project portable it is recommended that all such files are located at or under the root folder.
If the project contains optimization, one or a series of Pipe-It Project Optimization (*.ppo) files are also present. They can be located anywhere.
A complete project will also contain data files that are connected to each resource. All files connected to initiating resources (those that are not downstream of any process) are also mandatory for the proper execution of the project. These files are typically located at the root or in folders under the root.
Designing a Pipe-It project requires some planning. Simple projects might contain only a handful of data files linked to resources and these can live in the project root folder. But real life projects contain hundreds if not thousands of files. It makes sense to organize these files in a sensible folder hierarchy. The project within Pipe-It should also follow the folder structure using composites and sub-composites as far as possible. This helps in cloning of composites in case a new instance of the functionality encapsulated in the composite is to be created. Pipe-It estimates the % of files that resides within a particular sub-folder of the project and offers to the user the possibility of duplicating files. This is really a powerful option as all files, their connections to resources, their Linkz variables, and all process command lines are cloned.
Making order of the hundreds and thousand files is another reason for collecting related files together.
Since version 1.5.0 connector loops are not allowed in Pipe-It projects. Any Process with downstream results connected as an upstream dependency is considered a connector loop. In many cases a connector loop can be replaced by using two resources to the same file, and connect one of them as input to the process and the other as output. If a connector loop is detected when the project is run, an error will be reported, and the project execution is halted.
In earlier versions of Pipe-It, Manifold-to-Manifold connector loops were in some cases used to force execution of parts of the project. This is no longer allowed. Instead an Always Run Element should be used to achieve this.
In process of designing projects you may find that you use similar script commands repeatedly. Usually they only differ in filenames and few other arguments. To make the script writing easier and some-what automated, rather than writing it from scratch each time, you can use Utilities available to a default installation of Pipe-It to use templates applicable to your project. Or create new ones.
See Script Assistant for more details.