This new version adds the following features to the platform:
· New Sofia2-DataFlow module
This module extends the Big Data capabilities of the Platform making it a genuine Internet of Everything Platform.
This component allows massive ingestion of data from any source (file, database, mailings, Social Networks, JMS, Kafka, FTP, …), online transformations of information (filtering, parsing, forks, joins, cloning, …) and routed to another destination (Sofia2, Hadoop, Data Base, …). The Platform can incorporate plugins to add new sources, transformations, and destinations.
The data flow is built visually from the Sofia2 Control Panel (Sofia2 Console) in the new dataflow menu:
Data Flow Engine supports Streams and Jobs, the Streams represent intakes in real time (eg Syslog events) while Jobs represent scheduled Batch processes (loads from a file to a database).
A new chapter has been added in the Web Console Guide to explain the operation of this module.
Here you can see an example of a Stream formed by the source File and the Log destination, two of many modules that can be found in the component:
In addition to creating flows, the module allows to create online displays that show the status of the process:
· Crawling Web
New Engine and Web crawling capabilities have been added in this Sofia2 release to configure and automate information addition to the platform from web sites, portals or forums.
From Sofia2’s Control Panel, this crawling can be configured:
The engine is based on fully configurable patterns, allowing to choose between different scans and patterns over the elements of a Web, among these:
– Own HTML Web
– Extracted text
– Output Links
– Web Metadata
– Web Title
Binary files Metadata exploration is also allowed. For example, to access to Images Metadata.
Another important feature is the ability to store the gathered Web in an instance of a given ontology allowing ETL processing capabilities to perform even more potent operations over the stored data.
· Gadgets with access to HDB and RTDB in native language
This version allows gadgets to consult the Historical Database (HDB), Hadoop in the Reference Implementation:
and to consult in BDTR through the native language (MongoDB in reference implementation).
Another improvement is multifield access processing on fields, enabling more potent logic on Gadgets.
· Sofia2Scriptrunning from the Control Panel (Sofia2 Console)
In this new version from the console we can execute timed scripts to verify correct operation.
For this, a new button has been created in the Options section of the list of Scripts:
This option is available for TIMER scripts, selecting the button they can be executed:
The status of the execution will be reported in a panel, which helps to debug scripts:
· New API Manager Life cycle
API Manager’s life cycle adds a new state: “in development”.
This is the upgraded life cycle diagram:
APIs in the new state will be able to be modified by the owner, but they would be able to be invoked as if they were published.
With the addition of the new state developers can keep modifying their APIs at the same time they can be consumed by other users.
· Throttling Control in APIs:
When defining APIS it’s now possible to control the number of requests per unit of time they will be able to support. Once limit is reached new requests will fail returning an error code 423.
After unit of time passed, requests will be allowed to the user:
This feature allows Freemium API models in the MarketPlace.
· Sofia2 Dashboards visual improvements
Sofia2 Dashboard keeps evolving pursuing a major quality in created applications:
· APIs Cloning.
To simplify the creation of new API versions, now APIs can be cloned. This feature allows to generate a new version copying another existing one (independently of the current state). This can be accessed through the clone button in the API management screen.
A new API is generated with the properties and operations from the original one. It will be found in a “created” state to allow further modifications.
· API Script to replicate information between different Sofia2 instances:
It has been created a new API Script APIReplicationRest, this API allows to forward information (ontologies instances) coming to a Sofia2 instance to another Sofia2 instance.
As a example, with this functionality you can share information between a City Platform and a Hospital installation of Sofia2.
It has been created 2 methods to invoke:
The syntax used is as follows:
apireplicationrest.insertInstance(SERVICE_URL, TOKEN, KP_INSTANCE,ONTOLOGY_NAME,ONTOLOGY_INSTANCE);
· Bugs fixing and usability enhancements