FEEP IoT & Big Data Platform Sofia2 3.1 RELEASED

New Release FEEP IoT & Big Data Platform Sofia2 is now available ,this releases is available on the Experimentation Platform Sofia2 CloudLab.

(FEEP IoT & Big Data Platform Sofia2 is part of the FEEP Enablement Platform from minsait).

View releases

This release includes several features, such as:

· New Version Model on Sofia2

From this release three Sofia2 versions are available: Base Edition, IoT Edition y Advanced Analytics Edition.

Each one of them is meant to fit concrete needs:

Base Edition allows towork with platform fundamentals (ontologies, rule engines, unified console, platform connectors, real-time DB,…)

IoT Edition is designed for high data volume systems that need to analyze and publish information with ease. This version is available in the CloudLab environment for users with COLLABORATOR role.

Advanced Analytics Edition
adds real time processing and analytics over BigData. This version is only available for premium users in the Sofia2 Cloudlab.

 

  • New Sofia2 DataFlow module implementation

This version switches from
Spring XD
to
StreamSets
to make the reference implementation of the module.

The new implementation enhances visual capabilities to model data flows and pipelines.

A Pipeline represents a massive data ingestion from several origin types (HTTP, HDFS, Excel, CSV, file folders, Flume, MongoDB, Kafka,…)), an optional data transformation process,
and one or more destinations (Kafka, Sofia2 Ontologies, HDFS, FRP, HIVE, Kudu, Hbase,…)

Next image shows a Pipeline ingesting data from a file folder, several processing modules, and a final data storage into Hadoop file system (HDFS):

The module allows to run processes in PREVIEW mode to visualize data flowing through each module:

Complete statistics of current operations are also available:

Sofia2 enhances Streamsets capabilities allowing to run Pipelines in 3 modes: Streaming (continuous execution), Batch (schedulled execution that stops when job is done or timeout
is reached) and instant execution.

Custom components have been developed such as Excel data processing (with custom mappings), and Sofia2 Origin and Destination modules.

These modules allow to load information from Sofia2 based on a query/subscription and persist data into Sofia2 ontologies

· BigBox Integration

Sofia2 integrates with BigBox, Indra’s bigdata solution for telecommunications market. BigBox meets the new analytical requirements such as handling of enormous volumes of information and inclusion of unstructured data. In this new integration Sofia2 provides features such as real-time processing, visualization and integration of external data sources, RRSS, analytical capabilities …

· Invocation of REST APIst & load on Ontologies

This functionality allows configure and Schedule URLs and APIs (both internal and external) and define how this results mapped to Ontologies.

This way Sofia2 is able to act as a aggregator and normalizing all types of APIs platform, in turn publishing that information collected as standardized APIs from the API Manager.

The functionality allows users to create groups CONTRIBUTOR Role APIs that can be activated and plan together.

Once created a group of APIS I can asign APIs to the group. The APIs can be of type URL or API Manager:

Cuando defino un API de tipo URL simplemente definiré la URL completa y mapearé la información obtenida a una Ontología.

En el caso de invocaciones a APIs del API Manager seleccionaré el API a usar, me aparecerán los parámetros que tiene ese API y haré el mapeo con la ontología:

For invocations to the API Manager I will select the API to use, then appears the parameters of this API and I´lI do with the ontology mapping:

Una vez tengo planificada la invocación a la URL o API y los datos se cargan en una ontología podré publicar esta información en el API Manager completando el ciclo de normalización y publicación.

Once I have planned the invocation to the URL or API and the data is loaded in one ontology I can publish this information in the API Manager ,completing the cycle of normalization and publication.

Associated with this functionality Yahoo Finance and Yahoo Weather APIs have been created in the Manager API to invoke easily (it is only necessary to report parameters).

· Big Data Tools:

A new set of tools have been added in order to simplify the bigdata infrastructure management. These tools will be available for users with ANALYTICS role.

File uploading to Staging area

Any user with analytics role will be able to upload files to his personal staging area in Sofia2. This area is implemented by default in HDFS. Files uploaded from this tool will be available to to be operated from DataFlow and Notebook modules.

This functionality allows to upload different file types, including compressed files (.zip & tar.gz) that will be automatically extracted once uploaded. Multiple file uploading is also available.

Once files are uploaded the HDFS path will be shown:

Hive/Impala table creation from HDFS file

Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. Nowadays it’s been assumed as standard in storaging, SQL querying, and analysis.

This feature allows to create tables from staging area files attributes, allowing users to define data mappings as desired:

Once mappings are defined the HIVE table creation script is generated to be executed:

Ontology creation from HIVE tables

Sofia2 uses the Ontology concept, decoupling data models from the underlying technology (i.e MongoDB documents, relational databases tables or Hive tables)

This feature allows to create an Ontology from a Hive table, allowing user to define all attributes (name, config, schema, …).

Once data is converted to ontology all platform features would be available: querying, analytics, dashboards, APIs

· Initial versión of the Sofia2 Monitoring module

This release has been included a basic monitoring functionality which is going to complete in upcoming releases.

This functionality allows to the Platform Administrator to monitor different areas like, for example, the ontology size or the ontologies not migrated to the Historical Data Base:

The module indicates in a visual form the anomaled values:

· Initial versión of the Sofia2 Monitoring module

This release has been included a basic monitoring functionality which is going to complete in upcoming releases.

This functionality allows to the Platform Administrator to monitor different areas like, for example, the ontology size or the ontologies not migrated to the Historical Data Base:

The module indicates in a visual form the anomaled values:

· Guided creating indexes in RTDB:

This functionality allows to do a complete management of the index on the DB in real time, without knowing the command for creating the index.

The Platform allows the owner of the ontology to create simple and compounds indexes on the ontology:

Once elected the ontology on which we want to create the index, the fields of the ontology are shown, and also the fields with an index previously created.

In the CloudLab environment it has limited the number of indexes that can be created by ontology.

· New authentication mechanism for Project Web Sofia2

With the aim of enabling Sofia2 Web Projects deployed on the platform to use users and roles of the platform, it has been enabled a RESTFul service that allows external systems login and authorizing on the platform

At CloudLab https://sofia2.com/console/api/rest/autentication

This mechanism can also be used from external systems acting in that case as a simple SSO.

In addition to the RESTful service a Javascript API has been created, this API automates the call to these services exposing 3 operations:

function authenticate(url, page)

function authorize(page)

function logout(page)

In a next post we will explain this functionality with an example.

· New Demos and PoV:

In the scope of this release new demos have been created:

O

If you are interested in doing a PoC with us you can contact in our mail.

· Improvements in the Rules Script engine debugging capabilities:

A TEST button is added to the Script Editor, it runs the script and displays traces generated by the script.

· Various improvements in stability, performance and usability of the Platform

FEEP IoT & Big Data Platform Sofia2 3.1 RELEASED

Responder

Introduce tus datos o haz clic en un icono para iniciar sesión:

Logo de WordPress.com

Estás comentando usando tu cuenta de WordPress.com. Cerrar sesión / Cambiar )

Imagen de Twitter

Estás comentando usando tu cuenta de Twitter. Cerrar sesión / Cambiar )

Foto de Facebook

Estás comentando usando tu cuenta de Facebook. Cerrar sesión / Cambiar )

Google+ photo

Estás comentando usando tu cuenta de Google+. Cerrar sesión / Cambiar )

Conectando a %s