Big Data Business Intelligence for Govt. Agencies Training Course

Each session lasts 2 hours

Day 1: Session 1: Business Overview of Why Big Data Business Intelligence in Government

• Case studies from the NIH and Department of Energy
• The adoption rate of Big Data in government agencies and how they are aligning future operations around Big Data Predictive Analytics
• Broad-scale application areas in the Department of Defense, NSA, IRS, USDA, and others
• Integrating Big Data with legacy data systems
• Basic understanding of enabling technologies in predictive analytics
• Data integration and dashboard visualization
• Fraud management
• Business rule and fraud detection generation
• Threat detection and profiling
• Cost-benefit analysis for Big Data implementation

Day 1: Session 2: Introduction to Big Data - Part 1

• Key characteristics of Big Data: volume, variety, velocity, and veracity. MPP architecture for handling volume.
• Data Warehouses – static schemas and slowly evolving datasets
• MPP Databases such as Greenplum, Exadata, Teradata, Netezza, and Vertica
• Hadoop-Based Solutions – no conditions imposed on the structure of the dataset
• Typical pattern: HDFS, MapReduce (crunch), retrieval from HDFS
• Batch processing – suited for analytical/non-interactive tasks
• Volume: CEP streaming data
• Typical choices – CEP products (e.g., Infostreams, Apama, MarkLogic)
• Less production-ready options – Storm/S4
• NoSQL Databases (columnar and key-value): Best suited as an analytical adjunct to data warehouses/databases

Day 1: Session 3: Introduction to Big Data - Part 2

NoSQL Solutions

• KV Store: Keyspace, Flare, SchemaFree, RAMCloud, Oracle NoSQL Database (OnDB)
• KV Store: Dynamo, Voldemort, Dynomite, SubRecord, Mo8onDb, DovetailDB
• KV Store (Hierarchical): GT.m, Cache
• KV Store (Ordered): TokyoTyrant, Lightcloud, NMDB, Luxio, MemcacheDB, Actord
• KV Cache: Memcached, Repcached, Coherence, Infinispan, EXtremeScale, JBossCache, Velocity, Terracoqua
• Tuple Store: Gigaspaces, Coord, Apache River
• Object Database: ZopeDB, DB40, Shoal
• Document Store: CouchDB, Cloudant, Couchbase, MongoDB, Jackrabbit, XML-Databases, ThruDB, CloudKit, Prsevere, Riak-Basho, Scalaris
• Wide Columnar Store: BigTable, HBase, Apache Cassandra, Hypertable, KAI, OpenNeptune, Qbase, KDI

Varieties of Data: Introduction to Data Cleaning Issues in Big Data

• RDBMS – static structure/schema, does not promote an agile, exploratory environment.
• NoSQL – semi-structured, sufficient structure to store data without defining an exact schema beforehand
• Data cleaning challenges

Day 1: Session 4: Big Data Introduction - Part 3: Hadoop

• When to select Hadoop?
• STRUCTURED – Enterprise data warehouses/databases can store massive data (at a cost) but impose structure (not ideal for active exploration)
• SEMI-STRUCTURED data – difficult to manage with traditional solutions (DW/DB)
• Warehousing data requires significant effort and remains static even after implementation
• For variety and volume of data processed on commodity hardware – HADOOP
• Commodity hardware is needed to create a Hadoop Cluster

Introduction to Map Reduce /HDFS

• MapReduce – distribute computing across multiple servers
• HDFS – make data available locally for the computing process (with redundancy)
• Data – can be unstructured/schema-less (unlike RDBMS)
• Developer responsibility to derive meaning from the data
• Programming MapReduce = working with Java (pros/cons), manually loading data into HDFS

Day 2: Session 1: Big Data Ecosystem - Building Big Data ETL: The Universe of Big Data Tools - Which one to use and when?

• Hadoop vs. other NoSQL solutions
• For interactive, random access to data
• Hbase (column-oriented database) on top of Hadoop
• Random access to data with restrictions imposed (max 1 PB)
• Not ideal for ad-hoc analytics, but suitable for logging, counting, and time-series data
• Sqoop – Import from databases to Hive or HDFS (JDBC/ODBC access)
• Flume – Stream data (e.g., log data) into HDFS

Day 2: Session 2: Big Data Management Systems

• Moving parts, compute node start/failure: ZooKeeper - For configuration/coordination/naming services
• Complex pipeline/workflow: Oozie – manage workflow, dependencies, and daisy chains
• Deploy, configure, cluster management, upgrades, etc. (sys admin): Ambari
• In Cloud: Whirr

Day 2: Session 3: Predictive Analytics in Business Intelligence - Part 1: Fundamental Techniques & Machine Learning based BI

• Introduction to Machine Learning
• Learning classification techniques
• Bayesian Prediction – preparing the training file
• Support Vector Machine
• KNN p-Tree Algebra & vertical mining
• Neural Networks
• Big Data large variable problem – Random Forest (RF)
• Big Data Automation problem – Multi-model ensemble RF
• Automation through Soft10-M
• Text analytic tool – Treeminer
• Agile learning
• Agent-based learning
• Distributed learning
• Introduction to Open Source Tools for predictive analytics: R, Rapidminer, Mahut

Day 2: Session 4: Predictive Analytics Ecosystem - Part 2: Common Predictive Analytics Problems in Government

• Insight analytics
• Visualization analytics
• Structured predictive analytics
• Unstructured predictive analytics
• Threat/fraud/vendor profiling
• Recommendation Engines
• Pattern detection
• Rule/Scenario discovery – failure, fraud, optimization
• Root cause discovery
• Sentiment analysis
• CRM analytics
• Network analytics
• Text Analytics
• Technology-assisted review
• Fraud analytics
• Real-Time Analytics

Day 3: Session 1: Real-Time and Scalable Analytics Over Hadoop

• Why common analytics algorithms fail in Hadoop/HDFS
• Apache Hama – for Bulk Synchronous distributed computing
• Apache SPARK – for cluster computing for real-time analytics
• CMU Graphics Lab 2 – Graph-based asynchronous approach to distributed computing
• KNN p-Algebra based approach from Treeminer for reduced hardware cost of operation

Day 3: Session 2: Tools for eDiscovery and Forensics

• eDiscovery over Big Data vs. Legacy data – a comparison of cost and performance
• Predictive coding and technology-assisted review (TAR)
• Live demo of a TAR product (vMiner) to understand how TAR works for faster discovery
• Faster indexing through HDFS – velocity of data
• NLP or Natural Language processing – various techniques and open source products
• eDiscovery in foreign languages – technology for foreign language processing

Day 3: Session 3: Big Data BI for Cyber Security – Understanding the whole 360-degree view of speedy data collection to threat identification

• Understanding the basics of security analytics – attack surface, security misconfiguration, host defenses
• Network infrastructure / Large data pipe / Response ETL for real-time analytics
• Prescriptive vs. predictive – Fixed rule-based vs. auto-discovery of threat rules from Metadata

Day 3: Session 4: Big Data in USDA : Application in Agriculture

• Introduction to IoT (Internet of Things) for agriculture – sensor-based Big Data and control
• Introduction to Satellite imaging and its application in agriculture
• Integrating sensor and image data for soil fertility, cultivation recommendations, and forecasting
• Agriculture insurance and Big Data
• Crop Loss forecasting

Day 4: Session 1: Fraud prevention BI from Big Data in Government – Fraud Analytics:

• Basic classification of Fraud analytics – rule-based vs. predictive analytics
• Supervised vs. unsupervised Machine Learning for Fraud pattern detection
• Vendor fraud/overcharging for projects
• Medicare and Medicaid fraud – fraud detection techniques for claim processing
• Travel reimbursement frauds
• IRS refund frauds
• Case studies and live demos will be provided wherever data is available.

Day 4: Session 2: Social Media Analytics – Intelligence gathering and analysis

• Big Data ETL API for extracting social media data
• Text, image, metadata, and video
• Sentiment analysis from social media feeds
• Contextual and non-contextual filtering of social media feeds
• Social Media Dashboard to integrate diverse social media platforms
• Automated profiling of social media profiles
• Live demo of each analytics module will be conducted using the Treeminer Tool.

Day 4: Session 3: Big Data Analytics in image processing and video feeds

• Image Storage techniques in Big Data – Storage solution for data exceeding petabytes
• LTFS and LTO
• GPFS-LTFS (Layered storage solution for Big image data)
• Fundamentals of image analytics
• Object recognition
• Image segmentation
• Motion tracking
• 3-D image reconstruction

Day 4: Session 4: Big Data applications in NIH:

• Emerging areas of Bio-informatics
• Meta-genomics and Big Data mining issues
• Big Data Predictive analytics for Pharmacogenomics, Metabolomics, and Proteomics
• Big Data in downstream Genomics processes
• Application of Big Data predictive analytics in Public Health

Big Data Dashboard for quick accessibility of diverse data and display:

• Integration of existing application platforms with Big Data Dashboard
• Big Data management
• Case Study of Big Data Dashboard: Tableau and Pentaho
• Use Big Data apps to push location-based services in Government
• Tracking system and management

Day 5: Session 1: How to justify Big Data BI implementation within an organization:

• Defining ROI for Big Data implementation
• Case studies for saving Analyst Time for collection and preparation of Data – increase in productivity gain
• Case studies of revenue gain from saving licensed database costs
• Revenue gain from location-based services
• Savings from fraud prevention
• An integrated spreadsheet approach to calculate approximate expense vs. Revenue gain/savings from Big Data implementation.

Day 5: Session 2: Step-by-step procedure to replace legacy data systems with Big Data Systems:

• Understanding a practical Big Data Migration Roadmap
• Key information needed before architecting a Big Data implementation
• Methods for calculating the volume, velocity, variety, and veracity of data
• How to estimate data growth
• Case studies

Day 5: Session 4: Review of Big Data Vendors and their products. Q/A session:

• Accenture
• APTEAN (Formerly CDC Software)
• Cisco Systems
• Cloudera
• Dell
• EMC
• GoodData Corporation
• Guavus
• Hitachi Data Systems
• Hortonworks
• HP
• IBM
• Informatica
• Intel
• Jaspersoft
• Microsoft
• MongoDB (Formerly 10Gen)
• MU Sigma
• Netapp
• Opera Solutions
• Oracle
• Pentaho
• Platfora
• Qliktech
• Quantum
• Rackspace
• Revolution Analytics
• Salesforce
• SAP
• SAS Institute
• Sisense
• Software AG/Terracotta
• Soft10 Automation
• Splunk
• Sqrrl
• Supermicro
• Tableau Software
• Teradata
• Think Big Analytics
• Tidemark Systems
• Treeminer
• VMware (Part of EMC)