H13-723_V2.0 HCIP-Big Data Developer V2.0 Questions and Answers

Real-time online data analysis.

Data mining, such as user behavior analysis, interest division, regional display.

Data summary, such as daily and weekly user clicks, click ranking.

Non-real-time analysis, such as log analysis, statistical analysis.

HDFS

HBase

Hive

Flink

mouth viewOozieofJava APIWhether the returned result is as expected

mouth throughHueofworkflowDashboard to view program running results

mouth throughHue ' s file browser to see ifHDFSThe specified directory produces the expected file

mouthOozieself-containedWebThe interface can also view the job execution results

True

False

twoworkerone of theworkerdistribute two

twoworkerone for each

no fixed rules

twoworkerBoth are distributed

filer(conditionExpr:String):DataFrameFilter data based on filter criteria

Join(right:DataFrame,joinExprs:Column):DataFrameaccording to the specified expressionjoin

intersect(other:DataFrame):DataFrameright2indivualdataframeTake the intersection

dropDuplicates():DataFramedata deduplication

True

False

X is more performant than Y

X is less performant than Y

X and Y are the same as performance

Uncertainty about the performance difference between X and Y

Active and standbyNameNodeThe disk space where the metadata is located is insufficient.

The number of missing blocks exceeds the read value.

Lost copies exceed read value.

The corrupted copy exceeds the threshold.

True

False

Memory Channel

File Channel

JDB, C Channel

HDFS Channel

Key

Value

timeStamp

KeyType

user definedUDFBefore being used, it needs to beHivecreate this in the systemUDF.

user definedUDF, information such as summary and status is not allowed.

user definedUDF, can be added according to the actual situationdeterministicandstatefulletc. Annotation

Under the secure cluster, user-definedUDF, it is recommended to create it once before each use.

True

False

enabledAckerLater,StreamingIt will identify the failure to sendTupleAnd automatically resend, no need for human intervention pre.

AckerThe message that will identify the processing timeout or processing failure isfail.

fromspoutstarted, formedTupleFailure of any link in the tree will mark the entire tree as failed.

The application needs to be inspoutoffail() interface method to implement message resend logic.

pre-scoreregion,MakeregionEvenly distributed, improve concurrency

Query frequent attributes are placed inrowkeyThe front part

Use secondary indexes to adapt to more query scenarios

Data read at the same time are stored in the samecellmiddle

list

set

hash

sorted set

True

False

SchemaSimilar to the table structure in a relational database, the structure used to define index data mainly includes three

part:uniquekye,FieldandFieldType

For fields that do not need to store the original value, their propertiesstoredcan be set tofalse

for the samedocumentHow many values of a field of the

multiValuedmust be set totrue

copyFieldThe following definitions can be made: < field name= " cat " type= " string " indexed= " true "

stored= " true " docValues= " true " multiValued= " false " / >

< field name= " dog " type= " string " indexed= " true " stored= " true "

docValues= " true " multiValued= " false " / >

< field name= " text " type= " text_general " indexed= " true " stored= " false "

multiValued= " false " / >

< copyFie

JDB, C

spark-beeline

spark-sql

twoworkerone of theworkerdistribute two

twoworkerone for each

no fixed rules

twoworkerBoth are distributed

HDFS

HBase

Kafka

Redis

storage

Resource management

search

data transmission

suspend

getJoblnfo

submit

run

True

False

Partition Table

bucket table

Tilt table

partition+bucket table

storm jar example jar wcTest om huawei example WordCount

storm jar example jar wcTest WrodCount

storm jar example jar om huawei example WordCount wcTest

storm jar example jar WorkCount wcTest

HIVE

HBASE

HDFS

DB

mapandcombine

reduceandcombine

combineandsort

mapandreduce

In a network isolation environment,ZooKeeperofIPCan be configured as a businessIP, can also be configured to manageIP.

HiveUnlike traditional databases, soJDB, CThe established connection does not need to be closed.

In a secure cluster environment, establish andHiveBefore connecting, you need to use unified authentication to log in.

useJDB, CsubmitSQL, you canSQLA semicolon " ; " is added after the statement.

Nimbus

Supervisor

Broker

quorumpeer

eachDataNodemaximum storage1copies.

Support multiple users to write to the same file at the same time.

The first copy of the data block is placed preferentially on the node where the client writing the data block is located.

Copied file blocks all exist on the same rack by default.

C

Scala

Java

Python

Create an index on solr, save the data, and return all the data during full-text search

Store data on HBase, and satisfy fuzzy matching queries through HBase ' s filtering feature

When storing data, build an index on solr, and store complete data in HBase: query full text through solr

Retrieve the key information of the record, and obtain the complete record on HBase through the key information

NodeManager

container

ResourceManager

AppMaster

Versions

TTL

Coprocessor

Compression

AssumeSmapinput compression method

set upmapHow the output is delivered

set upmapoutput compression method

set upreduceoutput compression method

True

False

1

2

3

4

in the cluster1Create the account required for this application on theHBaseServe

in the cluster2created onAThe account with the same name in the options, and set this account to be able to access the cluster ' sHiveServe

Aoptions andBThe account created by the option must belong tosupergroupGroup

cluster1and cluster2Complete the mutual trust operation

True

False

ProducerResponsible for production data,Consumerresponsible for consuming data,ProducerandConsumerneed between EstablishSocketconnect

Producercan either send data toBroker, and can send data toConsumersuperiorProducer

As a message producer, you can directly write data toZooKeeper

mouthProducerconnect any normalBrokerInstances can produce data

Streaming data monitoring

Real-time visit statistics of the website

Offline log analysis

Traffic flow analysis

BufferedOutputStream.write

BufferedOutputStream.flush

FileSystem.create

FileSystem.append

When the client initiates a scan request, it willscan cachingset too large

Initiate by client-specified methodFlushdo

RegionServerTotal memory exceeds threshold

RegionofMemStoreset over threshold

OneTopologyis made up of a group ofSpoutcomponents andBoltcomponents passStream GroupingsA connected directed acyclic graph

(DAG)

Topologywill run until it is explicitlykill

Business logic is encapsulatedTopologymiddle

OneTopologyOnly one can be specified to startWorkerprocess

available fromHDFSdirectly into thehivesurface

available fromHiveServerThe local hard disk of the node is directly loaded into thehivesurface

It can be directly loaded from the local hard disk of the node where the client is located.hivesurface

HiveThe data loading process will not parse the specific file content, mainly the process of file transfer

taildir

Log4j

JMS

Thrift

low latency

Rich data structures

Support data persistence

supportSQLquery interface

Attribute values with high access weight are placed inRowKeyFront

Attribute values with high access weight are placed inRowkeysecond half

Attribute values with good dispersion are placed inRowkeyfirst half

Attribute values with good dispersion are placed inRowkeysecond half

The client caches the topology information of the cluster

The server caches the topology information of the cluster

both are

more than two

SolrUser group users can create configuration sets and createCollectionwhen specifying the configuration set

SolrUsers of the user group can createCollection

Collectioncreator, theCollectionroles with read and write permissions andSolr adminrole can beCollectionEnter row update

onlyCollectioncreator andSolr adminRoles can be deletedCollection

Avro

HBase

Kafka

HTTP

Producer

Broker

Consumer

ZooKeeper

True

False

clickClouddownTree,CheckSolr CloudmiddleCollections, a configuration set, andlive_nodesand other metadata information

clickCloud, see eachCollectionunder eachShardReplicadistribution and status

enter aCollectionofShard ReplicaofCore Overviewinterface, you can view theReplicaIndex the actual document

Number of bars, storage size and location information

Solruser group,Solr adminrole andSuper groupof users have access toSolr Admin UI

Kafka-topics.sh–describe

Kafka-topics.sh–list

kafka-topics.sh–alter

kafka-topics.sh–create

Generate files directly according to the source file name

Set the file generation time interval, close the old file when the time interval is reached, and the new data will be written to the new file

Set the size of a single file, when the file size reaches the set value, the old file will be closed, and the new data will be written to the new file

Set up single file storageeventquantity, when the file is storedeventWhen the number reaches the set value, the old file will be closed, and the new data will be

is written to a new file

True

False

NodeManager

container

ResourceManager

AppMaster

True

False

INCR(aThe stored digital value is incremented by one)(DECR)

HOY(eesurfacekeydomain infieldvalue plus incrementincrement)

UURH(MRnpuhoperate)

All of the above

True

False

FusionInsight ManagerofWebinterface

Loader Webinterface

Solr Webinterface

Yarn Webinterface

supportCSVandJSON

supportCSV,JSONandHTML

supportCSV,JSONandXML

XCompareYhigher performance

XCompareYlower performance

XandYsame as performance

UncertaintyXandYdifference in performance

SolrUser group users can create configuration sets and createCollectionwhen specifying the configuration set

SolrUsers of the user group can createCollection

Collectioncreator, theCollectionroles with read and write permissions andSolr adminrole can beCollectionEnter

row update

onlyCollectioncreator andSolr adminRoles can be deletedCollection

user definedUDFBefore being used, it needs to beHivecreate this in the systemUDonly

user definedUDF, information such as summary and status is not allowed.

user definedUDF, can be added according to the actual situationdeterministicandstatefulletc. Notes.

Under the secure cluster, user-definedUDF, it is recommended to create it once before each use.

Producer

Broker

Consumer

ZooKeeper

storm jar example jar wcTest om huawei example WordCount

storm jar example jar wcTest WrodCount

storm jar example jar om huawei example WordCount wcTest

storm jar example jar WorkCount wcTest

True

False

True

False

True

False

HBase

Hive

Spark

Redis

The disk configuration (such as the number of disks, disk size, etc.) used to store Kafka data cannot meet the current industry requirements.

service data traffic, causing the disk usage to reach the upper limit

The data storage time is configured too long, and the accumulated data reaches the upper limit of the disk usage.

Unreasonable business planning results in uneven data distribution and some disks reach the upper limit of usage

Broker node failure caused

True

False

INCR(increment the stored digital value by one)

HINCRBY(for the hash tablekeydomain infieldvalue plus incrementincrement)

LPUSH(for linked listspushoperate)

All of the above

Partition fields are defined when the table is created

There can only be one partition field, and multi-level partitions cannot be created

Using partitions can reduce the data scan range of some queries, thereby improving query efficiency

Partition fields can be used aswherecondition of the sentence

sc.textFile( " /data/file/path”).map(.split( " , " )).map(p= > FemaleInfo(p

(O),p(1),p(2).trim.toInt)).toDF.registerTemp Table( " FemaleinfoTable”)sqlContext.sql

( " select name,sum(stay Time)as stay Time from FemaleInfoTable where gender=

female ' group by name”).filter( " stay Time > =120”).collect).foreach(printin)

sc.textFile( " /data/file/path " ).map(.split( " , " )).map(p= > FemaleInfo

(p(O),p(1),p(2).trim.toInt)).toDF.registerTemp Table( " Female infoTable " )

sqlContext.sq(l " select name, sum(stay Time)as stay Time from FemaleInfoTable where

gender=female).filter(“stay Time > =120”).collect).foreach(printin)

val text=sc.textFile( " /data/fle/path " )

val data=text.filter(_.contains( " female " ))

Val fermaleData:RDD(String,Int) =data.map{line= > val t=line.split( ' , ' )

(t(0),t(2).toInt)

}.reduceByKey(_+_)

Val result=femaleData.filter(line= > line.2 > 120)

Rusult.collect.map(x= > x._1 + ' , ' + x._2).foreach(println)

val text=sc.textFile( " /data/fle/path”)

val data=text.filter(_.contains( " female”))

Val fermaleData:RDD(String,Int) =data.map(line= > val t=line.split( ' , ' )

(t(0),t(2).toInt)

}

Val result=femaleData.filter(line= > line.2 > 120)

Rusult.collect.map(x= > x._1 + ' , ' + x._2).foreach(println)

(1) data fromsourcearrivechannel;(2)sinkfromchannelfetch data

(1) data fromchannelarrivesource;(2)sinkfromsourcefetch data.

(1) data fromsinkarrivechannel;(2)sourcefromchannelfetch datao

(1) data fromchannelarrivesource;(2)sinkfromsourcefetch data

Produceris a message producer

ProducerProduction data needs to be specifiedTopic

Can play multiple at the same timeProducerprocess to the sameTopicsend data

ProducerNeed to connect first when producing dataZooKeeper, and then connectBroker

Give this user accountglobelread permission

Give this user accountNamespace1read and write permissions

Give this user accountTable1andTable2read and write permissions

Give wave user accountNamespace1andNamespace2read and write permissions,

Spark

Streaming

MapReduce

HDFS

True

False

Table Name

table names and columns

Table names and column families

can be empty

Krb5.conf

TGT

ST

jaas.conf

When the client initiates a scan request, it willscan cachingset too large

Initiate by client-specified methodFlushdo

RegionServerTotal memory exceeds threshold

RegionofMemStoreset over threshold

name-node

source

mapred.mapper.class

job-tracker

lines.flatMap(_.split??)).map(word= > (word,1)).reduce(_+_).print ( )

lines.flatMap(_.split??)).map(word= > (word,word.length())).reduceByKey(_+_).print ( )

lines.flatMap(_.split??)).map(word= > (word,1)).reduceByKey(_+_).print ( )

lines.flatMap(_.split??)).flatmap(word= > (word,1)).groupByKey(_+_).print ( )

easy to program

good scalability

real-time computing

High fault tolerance

Support based onRESTquery request

WebHCatThe return data format isSML

WebHCatbased onHTTPandHTTPSAgreement to provide services to the outside world

able to passWebHCatCreate tables, query, etc.

HDFS

HBase

Kafka

Redis

filter (condifionExpr: String): DataFrame

join (right: DataFrame, join Exprs: Column): DataFrame

intersect (other: Data Frame): Data Frame

dropDuplicates (): DataFrame

Build queuing systems, such as message queues

uniqOperations, such as getting the ranking value of all data in a certain period of time

get latestNdata operations: for example, for a certain microblog, get the latest10comments

Simulate stack operations

True

False

to emit

execute

open

nextTuple

HDfS

HBase

Kafka

Redis

index name

index column

index column type

The name of the column family to which the indexed column belongs

mapandcombine

reduceandcombine

combineandsort

mapandreduce

Versions

TTL

Coprocessor

Compression

Table Name

table names and columns

Table names and column families

can be empty

True

False

in the cluster1Create the account required for this application on theHBaseServe

in the cluster2created onAThe account with the same name in the options, and set this account to be able to access the cluster ' sHiveServe

Aoptions andBThe account created by the option must belong tosupergroupGroup

cluster1and cluster2Complete the mutual trust operation

existSolrCreate an index and save the data, and return all the data during full-text search.

existHBasestore data on theHBaseThe filtering characteristics of , satisfy fuzzy matching query.

When storing data, inSolrindex on theHBaseStore complete data? ?when, throughSolrFull-text search to obtain and record

record key information, through the key information inHBasefor the full record.

Supports distributed and streaming acquisition

with the help ofMapReducetask parallel collection

Only structured data can be collected

Trigger data collection with external timing tasks

5

10

20

30

mergemapThe result of the stage has the samekeyvalue key-value pair

put allmapThe results of the stages are merged to get the final result

rightmapThe results of the stages are sorted and merged

Split the input data into individualsplit, and willsplitfurther split into < key,value > yes, then asmapletter input of numbers

Bucket table, bucket according to day

Partition table, partitioned by day

skew table, skew field select date, skew value random

partition+Bucket table, partitioned by time, each partition2560buckets

hdfs fsck /test.txt–files

hdfs fsck /text.txt–locations

hdfs fsck /test.txt–blocks

hdfs fsck /test.txt–list–corruptfileblocks

Partition Table

bucket table

Tilt table

partition+bucket table

True

False

Krb5.conf

TGT

ST

jaas.conf

name ten mobile number

date ten phone number

Phone number+date

Phone number+Name

rval counts = textFile.map(line = > line.split( " " )).map(word = > (word,

1)).reduceByKey(+)

val counts = textFile.flatMap(line = > line.split( " )).map(word = > (word,

1)).reduceByKey(+)

vrval counts = textFie.map(line = > line.split( " )).map(word = > (word,

1)).groupByKey( )

vrval counts = textFile.flatMap(line = > line.split( " " )).map(word = > (word,

1)).groupByKey( )

In a network isolation environment,ZooKeeperofIPCan be configured as a businessIP, can also be configured to manageIP.

hiveUnlike traditional databases, soJDB, CThe established connection does not need to be closed.

In a secure cluster environment, establish andHiveBefore connecting, you need to use unified authentication to log in.

useJDB, CsubmitSQL, you canSQLPut a semicolon after the statement " ; "

HFile

HLog

MemStore

HMaster

The port preferentially downloads data from the nearest DataNode.

The port file data is first returned from the DataNode to the NameNode, and then downloaded from the NameNode to the client.

If the client and the connected DataNode fail while reading, the client will abandon the failure

node, to connect to the node where its replica is located.

The port supports multiple clients reading the same file data from the DataNode at the same time.

True

False

Easy to program

Good scalability

Real-time computing

High fault tolerance

C

Scala

Java

Python

HIVE

HBASE

HDFS

DB

Batch Iterative

Computing Stream

Processing

Data Storage

True

False

1

2

6

10

broadcast variable in eachTasksave a copy

broadcast variable in eachExecutorsave a copy

Broadcast variables are read-only

broadcast variable viaSparkContext.broadcast) method to generate

Attribute values with high access weight are placed inRowKeyFront

Attribute values with high access weight are placed inRowkeysecond half

Attribute values with good dispersion are placed inRowkeyfirst half

Attribute values with good dispersion are placed inRowkeysecond half

True

False

True

False

Table Name

Table names and column families(or table name andfamily)

table names and columns

can be empty

clickClouddownTree,CheckSolr CloudmiddleCollections, a configuration set, andlive_nodesequivalence information

clickCloud, see eachCollectionunder eachShardofReplicadistribution and status

enter aCollectionofShard ReplicaofCore Overviewinterface, you can view theRepicaSow

Quoting the actual number of documents, storage size and location information

Solruser group,Solr adminrole andSuper groupof users have access toSolr Admin UI

True

False

Spark

Streaming

MapReduce

HDFS

True

False

hdfs fsck/test.txt-files

hdfs fsck/test.txt-locations

hdfs fsck/test.txt-blocks

hdfs fsck/test.txt-list-corruptfileblocks

SNMP

VPN

BGP

syslog

taildir

Log4j

JMS

Thrift

True

False

used to filter data

Used to optimize random performance

will increase storage consumption

Can accurately determine that a certain data does not exist

mergemapThe result of the stage has the samekeyvalue key-value pair

put allmapThe results of the stages are merged to get the final result

rightmapThe results of the stages are sorted and merged

Divide the input data into one by one. and split this further into pairs, then asmapfunction input of

Partition fields are defined when the table is created

There can only be one partition field, and multi-level partitions cannot be created

Using partitions can reduce the data scan range of some queries, thereby improving query efficiency

Partition fields can be used aswherecondition of the sentence

In a network isolation environment,ZooKeeperofIPCan be configured as a businessIP, can also be configured to manageIP.

HiveUnlike traditional databases, soJDB, CThe established connection does not need to be closed.

In a secure cluster environment, establish andHiveBefore connecting, you need to use unified authentication to log in.

useJDB, CsubmitSQL, you canSQLA semicolon " ; " is added after the statement.

True

False

Producer

Broker

Consumer

ZooKeeper

True

False

submitcommand to change the status of the job toPREP

statcommand allowsPREPThe status of the job changes toRUNNINGrun

Runcommand to change the job status toRUNNINGl

KillCommands can act onPREP/SUSPEND/RUNNINGstatus job

Attribute values with high access weight are placed inRowKeyFront

Attribute values with high access weight are placed inRowkeysecond half

Attribute values with good dispersion are placed inRowkeyfirst half

Attribute values with good dispersion are placed inRowkeysecond half

Can start, stop and restart the service

Services can be added and uninstalled.

Uncommon services can be set to hide or show

Can view the current status of the service

yarn.nodemanager.local-dirs

yarn.nodemanager.log-dirs

yarn.acl.enable

yarn.log-aggregation-enable

C

Scala

Java

Python

INCR(increment the stored digital value by one)

HINCRBY(for the hash tablekeydomain infieldvalue plus incrementincrement

LPUSH(for linked listspushoperate)

All of the above

True

False

usehashstructure, and a table uses ahashKEY, a row in the table records forhash keyone of indivualed.

usehashTable structure, each row record of each table uses onehashKEY, hashKEYofebCorrespondence table

record fields, and designKEYEach table adds a different prefix to distinguish it.

useringstructure, each field of each row record of each table uses oneKEY.

usem3structure, each row record of each table uses oneKEY,,aueAll records for a row in the table. The value after the field is concatenated.

stream processing

high performance

Scalable

High fault tolerance

Updates sent by the client are applied in the order in which they were sent.

A message to be read by more than halfserverreceive, it will be able to successfully write to disk.

Message updates can only succeed or fail, with no intermediate states.

ZooKeeperThe number of nodes must be an odd number.