zookeeper是一个挺好玩的东西
有着独特的选举机制,一般在中小型集群中,zookeeper一般装在三个节点
其中只有一个节点对外提供服务,处于leader状态,另外两台未follower状态
这得益于zookeeper独特的选举机制,可以保证leader节点的主机挂掉后,
从节点可以通过选举机制很快成为leader节点并对外提供服务。
zookeeper的结构为树状结构,其每个树状节点中存储着其他组件的元数据。
本篇内容主讲znode节点的一些功能和使用,以及ACL的一些问题。
#找到zookeeper的运行目录
cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/
运行脚本zkCli.sh
[root@cloud01 bin]# ./zkCli.sh
Connecting to localhost:2181
2024-04-07 11:29:08,782 [myid:] – INFO [main:Environment@100] – Client environment:zookeeper.version=3.4.5-cdh6.3.2–1, built on 02/01/2022 16:17 GMT
2024-04-07 11:29:08,786 [myid:] – INFO [main:Environment@100] – Client environment:host.name=cloud01
2024-04-07 11:29:08,786 [myid:] – INFO [main:Environment@100] – Client environment:java.version=1.8.0_161
2024-04-07 11:29:08,790 [myid:] – INFO [main:Environment@100] – Client environment:java.vendor=Oracle Corporation
2024-04-07 11:29:08,790 [myid:] – INFO [main:Environment@100] – Client environment:java.home=/usr/local/jdk8/jre
2024-04-07 11:29:08,790 [myid:] – INFO [main:Environment@100] – Client environment:java.class.path=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/build/classes:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/build/lib/.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/lib/slf4j-log4j12-1.7.25.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/lib/slf4j-api-1.7.25.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/lib/netty-3.10.6.Final.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/lib/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/lib/jline-2.11.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/lib/audience-annotations-0.5.0.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/zookeeper-3.4.5-cdh6.3.2.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/src/java/lib/.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin/…/conf:.:/usr/local/jdk8/lib:/usr/local/jdk8/jre/lib:
2024-04-07 11:29:08,790 [myid:] – INFO [main:Environment@100] – Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2024-04-07 11:29:08,790 [myid:] – INFO [main:Environment@100] – Client environment:java.io.tmpdir=/tmp
2024-04-07 11:29:08,790 [myid:] – INFO [main:Environment@100] – Client environment:java.compiler=
2024-04-07 11:29:08,790 [myid:] – INFO [main:Environment@100] – Client environment:os.name=Linux
2024-04-07 11:29:08,791 [myid:] – INFO [main:Environment@100] – Client environment:os.arch=amd64
2024-04-07 11:29:08,791 [myid:] – INFO [main:Environment@100] – Client environment:os.version=3.10.0-1127.13.1.el7.x86_64
2024-04-07 11:29:08,791 [myid:] – INFO [main:Environment@100] – Client environment:user.name=root
2024-04-07 11:29:08,791 [myid:] – INFO [main:Environment@100] – Client environment:user.home=/root
2024-04-07 11:29:08,791 [myid:] – INFO [main:Environment@100] – Client environment:user.dir=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p5253.21605619/lib/zookeeper/bin
2024-04-07 11:29:08,791 [myid:] – INFO [main:ZooKeeper@619] – Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMainM
y
W
a
t
c
h
e
r
@
7506
e
922
W
e
l
c
o
m
e
t
o
Z
o
o
K
e
e
p
e
r
!
2024
−
04
−
0711
:
29
:
08
,
895
[
m
y
i
d
:
]
−
I
N
F
O
[
m
a
i
n
−
S
e
n
d
T
h
r
e
a
d
(
l
o
c
a
l
h
o
s
t
:
2181
)
:
C
l
i
e
n
t
C
n
x
n
MyWatcher@7506e922 Welcome to ZooKeeper! 2024-04-07 11:29:08,895 [myid:] – INFO [main-SendThread(localhost:2181):ClientCnxn
MyWatcher@7506e922WelcometoZooKeeper!2024−04−0711:29:08,895[myid:]−INFO[main−SendThread(localhost:2181):ClientCnxnSendThread@1118] – Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2024-04-07 11:29:09,008 [myid:] – INFO [main-SendThread(localhost:2181):ClientCnxnS
e
n
d
T
h
r
e
a
d
@
962
]
−
S
o
c
k
e
t
c
o
n
n
e
c
t
i
o
n
e
s
t
a
b
l
i
s
h
e
d
,
i
n
i
t
i
a
t
i
n
g
s
e
s
s
i
o
n
,
c
l
i
e
n
t
:
/
127.0.0.1
服务器托管
:33928
,
s
e
r
v
e
r
:
l
o
c
a
l
h
o
s
t
/
127.0.0.1
:
21812024
−
04
−
0711
:
29
:
09
,
068
[
m
y
i
d
:
]
−
I
N
F
O
[
m
a
i
n
−
S
e
n
d
T
h
r
e
a
d
(
l
o
c
a
l
h
o
s
t
:
2181
)
:
C
l
i
服务器托管 e
n
t
C
n
x
n
SendThread@962] – Socket connection established, initiating session, client: /127.0.0.1:33928, server: localhost/127.0.0.1:2181 2024-04-07 11:29:09,068 [myid:] – INFO [main-SendThread(localhost:2181):ClientCnxn
SendThread@962]−Socketconnectionestablished,initiatingsession,client:/127.0.0.1:33928,server:localhost/127.0.0.1:21812024−04−0711:29:09,068[myid:]−INFO[main−SendThread(localhost:2181):ClientCnxnSendThread@1378] – Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0xff8ea1e469371d73, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
查看zookeeper下面的节点
[zk: localhost:2181(CONNECTED) 2] ls /
[cluster, controller, brokers, zookeeper, yarn-leader-election, hadoop-ha, admin, isr_change_notification, log_dir_event_notification, controller_epoch, rmstore, consumers, hive_zookeeper_namespace_hive, latest_producer_id_block, config, hbase, sentry]
zookeeper在cdh集群中又扮演什么角色呢?
这是在hbase配置中的设置
ZooKeeper 是 HBase 集群的协调服务,它存储了 HBase 集群的状态信息、表结构信息、RegionServer 的信息。一般存储在zookeeper中的hbase节点下
[zk: localhost:2181(CONNECTED) 3] ls /hbase
[meta-region-server, rs, splitWAL, backup-masters, table-lock, flush-table-proc, master-maintenance, online-snapshot, acl, switch, master, running, balancer, tokenauth, draining, namespace, hbaseid, table]
这是hdfs组件中的配置
hdfs要依赖于zookeeper来实现自身的高可用性(zkfc机制);
[zk: localhost:2181(CONNECTED) 4] ls /
[cluster, controller, brokers, zookeeper, yarn-leader-election, hadoop-ha, admin, isr_change_notification, log_dir_event_notification, controller_epoch, rmstore, consumers, hive_zookeeper_namespace_hive, latest_producer_id_block, config, hbase, sentry]
[zk: localhost:2181(CONNECTED) 5] ls /hadoop-ha
[nameservice1]
[zk: localhost:2181(CONNECTED) 6]
这是hive中的配置
Hive 的 HA 配置:在配置 Hive 的高可用性(High Availability,HA)环境时,ZooKeeper 可以用于管理和协调 Hive 的主从节点,确保 Hive 元数据服务的可用性和一致性。
这是impala中的配置
ZooKeeper 可以用作元数据存储,帮助 Impala 管理元数据信息的分布和一致性。
这是在kafka上的配置
做元数据存储,还有就是协助选举leader
这是在Sentry中的配置
元数据存储和管理。
这是在yarn中的配置
高可用,元数据存储。
关于ACL权限,开启Kerberos认证后,这个就不开了。
而且这个权限每个目录节点的权限都是独立的。
服务器托管,北京服务器托管,服务器租用 http://www.fwqtg.net