什么是TBase?
TBase是一个提供写可靠性，多主节点数据同步的关系数据库集群平台。你可以将TBase配置一台或者 多台主机上，TBase数据存储在多台物理主机上面。数据表的存储有两种方式， 分别是distributed或者replicated ，当向TBase发送查询 SQL时，TBase会自动向数据节点发出查询语句并获取终结果
TBase采用分布式集群架构（如下图）， 该架构分布式为无共享(share nothing)模式，节点之间相应独立，各自处理自己的数据，处理后的结果可能向上层汇总或在节点间流转，各处理单元之间通过网络协 议进行通信，并行处理和扩展能力更好，这也意味着只需要简单的x86服务器就可以部署TBase数据库集群
下面简单解读一下TBase的三大模块
Coordinator：协调节点（简称CN）
业务访问入口，负责数据的分发和查询规划，多个节点位置对等，每个节点都提供相同的数据库视 图；在功能上CN上只存储系统的全局元数据，并不存储实际的业务数据。
Datanode：数据节点（简称DN）
每个节点还存储业务数据的分片在功能上，DN节点负责完成执行协调节点分发的执行请求。
GTM:全局事务管理器(Global Transaction Manager)
负责管理集群事务信息，同时管理集群的全局对象，比如序列等。 接下来，让我们来看看如何从源码开始，完成到TBase集群环境的搭建。
TBase源码编译安装
创建tbase用户
注意：所有需要安装TBase集群的机器上都需要创建
mkdir /datauseradd -d /data/tbase tbase
源码获取
git clone https://github.com/Tencent/TBase
源码编译
cd ${SOURCECODE_PATH}rm -rf ${INSTALL_PATH}/tbase_bin_v2.0 chmod +x configure*./configure --prefix=${INSTALL_PATH}/tbase_bin_v2.0 --enable-user-switch -- with-openssl --with-ossp-uuid CFLAGS=-gmake clean make -sj make installchmod +x contrib/pgxc_ctl/make_signature cd contribmake -sj make install
本文的使用环境中，上述两个参数如下
${SOURCECODE_PATH}=/data/tbase/TBase-master
${INSTALL_PATH}=/data/tbase/install
集群安装
集群规划
下面以两台服务器上搭建1GTM主，1GTM备，2CN主（CN主之间对等，因此无需备CN），2DN 主，2DN备的集群，该集群为具备容灾能力的小配置
机器1：10.215.147.158机器2：10.240.138.159
集群规划如下
机器间的ssh互信配置 [tbase@TENCENT64 ~]$ vim ~/.bashrcexport TBASE_HOME=/data/tbase/install/tbase_bin_v2.0 export PATH=$TBASE_HOME/bin:$PATHexport LD_LIBRARY_PATH=$TBASE_HOME/lib:${LD_LIBRARY_PATH}
[tbase@TENCENT64 ~]$ mkdir /data/tbase/pgxc_ctl [tbase@TENCENT64 ~]$ cd /data/tbase/pgxc_ctl [tbase@TENCENT64 ~/pgxc_ctl]$ vim pgxc_ctl.conf
如下，是结合上文描述的IP，端口，数据库目录，二进制目录等规划来写的pgxc_ctl.conf文件。具 体实践中只需按照自己的实际情况配置好即可.
#!/bin/bashpgxcInstallDir=/data/tbase/install/tbase_bin_v2.0 pgxcOwner=tbasedefaultDatabase=postgrespgxcUser=$pgxcOwner tmpDir=/tmp localTmpDir=$tmpDir configBackup=n configBackupHost=pgxc-linker configBackupDir=$HOME/pgxc configBackupFile=pgxc_ctl.bak#---- GTM ----------gtmName=gtm gtmMasterServer=10.215.147.158 gtmMasterPort=50001 gtmMasterDir=/data/tbase/data/gtm gtmExtraConfig=none gtmMasterSpecificExtraConfig=none gtmSlave=y gtmSlaveServer=10.240.138.159 gtmSlavePort=50001 gtmSlaveDir=/data/tbase/data/gtm gtmSlaveSpecificExtraConfig=none#---- Coordinators -------coordMasterDir=/data/tbase/data/coord coordMasterDir=/data/tbase/data/coord coordArchLogDir=/data/tbase/data/coord_archlogcoordNames=(cn001 cn002 ) coordPorts=(30004 30004 )poolerPorts=(31110 31110 ) coordPgHbaEntries=(0.0.0.0/0) coordMasterServers=(10.215.147.158 10.240.138.159) coordMasterDirs=($coordMasterDir $coordMasterDir) coordMaxWALsernder=2coordMaxWALSenders=($coordMaxWALsernder $coordMaxWALsernder ) coordSlave=ncoordSlaveSync=ncoordArchLogDirs=($coordArchLogDir $coordArchLogDir)coordExtraConfig=coordExtraConfig cat > $coordExtraConfig <<EOF#================================================# Added to all the coordinator postgresql.conf # Original: $coordExtraConfiginclude_if_exists = '/data/tbase/global/global_tbase.conf'wal_level = replica wal_keep_segments = 256max_wal_senders = 4 archive_mode = on archive_timeout = 1800 archive_command = 'echo 0' log_truncate_on_rotation = onlog_filename = 'postgresql-%M.log' log_rotation_age = 4h log_rotation_size = 100MB hot_standby = onwal_sender_timeout = 30min wal_receiver_timeout = 30min shared_buffers = 1024MB max_pool_size = 2000 log_statement = 'ddl' log_destination = 'csvlog' logging_collector = on log_directory = 'pg_log' listen_addresses = '*' max_connections = 2000EOFcoordSpecificExtraConfig=(none none) coordExtraPgHba=coordExtraPgHbacat > $coordExtraPgHba <<EOFlocal all all trusthost all all 0.0.0.0/0 trusthost replication all 0.0.0.0/0 trusthost all all ::1/128 trusthost replication all ::1/128 trustEOFcoordSpecificExtraPgHba=(none none) coordAdditionalSlaves=ncad1_Sync=n#---- Datanodes ---------------------dn1MstrDir=/data/tbase/data/dn001 dn2MstrDir=/data/tbase/data/dn002 dn1SlvDir=/data/tbase/data/dn001 dn2SlvDir=/data/tbase/data/dn002 dn1ALDir=/data/tbase/data/datanode_archlog dn2ALDir=/data/tbase/data/datanode_archlogprimaryDatanode=dn001 datanodeNames=(dn001 dn002) datanodePorts=(40004 40004)datanodePoolerPorts=(41110 41110) datanodePgHbaEntries=(0.0.0.0/0) datanodeMasterServers=(10.215.147.158 10.240.138.159) datanodeMasterDirs=($dn1MstrDir $dn2MstrDir) dnWALSndr=4datanodeMaxWALSenders=($dnWALSndr $dnWALSndr)datanodeSlave=y datanodeSlaveServers=(10.240.138.159 10.215.147.158)datanodeSlavePorts=(50004 54004)datanodeSlavePoolerPorts=(51110 51110) datanodeSlaveSync=n datanodeSlaveDirs=($dn1SlvDir $dn2SlvDir)datanodeArchLogDirs=($dn1ALDir/dn001 $dn2ALDir/dn002)datanodeExtraConfig=datanodeExtraConfigcat > $datanodeExtraConfig <<EOF #================================================# Added to all the coordinator postgresql.conf # Original: $datanodeExtraConfiginclude_if_exists = '/data/tbase/global/global_tbase.conf' listen_addresses = '*'wal_level = replica wal_keep_segments = 256max_wal_senders = 4 archive_mode = on archive_timeout = 1800 archive_command = 'echo 0' log_directory = 'pg_log' logging_collector = on log_truncate_on_rotation = onlog_filename = 'postgresql-%M.log' log_rotation_age = 4h log_rotation_size = 100MB hot_standby = on wal_sender_timeout = 30min wal_receiver_timeout = 30min shared_buffers = 1024MB max_connections = 4000max_pool_size = 4000 log_statement = 'ddl' log_destination = 'csvlog' wal_buffers = 1GBEOFdatanodeSpecificExtraConfig=(none none) datanodeExtraPgHba=datanodeExtraPgHba cat > $datanodeExtraPgHba <<EOFlocal all all trusthost all all 0.0.0.0/0 trusthost replication all 0.0.0.0/0 trusthost all all ::1/128 trusthost replication all ::1/128 trustEOFdatanodeSpecificExtraPgHba=(none none)datanodeAdditionalSlaves=n walArchive=n
分发二进制包
在一个节点配置好配置文件后，需要预先将二进制包部署到所有节点所在的机器上，这个可以使用pgxc_ctl工具，执行deploy all命令来完成。
[tbase@TENCENT64 ~/pgxc_ctl]$ pgxc_ctl/usr/bin/bashInstalling pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Installing pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Reading configuration using /data/tbase/pgxc_ctl/pgxc_ctl_bash --home/data/tbase/pgxc_ctl --configuration /data/tbase/pgxc_ctl/pgxc_ctl.conf Finished reading configuration.******** PGXC_CTL START ***************Current directory: /data/tbase/pgxc_ctl PGXC deploy allDeploying Postgres-XL components to all the target servers. Prepare tarball to deploy ...Deploying to the server 10.215.147.158. Deploying to the server 10.240.138.159. Deployment done.登录到所有节点，check二进制包是否分发OK[tbase@TENCENT64 ~/install]$ ls /data/tbase/install/tbase_bin_v2.0 bin include lib share
[tbase@TENCENT64 ~]$ pgxc_ctl/usr/bin/bashInstalling pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Installing pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Reading configuration using /data/tbase/pgxc_ctl/pgxc_ctl_bash --home/data/tbase/pgxc_ctl --configuration /data/tbase/pgxc_ctl/pgxc_ctl.conf Finished reading configuration.******** PGXC_CTL START ***************Current directory: /data/tbase/pgxc_ctl PGXC init allInitialize GTM master........Initialize datanode slave dn001 Initialize datanode slave dn002mkdir: cannot create directory '/data1/tbase': Permission deniedchmod: cannot access '/data1/tbase/data/dn001': No such file or directory pg_ctl: directory "/data1/tbase/data/dn001" does not existpg_basebackup: could not create directory "/data1/tbase": Permission denied安装错误处理
一般init集群出错，终端会打印出错误日志，通过查看错误原因，更改配置即可，或者可以通 过/data/tbase/pgxc_ctl/pgxc_log路径下的错误日志查看错误，排查配置文件的错误
通过运行 pgxc_ctl 工具，执行clean all命令删除已经初始化的文件，修改pgxc_ctl.conf文件，重新执行init all命令重新发起初始化。
[tbase@TENCENT64 ~]$ pgxc_ctl/usr/bin/bashInstalling pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Installing pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Reading configuration using /data/tbase/pgxc_ctl/pgxc_ctl_bash --home/data/tbase/pgxc_ctl --configuration /data/tbase/pgxc_ctl/pgxc_ctl.conf Finished reading configuration.******** PGXC_CTL START ***************Current directory: /data/tbase/pgxc_ctl PGXC clean all[tbase@TENCENT64 ~]$ pgxc_ctl/usr/bin/bashInstalling pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Installing pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Reading configuration using /data/tbase/pgxc_ctl/pgxc_ctl_bash --home/data/tbase/pgxc_ctl --configuration /data/tbase/pgxc_ctl/pgxc_ctl.conf Finished reading configuration.******** PGXC_CTL START ***************Current directory: /data/tbase/pgxc_ctl PGXC init allInitialize GTM masterEXECUTE DIRECT ON (dn002) 'ALTER NODE dn002 WITH (TYPE=''datanode'', HOST=''10.240.138.159'', PORT=40004, PREFERRED)';EXECUTE DIRECTEXECUTE DIRECT ON (dn002) 'SELECT pgxc_pool_reload()'; pgxc_pool_reload------------------t(1 row)Done.查看集群状态
[tbase@TENCENT64 ~/pgxc_ctl]$ pgxc_ctl/usr/bin/bashInstalling pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Installing pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Reading configuration using /data/tbase/pgxc_ctl/pgxc_ctl_bash --home/data/tbase/pgxc_ctl --configuration /data/tbase/pgxc_ctl/pgxc_ctl.conf Finished reading configuration.******** PGXC_CTL START ***************Current directory: /data/tbase/pgxc_ctl PGXC monitor allRunning: gtm master Not running: gtm slaveRunning: coordinator master cn001 Running: coordinator master cn002 Running: datanode master dn001 Running: datanode slave dn001
一般的如果配置的不是强同步模式，gtm salve，dn slave的故障不会影响的访问。
[tbase@TENCENT64 ~/pgxc_ctl]$ psql -h 10.215.147.158 -p 30004 -d postgres -U tbasepsql (PostgreSQL 10.0 TBase V2)Type "help" for help.postgres=# \dDid not find any relations. postgres=# select * from pgxc_node;node_name | node_type | node_port | node_host | nodeis_primarynodeis_preferred | node_id | node_cluster_name-----------+-----------+-----------+----------------+----------------+------------------+------------+-------------------gtm | G | 50001 | 10.215.147.158 | t | f| 428125959 | tbase_clustercn001 | C | 30004 | 10.215.147.158 | f | f| -264077367 | tbase_clustercn002 | C | 30004 | 10.240.138.159 | f | f| -674870440 | tbase_clusterdn001 | D | 40004 | 10.215.147.158 | t | t| 2142761564 | tbase_clusterdn002 | D | 40004 | 10.240.138.159 | f | f| -17499968 | tbase_cluster(5 rows)
TBase使用datanode group来增加节点的管理灵活度，要求有一个default group才能使用，因此需要预先创建；一般情况下，会将节点的所有datanode节点加入到default group里。
postgres=# create default node group default_group with (dn001,dn002); CREATE NODE GROUPpostgres=# create sharding group to group default_group; CREATE SHARDING GROUP
postgres=# create database test; CREATE DATABASEpostgres=# create user test with password 'test'; CREATE ROLEpostgres=# alter database test owner to test; ALTER DATABASEpostgres=# \c test testYou are now connected to database "test" as user "test".test=> create table foo(id bigint, str text) distribute by shard(id); CREATE TABLEtest=> insert into foo values(1, 'tencent'), (2, 'shenzhen');COPY 2test=> select * from foo; id | str----+----------1| tencent2| shenzhen (2 rows)
PGXC stop all -m fastStopping all the coordinator masters. Stopping coordinator master cn001.Stopping coordinator master cn002. Done.Stopping all the datanode slaves. Stopping datanode slave dn001.Stopping datanode slave dn002.pg_ctl: PID file "/data/tbase/data/dn002/postmaster.pid" does not exist Is server running?Stopping all the datanode masters. Stopping datanode master dn001.Stopping datanode master dn002. Done.Stop GTM slavewaiting for server to shut down doneserver stopped Stop GTM masterwaiting for server to shut down doneserver stopped PGXC monitor allNot running: gtm master Not running: gtm slaveNot running: coordinator master cn001
Not running: coordinator master cn002: datanode master dn001: datanode slave dn001Not running: datanode master dn002: datanode slave dn002
启动集群
[tbase@TENCENT64 ~]$ pgxc_ctl/usr/bin/bashInstalling pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Installing pgxc_ctl_bash script as /data/tbase/pgxc_ctl/pgxc_ctl_bash. Reading configuration using /data/tbase/pgxc_ctl/pgxc_ctl_bash --home/data/tbase/pgxc_ctl --configuration /data/tbase/pgxc_ctl/pgxc_ctl.conf Finished reading configuration.******** PGXC_CTL START ***************Current directory: /data/tbase/pgxc_ctl PGXC start all
