cloud
共 28 篇文章
Cloud Security?
看了一些文章,主要是保证用户怎么保证存储在公有云的数据的完整性和正确性,即使用一套逻辑或者叫协议供用户和云存储提供者使用,使得用户能够相信数据放在存储厂商那里没有被删改,而且数据对于厂商是加密的。基于早就成型的密码学算法,这些文章都给出了有效的算法。 现在问题是如果用户不但要用公有云的存储,还要用它的计算结点,怎么保证整个过程的安全呢?
2011-09-02 14:23 · cloud
【读书笔记】Data warehousing and analytics infrastructure at facebook
这好像是sigmod2010上的paper。 读了之后做了以下几点记录: 1. facebook的hadoop cluster分成 scribe hadoop cluster: scribe servers将web service的log汇总然后存到HDFS上。通常会有带宽成为bottleneck的问题,这时候可以考虑压缩,但是一个副作用就是在buffer待压缩的数据的同时导致latency增大; production-hive hadoop cluster:通常处理schedule好的,有严格deadline的任务; Adhoc hive hadoop cluster:优先级较低的批处理任…
cassandra example
http://www.rackspacecloud.com/blog/2010/05/12/cassandra-by-example/
2011-01-19 16:39 · cloud
impact of total region numbers?
这几天tune了hbase的几个参数,有些有意思的结果。具体看我下面的邮件吧。 For example, I have total some data and I can tune hbase.hregion.max.filesize to increase/decrease total region number, rite? I want to know if the region number has performance impact to random read tests. I observed that in my ycsb test, with larger hfile s…
Will all HFiles managed by a regionserver kept open
code 没看仔细,所以在hbase 的mail list上面问了这么个问题。其实再仔细看一下big table的paper就知道肯定是open的。现在分析的结果是hbase random read的performance决定在几个方面: 1)HDFS的seek操作,平均每次random get导致几次seek? 2)memory copy; 这个问题尤其在data locality差的时候,比如datanode 和regionserver不在一个node上的时候; 3)block cache? hi, I know generally regionserver manages HRegio…
problems in building hadoop
When I try to modify some code in ASF hadoop-0.20.2 and compile it with command "ant jar". It reports no *route to host* exceptions. I do this on a server that with no direct internet connection, that is, I need to set a proxy to access outside network. So to build hadoop, maven needs to download s…
HDFS scalability: the limits to growth
Abstract: T**he Hadoop Distributed File Sys-**tem (HDFS) is an open source system currently being used in situations where massive amounts of data need to be processed. Based on experience with the largest deployment of HDFS, I provide an analysis of how the amount of RAM of a single namespace serv…
hadoop-0.20.2+737 and hbase-0.20.6 not compatible?
master log里面发现 0 region servers, 0 dead, average load NaN regionserver hang住。 原因不明。
hbase master cannot start up
2010-10-06 15:21:44,704 INFO org.apache.hadoop.hbase.master.HMaster: vmName=Java HotSpot(TM) Client VM, vmVendor=Sun Microsystems Inc., vmVersion=16.3-b01 2010-10-06 15:21:44,704 INFO org.apache.hadoop.hbase.master.HMaster: vmInputArguments=[-Xmx1000m, -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcM…
细说HBase怎么完成一个Get操作 (client side)
源码解析基于HBase-0.20.6。 先看HTable类get()方法的code: HTable.java /** * Extracts certain cells from a given row. * @param get The object that specifies what data to fetch and from which row. * @return The data coming from the specified row, if it exists. If the row * specified doesn't exist, the {@link Result…
Hbase read performance with increasing number of client threads
在跑ycsb的时候,发现在做heavy的get操作的时候,ycsb统计的latency很大,100个thread的时候就接近100ms,而从ganglia上看hbase的“get_avg_time"这个metric发现只有20~30左右。最终查看code,发现100个thread共享同一个连接,所有Call的请求数据都走这一个连接,所以在大量请求并发时会造成拥堵,latency变大。具体看下面这个方法: public Writable[] call(Writable[] params, InetSocketAddress[] addresses) throws IOException { i…
What are some tips for configuring HBase?
Jeff Hammerbacher 8 endorsements 2 votes by Oleksiy Kovyrin and Alex Kamil Much of this content is taken from the HBase Overview [1] and the HBase Default Configuration [2] . System Increase the default per-process file handle limit [3] in /etc/security/limits.conf HDFS Set dfs.datanode.max.xceiver…
当hbase的master无法停掉的时候……
some times you will find the master in a hbase cluster cannot be stopped, so check the following two items suggested by some guy in hbase user mail list. But I'm not sure it helps. > If zookeeper is not running (or do not have the quorum) in a cluster setup, hbase does not go down..bring up zookeep…
云计算 定义
【云计算】概念是由Google提出的,这是一个美丽的网络应用模式。狭义云计算是指IT基础设施的交付和使用模式,指通过网络以按需、易扩展的方式获得所需的资源;广义云计算是指服务的交付和使用模式,指通过网络以按需、易扩展的方式获得所需的服务。这种服务可以是IT和软件、互联网相关的,也可以是任意其他的服务,它具有超大规模、虚拟化、可靠安全等独特功效;“云计算”图书版本也很多,都从理论和实践上介绍了云计算的特性与功用。 (1) 超大规模 具有相当的规模,Google云计算已经拥有100多万台服务器, Amazon、IBM、微软、Yahoo等的“云”均拥有几十万台服务器。企业私有云一般拥有数百上千台服…
2010-09-10 17:26 · cloud
zookeeper介绍
ZooKeeper是Hadoop的正式子项目,它是一个针对大型分布式系统的可靠协调系统,提供的功能包括:配置维护、名字服务、分布式同步、组服务等。ZooKeeper的目标就是封装好复杂易出错的关键服务,将简单易用的接口和性能高效、功能稳定的系统提供给用户。 Zookeeper是Google的Chubby一个开源的实现.是高有效和可靠的协同工作系统.Zookeeper能够用来leader选举,配置信息维护等.在一个分布式的环境中,我们需要一个Master实例或存储一些配置信息,确保文件写入的一致性等.Zookeeper能够保证如下3点: Watches are ordered with res…
HDFS
Run hadoop fsck / will give you summary of current HDFS status including some useful information : Minimally replicated blocks: 51224 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 7 (0.013665469 %) Default replication factor: 3 Average block r…
DataCenter使用SSD Drive的好处
http://arstechnica.com/business/news/2009/10/latest-migrations-show-ssd-is-ready-for-some-datacenters.ars 1. save floor space. 2. cost saving in the long run: take power and cooling cost into count, a datacenter usually spend 40 of its budge in power and cooling. Using SSD can save 80% cooling cost…
2009-10-29 14:27 · cloud
Exceptions in HDFS
Log them here for later analysis. 2009-08-26 01:17:37,798 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in receiveBlock for block blk_5223350282761282817_281131 java.nio.channels.ClosedByInterruptException 2009-08-26 01:17:37,799 INFO org.apache.hadoop.hdfs.server.datanode.DataNod…